On Thu, Jun 19, 2014 at 03:12:05AM +0800, Ade Lee wrote:

This is something that has been on the wishlist for awhile. There is no mechanism at this point to replicate profiles.

I agree that we should start this design.

Ade

LDAP Profile Storage Design proposal (work in progress) is up on the wiki: http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage Input and feedback greatly appreciated, especially if anyone could give guidance on the LDAP schema - I have no prior experience with developing LDAP schemata. Have a nice weekend, all. Fraser

On Wed, 2014-06-18 at 17:44 +1000, Fraser Tweedale wrote:

...

Hi all,

A requirement from the FreeIPA side is the ability to add and customise CA profiles. Dogtag's current profile creation behaviour writes the new profile to the filesystem beside the standard profiles (as well as making the appropriate update to the registry, etc.)

There does not seem to be a mechanism to distribute new/modified profiles to replicas - though perhaps I have missed something.

Because this behaviour is required, unless I have overlooked something or there is a better way (in which case please shout out), I think it makes sense to begin a design proposal for an LDAP-based profile store.

Finally, a brief mention of some tickets related to profile storage that could be good to tackle simultaneously should the proposed change go ahead:

- https://fedorahosted.org/pki/ticket/778 - https://fedorahosted.org/freeipa/ticket/4002

_______________________________________________ Pki-devel mailing list Pki-devel@redhat.com https://www.redhat.com/mailman/listinfo/pki-devel

I made comments in the design page in the file/ldap section. Its pretty late - so I'll continue to make comments tomorrow. Ade On Tue, 2014-06-24 at 17:07 +1000, Fraser Tweedale wrote:

On Fri, Jun 20, 2014 at 06:00:25PM +1000, Fraser Tweedale wrote:

...

On Thu, Jun 19, 2014 at 03:12:05AM +0800, Ade Lee wrote:

...

This is something that has been on the wishlist for awhile. There is no mechanism at this point to replicate profiles.

I agree that we should start this design.

Ade

LDAP Profile Storage Design proposal (work in progress) is up on the wiki: http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage

Input and feedback greatly appreciated, especially if anyone could give guidance on the LDAP schema - I have no prior experience with developing LDAP schemata.

Have a nice weekend, all.

Fraser

I've fleshed out the design proposal some more; getting close to ready now, modulo feedback and general approval.

Particular sections for which I would appreciate feedback are:

- http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage#Relationship_to_file-... - whether deletion of file-based profiles should be prohibited - whether a *restore profile* method is needed

- http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage#LDAP_schema - Need feedback from people who understand LDAP schema better than I :)

- http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage#Cloning - Need feedback from people who know more than me about the cloning process.

Cheers,

Fraser

...

...

On Wed, 2014-06-18 at 17:44 +1000, Fraser Tweedale wrote:

...

Hi all,

A requirement from the FreeIPA side is the ability to add and customise CA profiles. Dogtag's current profile creation behaviour writes the new profile to the filesystem beside the standard profiles (as well as making the appropriate update to the registry, etc.)

There does not seem to be a mechanism to distribute new/modified profiles to replicas - though perhaps I have missed something.

Because this behaviour is required, unless I have overlooked something or there is a better way (in which case please shout out), I think it makes sense to begin a design proposal for an LDAP-based profile store.

Finally, a brief mention of some tickets related to profile storage that could be good to tackle simultaneously should the proposed change go ahead:

- https://fedorahosted.org/pki/ticket/778 - https://fedorahosted.org/freeipa/ticket/4002

_______________________________________________ Pki-devel mailing list Pki-devel@redhat.com https://www.redhat.com/mailman/listinfo/pki-devel

_______________________________________________ Pki-devel mailing list Pki-devel@redhat.com https://www.redhat.com/mailman/listinfo/pki-devel

On Wed, Jun 25, 2014 at 02:51:08PM -0700, Christina Fu wrote:

Yes this has been on our wish list.

I only want to comment on the Access Control Considerations for profiles. Please make sure the current security control in place is preserved. i.e. a profile addition or update by an administrator requires an agent's approval -- * update of an existing profile - agent disables the profile, admin then is allowed to update, agent reviews the profile and enables it. * adding a new profile - admin creates the profile, agent approves it

Christina

Thanks Christina; the current security controls will be preserved. Fraser

On 06/24/2014 12:07 AM, Fraser Tweedale wrote:

...

On Fri, Jun 20, 2014 at 06:00:25PM +1000, Fraser Tweedale wrote:

...

On Thu, Jun 19, 2014 at 03:12:05AM +0800, Ade Lee wrote:

...

This is something that has been on the wishlist for awhile. There is no mechanism at this point to replicate profiles.

I agree that we should start this design.

Ade

LDAP Profile Storage Design proposal (work in progress) is up on the wiki: http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage

Input and feedback greatly appreciated, especially if anyone could give guidance on the LDAP schema - I have no prior experience with developing LDAP schemata.

Have a nice weekend, all.

Fraser

I've fleshed out the design proposal some more; getting close to ready now, modulo feedback and general approval.

Particular sections for which I would appreciate feedback are:

- http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage#Relationship_to_file-... - whether deletion of file-based profiles should be prohibited - whether a *restore profile* method is needed

- http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage#LDAP_schema - Need feedback from people who understand LDAP schema better than I :)

- http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage#Cloning - Need feedback from people who know more than me about the cloning process.

Cheers,

Fraser

...

...

On Wed, 2014-06-18 at 17:44 +1000, Fraser Tweedale wrote:

...

Hi all,

A requirement from the FreeIPA side is the ability to add and customise CA profiles. Dogtag's current profile creation behaviour writes the new profile to the filesystem beside the standard profiles (as well as making the appropriate update to the registry, etc.)

There does not seem to be a mechanism to distribute new/modified profiles to replicas - though perhaps I have missed something.

Because this behaviour is required, unless I have overlooked something or there is a better way (in which case please shout out), I think it makes sense to begin a design proposal for an LDAP-based profile store.

Finally, a brief mention of some tickets related to profile storage that could be good to tackle simultaneously should the proposed change go ahead:

- https://fedorahosted.org/pki/ticket/778 - https://fedorahosted.org/freeipa/ticket/4002

_______________________________________________ Pki-devel mailing list Pki-devel@redhat.com https://www.redhat.com/mailman/listinfo/pki-devel

_______________________________________________ Pki-devel mailing list Pki-devel@redhat.com https://www.redhat.com/mailman/listinfo/pki-devel

---

Pki-devel mailing list Pki-devel@redhat.com https://www.redhat.com/mailman/listinfo/pki-devel

_______________________________________________ Pki-devel mailing list Pki-devel@redhat.com https://www.redhat.com/mailman/listinfo/pki-devel

On 06/30/2014 03:28 PM, Endi Sukma Dewata wrote:

Hi, I have some comments about the design: http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage

I think all system/default profiles should remain file-based and all custom profiles should be LDAP-based. It will make a clean separation: system profiles are owned by us (Dogtag developers), custom profiles are owned by the admin.

The system profiles will be read-only. This way we will be able to update the system profiles without writing any upgrade scripts because the files will be updated automatically by RPM. Just one requirement, all server instances must be upgraded to the same version.

If the admin wants to change a system profile, they can clone it into a custom profile and make the changes there. The custom profiles cannot have the same names as the system profiles, so there's won't be any conflict/confusion, and no need to support a "restore" command. In general we won't need to write upgrade scripts for custom profiles except if we change the LDAP schema.

About the schema itself, I suppose we want to have something that resembles the actual Profile data structure (see ProfileData Java class). There should be an LDAP attribute for each single-valued Java attribute (e.g. name, description, enabled, visible). This way the profile is more manageable and can be queried based on these attributes. For collection attributes (e.g. inputs, outputs, policySets) we can use child LDAP entries to represent them.

About the names, right now we use "profile" to refer to both cert profile in CA and token profile in TPS. We probably should use separate terms to distinguish them (e.g. CertProfile and TokenProfile). This applies to Java/Python class names, REST interface, CLI, LDAP schema, etc.

So the LDAP entries for a profile may look like this:

dn: cn=MyUserCert,ou=Profiles,ou=CA,{suffix} objectClass: certProfile cn: MyUserCert displayName: Manual User Dual-Use Certificate Enrollment description: This certificate profile is for enrolling user certificates. visible: true enabled: true enabledBy: admin

dn: ou=Inputs,cn=MyUserCert,ou=Profiles,ou=CA,{suffix} objectClass: organizationalUnit ou: Inputs

dn: cn=i1,ou=Inputs,cn=MyUserCert,ou=Profiles,ou=CA,o=pki-tomcat objectClass: pkiCertProfileInput cn: i1 classId: keyGenInputImpl

About the REST interface & CLI, since this will be the primary way to edit profiles, we might want to have more granular commands to modify parts of the profile. Right now with ca-profile-mod command you need to send the entire profile in a file. It would be nice to be able to specify some parameters to change certain attributes only, or use separate commands to manage the inputs/outputs.

We'll also need an interface to find existing cert records that use a certain profile and bulk modify them to use a different profile. This will be useful when you create a clone to change the system profile.

I'm not sure if we should support 10.2 -> 10.3 cloning. When we release 10.3 the 10.2 will still be fairly new so it might be reasonable to require all clones to be upgraded. It will reduce the amount of testing requirement too.

Makes sense. -- Thank you, Dmitri Pal Sr. Engineering Manager IdM portfolio Red Hat, Inc.

On Tue, 2014-07-01 at 12:42 +1000, Fraser Tweedale wrote:

On Mon, Jun 30, 2014 at 02:28:09PM -0500, Endi Sukma Dewata wrote:

...

Hi, I have some comments about the design: http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage

I think all system/default profiles should remain file-based and all custom profiles should be LDAP-based. It will make a clean separation: system profiles are owned by us (Dogtag developers), custom profiles are owned by the admin.

The system profiles will be read-only. This way we will be able to update the system profiles without writing any upgrade scripts because the files will be updated automatically by RPM. Just one requirement, all server instances must be upgraded to the same version.

If the admin wants to change a system profile, they can clone it into a custom profile and make the changes there. The custom profiles cannot have the same names as the system profiles, so there's won't be any conflict/confusion, and no need to support a "restore" command. In general we won't need to write upgrade scripts for custom profiles except if we change the LDAP schema.

This was my original proposal, with the exception of making system/default profiles read-only and cloning them to make modifications. I think this is a good modification to that proposal.

Ade was in favour of moving to LDAP only, but perhaps this variation makes using a mix of LDAP and file-based profiles more palatable? It would be good to hear from more people on this part of the design - to see if there's a clear winner.

The problem with the above proposal is one of migration. Customers are used to both creating new profiles and modifying existing profiles to suit their needs. If we go along with this mechanism, we would have to tell these customers that they would need to rename their existing customized profiles (and update all the relevant clients etc.) There is another problem, and that is that it is not clear that we want updates to the default profiles to be propagated to existing instances. I have looked at the profiles and there have been only a handful of changes over the last 7 years. Those changes include things like updating the default signing algorithms or the default validity. More likely than not, admins would prefer that we not change the behavior of profiles in existing instances underneath them. The changes that I have found are all behavioral - and therefore things that admin can opt out of -- or would prefer to do on their own schedule. There have been no structural changes. If there are structural changes, then we need to (and can) provide an upgrade script which would run with the automatic upgrade. An example of this would be a schema upgrade as we sort out how to represent profiles in LDAP. So, I think we should do the following: 1. Continue to provide the system profiles in files. These files will be parsed and stored in LDAP when an instance is created. 2. All profiles for an instance should live in LDAP. This makes it simple - no need to check to see if a profile is in LDAP or files or both, and which has priority etc. Tools will be provided to manage/ create/delete profiles etc. 3. Updates to system profile files will not affect the existing LDAP profiles. We can provide update scripts or manual instructions for admins to run when they opt to do so. This will be for behavioral changes. 4. Structural changes will be done using upgrade scripts using the database upgrade mechanism. This framework is something that we had planned to do already in 10.3. We already have a model on how to do this in our current upgrade framework.

...

About the schema itself, I suppose we want to have something that resembles the actual Profile data structure (see ProfileData Java class). There should be an LDAP attribute for each single-valued Java attribute (e.g. name, description, enabled, visible). This way the profile is more manageable and can be queried based on these attributes. For collection attributes (e.g. inputs, outputs, policySets) we can use child LDAP entries to represent them.

About the names, right now we use "profile" to refer to both cert profile in CA and token profile in TPS. We probably should use separate terms to distinguish them (e.g. CertProfile and TokenProfile). This applies to Java/Python class names, REST interface, CLI, LDAP schema, etc.

Good point. I've s/profile/certProfile/ in the schema.

...

So the LDAP entries for a profile may look like this:

dn: cn=MyUserCert,ou=Profiles,ou=CA,{suffix} objectClass: certProfile cn: MyUserCert displayName: Manual User Dual-Use Certificate Enrollment description: This certificate profile is for enrolling user certificates. visible: true enabled: true enabledBy: admin

dn: ou=Inputs,cn=MyUserCert,ou=Profiles,ou=CA,{suffix} objectClass: organizationalUnit ou: Inputs

dn: cn=i1,ou=Inputs,cn=MyUserCert,ou=Profiles,ou=CA,o=pki-tomcat objectClass: pkiCertProfileInput cn: i1 classId: keyGenInputImpl

Introducing this schema (and the more fine-grained functionality it would support) could happen later. I see the benefits but I think it's better to just get the profiles in LDAP as soon as possible, then look at how we can break them down to enable richer features.

I would prefer to see if we could design something more fine grained at first, rather than assuming an upgrade. If nothing else, you will need to separate out whether the profile is visible and enabled and by whom.

Also, if we retain file-based profiles as you suggest, whatever queries LDAP makes easier, we would still need to be able to do those same queries across the file-based profiles. So while file-based profiles remain, a finer-grained LDAP profile schema doesn't buy us anything in terms of reducing complexity or the amount of code to write.

...

About the REST interface & CLI, since this will be the primary way to edit profiles, we might want to have more granular commands to modify parts of the profile. Right now with ca-profile-mod command you need to send the entire profile in a file. It would be nice to be able to specify some parameters to change certain attributes only, or use separate commands to manage the inputs/outputs.

We'll also need an interface to find existing cert records that use a certain profile and bulk modify them to use a different profile. This will be useful when you create a clone to change the system profile.

Again, I agree with the benefits here, but feel that this may be out of scope for the initial changeover to LDAP profiles, given how much we need that for FreeIPA.

Agreed. I think this can be a later development.

...

I'm not sure if we should support 10.2 -> 10.3 cloning. When we release 10.3 the 10.2 will still be fairly new so it might be reasonable to require all clones to be upgraded. It will reduce the amount of testing requirement too.

I'll incorporate this comment (and all your others) in the design proposal; you should see the updated document soon.

Cheers,

Fraser

...

-- Endi S. Dewata

On 6/18/2014 2:44 AM, Fraser Tweedale wrote:

...

Hi all,

A requirement from the FreeIPA side is the ability to add and customise CA profiles. Dogtag's current profile creation behaviour writes the new profile to the filesystem beside the standard profiles (as well as making the appropriate update to the registry, etc.)

There does not seem to be a mechanism to distribute new/modified profiles to replicas - though perhaps I have missed something.

Because this behaviour is required, unless I have overlooked something or there is a better way (in which case please shout out), I think it makes sense to begin a design proposal for an LDAP-based profile store.

Finally, a brief mention of some tickets related to profile storage that could be good to tackle simultaneously should the proposed change go ahead:

- https://fedorahosted.org/pki/ticket/778 - https://fedorahosted.org/freeipa/ticket/4002

_______________________________________________ Pki-devel mailing list Pki-devel@redhat.com https://www.redhat.com/mailman/listinfo/pki-devel

On 7/2/2014 1:09 PM, Christina Fu wrote:

In general, I think we all agree that putting profiles on ldap is a good idea. However, in practice, I think we want to allow admins to make changes to local files as they see fit. Please see my in-line response below.

This is probably something that the admins will have to adapt eventually. I think in a replicated environment we shouldn't treat each server as unique individuals with "local" settings in files (including profiles). From user's perspective all replicas should be (mostly) identical. An admin, agent, or end entity should be able to go to any replica and get the same services, see the same data, etc. For this to work consistently, we want to minimize possible misconfiguration, so the number of local files should be minimized. The configuration (including profiles) should gradually be moved to LDAP, and the CLI/UI will be the primary way to manage the configuration. To help the transition, CLI like proposed by Fraser could provide the experience of editing a local file but it's actually stored on LDAP. Also in the future the UI could provide an intuitive interface to manage the profiles, so the admins will no longer need to memorize the syntax of profile configuration file.

Perhaps we can view files v.s. ldap as "localized" v.s. "centralized" instead of "system" v.s. "customized". Again, the benefit of putting profiles in ldap is so that it can be configured once and utilized by many. My comments below are based on such view.

Let's call this proposal #4: Localized & centralized profiles. If I understand this correctly, as an example server1 may have local profile1 and profile2, server2 may have local profile1 and profile3, then we have centralized LDAP profile1 and profile4. The profile1 in server1, server2, and LDAP can be completely different although they share the same name.

...

1. Continue to provide the system profiles in files. These files will be parsed and stored in LDAP when an instance is created.

if the ldap copies are the "centralized" ones, then we don't want to override them at each installation of an instance. We could give admins an option to override what's stored in ldap during configuration.

I think the process to import the system profiles into LDAP should be done only once regardless of the number of replicas. Otherwise, suppose we removed one of the centralized profile (e.g. profile1), it might get added again when we install another replica.

When a profile with the same id exists in both ldap and local system, then by default the one on ldap takes precedence. Although we could add "per profile" config param for each instance to change the view (remember the unix ns/yp config where you can set the order of the source? so if "file" is 1st in the order, then the local ones have priority, etc.)

I don't have a lot of experience as system admin, but I think managing individual machines is difficult so NIS (or LDAP, NFS, DNS, etc.) is used to transition to a centrally managed system. The order of sources is set up such that eventually everything can be moved into a single source. For example, once we have NIS, we add new users in NIS instead of locally on each machine (unless absolutely necessary) even if a user might use a specific machine only.

With this, a site that has carefully crafted their profiles and stored in ldap to share among all ca instances don't have to bother changing anything for each ca instance created (because they are default to the ldap). Upgrade will also not affect the ldap copies inadvertently.

...

2. All profiles for an instance should live in LDAP. This makes it simple - no need to check to see if a profile is in LDAP or files or both, and which has priority etc. Tools will be provided to manage/ create/delete profiles etc.

It might be simple, but we need to think about what's the value of them existing in ldap and not being shared by others.

What is the use case for having local profiles that will not or cannot be shared with other replicas? If sharing the profile doesn't pose any problem we might as well store "local" profiles in LDAP so they can be handled more consistently.

...

3. Updates to system profile files will not affect the existing LDAP profiles. We can provide update scripts or manual instructions for admins to run when they opt to do so. This will be for behavioral changes.

so, in the view I specified above, update to the "local" profiles will only affect the instance where they exist, however, we can allow option for admins to "publish" to ldap.

The problem with local profiles in files is that they are not replicated, so in case the machine is hosed, all local settings is gone, including the profiles. Unless the admin created a backup for each machine, it will be difficult to restore the machine quickly. The replicated LDAP profiles will take care of this problem automatically. In other proposals the system profiles in files are read-only and not customizable, so there's no local profiles that need to be backed up. -- Endi S. Dewata

On 7/3/2014 11:18 AM, Christina Fu wrote:

Actually, I did not know that this discussion was restricted to a replicated ("clone") environment. My view was more for a general organizational environment where there are multiple sub-ca's for different departments (which are not necessarily clones, but some could be) and there are two types of profiles: 1. centralized profiles (shared by all) 2. local profiles (customized by each department)

I think this view (let's adopt your term and call it proposal #4) is more flexible in serving both non-clones and clones and yet retains the simplicity.

Thanks for clarifying. Yes, the main purpose of this enhancement is to simplify replicating the profiles within a cluster (e.g. IPA). Proposal #4 is probably meant to be an inter-cluster profile management. Within a cluster itself, the replicas should be indistinguishable. For example: * dept1 (cluster1: server1, server2): profile1, profile2 * dept2 (cluster2: server3, server4): profile1, profile3 * shared: profile1, profile4 I think proposal #4 can be done as a separate enhancement after we address the intra-cluster management (proposal #1-3).

You are right, Endi, about how "local profiles" don't necessarily have to be on the disk. It's just a personal preference that I feel most comfortable editing files directly than using ui's. I have never used the java console to edit profiles. I don't know if there are others who feel the same way. Maybe a market research on administrators is needed here.

Here's the CLI that Fraser proposed: http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage#Edit_profile So instead of executing the following commands on each replica (and risking inconsistent changes): $ vi /var/lib/pki/pki-tomcat/ca/profiles/ca/caUserCert.cfg $ systemctl restart pki-tomcatd@pki-tomcat.service you can call this just once (it will use the same vi editor) from any machine: $ pki <connection/auth params> profile-edit caUserCert and it should automatically propagate the changes to all replica. Same thing with the UI, you'll only need to do the changes once.

...

The problem with local profiles in files is that they are not replicated, so in case the machine is hosed, all local settings is gone, including the profiles. Unless the admin created a backup for each machine, it will be difficult to restore the machine quickly. The replicated LDAP profiles will take care of this problem automatically. In other proposals the system profiles in files are read-only and not customizable, so there's no local profiles that need to be backed up.

If they don't do backups, losing profiles would not be their only issue there.

I think, ideally, if we lose a replica, other replicas should be able to replace it immediately. We still need a backup for the shared data/configuration, but we shouldn't need to backup individual replica. A replica should be something that you can add/remove relatively quickly. So, as a long term goal we should gradually migrate local configuration files into LDAP, starting with the profiles.

But anyway, like I said above, I think putting "local" profiles in the lda is fine. It's just the user experience of administration that has to change. I have no opinion on this other than stating my personal preference of editing files directly ;-)

Is the above CLI a good substitute? -- Endi S. Dewata

Done. https://fedorahosted.org/pki/ticket/1067 thanks, Christina On 07/07/2014 04:44 AM, Ade Lee wrote:

On Thu, 2014-07-03 at 12:59 -0500, Endi Sukma Dewata wrote:

...

On 7/3/2014 11:18 AM, Christina Fu wrote:

...

Actually, I did not know that this discussion was restricted to a replicated ("clone") environment. My view was more for a general organizational environment where there are multiple sub-ca's for different departments (which are not necessarily clones, but some could be) and there are two types of profiles: 1. centralized profiles (shared by all) 2. local profiles (customized by each department)

I think this view (let's adopt your term and call it proposal #4) is more flexible in serving both non-clones and clones and yet retains the simplicity. Thanks for clarifying. Yes, the main purpose of this enhancement is to simplify replicating the profiles within a cluster (e.g. IPA). Proposal #4 is probably meant to be an inter-cluster profile management. Within a cluster itself, the replicas should be indistinguishable.

For example: * dept1 (cluster1: server1, server2): profile1, profile2 * dept2 (cluster2: server3, server4): profile1, profile3 * shared: profile1, profile4

I think proposal #4 can be done as a separate enhancement after we address the intra-cluster management (proposal #1-3).

Agreed. One of the things we would have to address, for instance, would be where the shared profiles would reside. Certainly it has to be a location that is available to all subCA's. Several possibilities come to mind: 1) under a baseDN 2) user-defined 3) in the security domain - particularly useful perhaps for storing customized system profiles to used during installation.

Christina, can you open a ticket for this feature?

...

...

You are right, Endi, about how "local profiles" don't necessarily have to be on the disk. It's just a personal preference that I feel most comfortable editing files directly than using ui's. I have never used the java console to edit profiles. I don't know if there are others who feel the same way. Maybe a market research on administrators is needed here. Here's the CLI that Fraser proposed: http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage#Edit_profile

So instead of executing the following commands on each replica (and risking inconsistent changes):

$ vi /var/lib/pki/pki-tomcat/ca/profiles/ca/caUserCert.cfg $ systemctl restart pki-tomcatd@pki-tomcat.service

you can call this just once (it will use the same vi editor) from any machine:

$ pki <connection/auth params> profile-edit caUserCert

and it should automatically propagate the changes to all replica. Same thing with the UI, you'll only need to do the changes once.

Right, the idea is for the pki utility to spawn an editor -- we would likely default to "vi", but I suppose we could allow the user to specify others too. Ideally, what you would see is exactly the same as if you were editing a file today.

The console would also continue to work, because it interacts with the ProfileSubsystem - which will know how to talk to LDAP.

...

...

...

The problem with local profiles in files is that they are not replicated, so in case the machine is hosed, all local settings is gone, including the profiles. Unless the admin created a backup for each machine, it will be difficult to restore the machine quickly. The replicated LDAP profiles will take care of this problem automatically. In other proposals the system profiles in files are read-only and not customizable, so there's no local profiles that need to be backed up.

If they don't do backups, losing profiles would not be their only issue there. I think, ideally, if we lose a replica, other replicas should be able to replace it immediately. We still need a backup for the shared data/configuration, but we shouldn't need to backup individual replica. A replica should be something that you can add/remove relatively quickly. So, as a long term goal we should gradually migrate local configuration files into LDAP, starting with the profiles.

...

But anyway, like I said above, I think putting "local" profiles in the lda is fine. It's just the user experience of administration that has to change. I have no opinion on this other than stating my personal preference of editing files directly ;-) Is the above CLI a good substitute?

On Mon, Jul 07, 2014 at 09:31:27AM -0700, Christina Fu wrote:

Oh, and Fraser, Ade, and Endi, please keep https://fedorahosted.org/pki/ticket/1067 in mind when you implement the "replication of new/modified profile" so that they can seamlessly co-exist in the future.

Sure thing, I will add a reference to the LDAP Profiles design document. There will be some similar questions as were raised for the hybrid LDAP/files model, i.e. questions of precedence, or whether name collisions are even allowed. I don't see any fundamental issues, though. Cheers, Fraser

thanks, Christina

On 07/07/2014 09:21 AM, Christina Fu wrote:

...

Done. https://fedorahosted.org/pki/ticket/1067

thanks, Christina

On 07/07/2014 04:44 AM, Ade Lee wrote:

...

On Thu, 2014-07-03 at 12:59 -0500, Endi Sukma Dewata wrote:

...

On 7/3/2014 11:18 AM, Christina Fu wrote:

...

Actually, I did not know that this discussion was restricted to a replicated ("clone") environment. My view was more for a general organizational environment where there are multiple sub-ca's for different departments (which are not necessarily clones, but some could be) and there are two types of profiles: 1. centralized profiles (shared by all) 2. local profiles (customized by each department)

I think this view (let's adopt your term and call it proposal #4) is more flexible in serving both non-clones and clones and yet retains the simplicity. Thanks for clarifying. Yes, the main purpose of this enhancement is to simplify replicating the profiles within a cluster (e.g. IPA). Proposal #4 is probably meant to be an inter-cluster profile management. Within a cluster itself, the replicas should be indistinguishable.

For example: * dept1 (cluster1: server1, server2): profile1, profile2 * dept2 (cluster2: server3, server4): profile1, profile3 * shared: profile1, profile4

I think proposal #4 can be done as a separate enhancement after we address the intra-cluster management (proposal #1-3).

Agreed. One of the things we would have to address, for instance, would be where the shared profiles would reside. Certainly it has to be a location that is available to all subCA's. Several possibilities come to mind: 1) under a baseDN 2) user-defined 3) in the security domain - particularly useful perhaps for storing customized system profiles to used during installation.

Christina, can you open a ticket for this feature?

...

...

You are right, Endi, about how "local profiles" don't necessarily have to be on the disk. It's just a personal preference that I feel most comfortable editing files directly than using ui's. I have never used the java console to edit profiles. I don't know if there are others who feel the same way. Maybe a market research on administrators is needed here. Here's the CLI that Fraser proposed: http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage#Edit_profile

So instead of executing the following commands on each replica (and risking inconsistent changes):

$ vi /var/lib/pki/pki-tomcat/ca/profiles/ca/caUserCert.cfg $ systemctl restart pki-tomcatd@pki-tomcat.service

you can call this just once (it will use the same vi editor) from any machine:

$ pki <connection/auth params> profile-edit caUserCert

and it should automatically propagate the changes to all replica. Same thing with the UI, you'll only need to do the changes once.

Right, the idea is for the pki utility to spawn an editor -- we would likely default to "vi", but I suppose we could allow the user to specify others too. Ideally, what you would see is exactly the same as if you were editing a file today.

The console would also continue to work, because it interacts with the ProfileSubsystem - which will know how to talk to LDAP.

...

...

...

The problem with local profiles in files is that they are not replicated, so in case the machine is hosed, all local settings is gone, including the profiles. Unless the admin created a backup for each machine, it will be difficult to restore the machine quickly. The replicated LDAP profiles will take care of this problem automatically. In other proposals the system profiles in files are read-only and not customizable, so there's no local profiles that need to be backed up.

If they don't do backups, losing profiles would not be their only issue there. I think, ideally, if we lose a replica, other replicas should be able to replace it immediately. We still need a backup for the shared data/configuration, but we shouldn't need to backup individual replica. A replica should be something that you can add/remove relatively quickly. So, as a long term goal we should gradually migrate local configuration files into LDAP, starting with the profiles.

...

But anyway, like I said above, I think putting "local" profiles in the lda is fine. It's just the user experience of administration that has to change. I have no opinion on this other than stating my personal preference of editing files directly ;-) Is the above CLI a good substitute?

_______________________________________________ Pki-devel mailing list Pki-devel@redhat.com https://www.redhat.com/mailman/listinfo/pki-devel

_______________________________________________ Pki-devel mailing list Pki-devel@redhat.com https://www.redhat.com/mailman/listinfo/pki-devel

IMHO, I think #3 is way too complicated. Complication invites issues and confuse people. Could we step back and try something simpler? When you copy the content of one profile and modify it to create a new one, then it's a new profile standing on its own. Why the parent-child relationship and all? Seems like an administrator's nightmare. Maybe I missed out on the irc discussion, but could you please give us a summary of the benefit and how the benefit weights against development time and administration maintenance, and support effort in the future on our end? Anyway, I hope you will consider what I said in my earlier response. I thought our goal was to provide a "centralized collection of profiles" to ease administration effort. I hope we achieve simplicity rather than create complication. It's just my personal preference. thanks, Christina On 07/02/2014 02:02 PM, Endi Sukma Dewata wrote:

On 7/2/2014 12:17 PM, Ade Lee wrote:

...

The problem with the above proposal is one of migration. Customers are used to both creating new profiles and modifying existing profiles to suit their needs. If we go along with this mechanism, we would have to tell these customers that they would need to rename their existing customized profiles (and update all the relevant clients etc.)

Just to clarify, so there are 3 proposals: 1. Stacked profiles (Fraser's proposal) 2. Pure LDAP profiles (Ade's proposal) 3. Separate file/LDAP profiles (my proposal)

As discussed over IRC, proposal #3 will be supplemented with a mechanism to allow creating aliases for backward compatibility. I'm trying to generalize the mechanism to become "profile inheritance".

Basically each LDAP profile will have an optional parent. The parent can be the file-based system/default profile, or another LDAP profile. A sub-profile will inherit all attributes, except when it's explicitly declared in the sub-profile. This mechanism allows us to create just a proxy/alias, a full clone, or anything in between. For example, a proxy profile might only have a few attributes:

dn: cn=caAdminCert,ou=Profiles,ou=CA,{suffix} objectClass: certProfile cn: caAdminCert parent: defaultAdminCert visible: true

Let's say in the old system we have an unchanged caAdminCert and a customized caUserCert. With proposal #1 we'll have: 1. caAdminCert (file) 2. caUserCert (LDAP) 3. caUserCert (file, inaccessible via REST due to the LDAP entry)

With proposal #2 we'll have: 1. caAdminCert (LDAP) 2. caUserCert (LDAP) 3. caAdminCert (file, for initial installation only) 4. caUserCert (file, for initial installation only)

With proposal #3 we'll have: 1. caAdminCert -> defaultAdminCert (proxy) 2. caUserCert (LDAP) 3. defaultAdminCert (file) 4. defaultUserCert (file)

The benefits of proposal #3 are:

1. The system/default profiles will be stored in /usr/share/pki. People will be naturally discouraged to change them since it will be overwritten during RPM upgrade. On the other hand, an LDAP profile, although we can flag it as system profile or make it read-only, is more tempting to change.

2. The original/unchanged system profiles will be accessible via REST as well so people can use it directly and rely on auto update.

3. Backward compatibility can be maintained with proxy profiles. These profiles will be updated automatically during RPM upgrade since it's pointing to the file-based system profiles. No need to write upgrade scripts.

4. The proxy profiles can be used to hide certain system profiles without creating a full clone. With stacking, to hide caAdminCert we probably would have to create a full clone then change the visibility, and lose the auto update ability.

5. The mechanism can be used to extend/change the behavior of the default profiles without having to create the full custom profile. For example we want to create IPA-specific caUserCert with different inputs. In this case we'll be able to just create a sub-profile with the new inputs. No need to clone the entire profile.

...

There is another problem, and that is that it is not clear that we want updates to the default profiles to be propagated to existing instances. I have looked at the profiles and there have been only a handful of changes over the last 7 years. Those changes include things like updating the default signing algorithms or the default validity. More likely than not, admins would prefer that we not change the behavior of profiles in existing instances underneath them.

The changes that I have found are all behavioral - and therefore things that admin can opt out of -- or would prefer to do on their own schedule. There have been no structural changes.

If there are structural changes, then we need to (and can) provide an upgrade script which would run with the automatic upgrade. An example of this would be a schema upgrade as we sort out how to represent profiles in LDAP.

As discussed, we're separating database changes into: 1. structural changes (e.g. schema change, adding/removing attributes) 2. behavioral changes (e.g. changing SHA1 to SHA512, default validity)

The structural changes will be semi-automatic. The admin decides when to run them, but it's not necessary to review them one-by-one. The behavioral changes should be reviewed by a security expert to make sure that it won't affect their particular system negatively.

With proposal #3 the admin can decide whether to (a) trust us (developers) to make the behavioral changes automatically by using the default profiles directly or using proxy profiles, or (b) maintain a full control of all behavioral changes using custom profiles. If all profiles are in LDAP they might be limited to option (b) because the upgrade script won't be able to tell whether the current value is a default value that we can change, or it was intentionally set by the admin.

Yes, deferring profile inheritance for later discussion would be the right approach at this time. thanks, Christina On 07/02/2014 07:51 PM, Endi Sukma Dewata wrote:

On 7/2/2014 4:42 PM, Christina Fu wrote:

...

IMHO, I think #3 is way too complicated. Complication invites issues and confuse people.

Could we step back and try something simpler? When you copy the content of one profile and modify it to create a new one, then it's a new profile standing on its own. Why the parent-child relationship and all? Seems like an administrator's nightmare. Maybe I missed out on the irc discussion, but could you please give us a summary of the benefit and how the benefit weights against development time and administration maintenance, and support effort in the future on our end?

Proposal #3 can be simplified into #3(a) without profile inheritance. Here we'll support just the file-based system profile, proxy LDAP profile, and custom LDAP profile. For immediate purposes this should be sufficient.

Profile inheritance is an idea that just came up after the IRC discussion. It would take a whole separate design page to see how it works, the potential benefits, and the impact on us. We don't have to implement this now, but later if we determine that it is useful, we can implement #3(b) with profile inheritance without changing the schema or the existing data.

...

Anyway, I hope you will consider what I said in my earlier response. I thought our goal was to provide a "centralized collection of profiles" to ease administration effort. I hope we achieve simplicity rather than create complication. It's just my personal preference.

Not disagreeing with that. Simplicity is always a goal, but sometimes the proper solution cannot be the simplest one.