Hi, This is the initial draft of the TPS REST interface design: http://pki.fedoraproject.org/wiki/TPS_REST_Interface Please take a look and let me know if you have any comments. Thanks!

On 6/27/2013 10:20 AM, Ade Lee wrote: > 1. One of the decisions you made was to allow admins, agents and > operators to access the same resources. So, unlike the rest of the java > subsystems, we will have /tps/rest/tokens as opposed > to /tps/rest/agent/tokens or /tps/rest/admin/tokens > > One reason one might want to add /agent or /admin is if the content of > GET /tps/tokens/X is different for admins/agents/operators. > > So what is the difference between op=search and op=search_admin, op=view > and op=view_admin, op=show and op=show_admin? etc. I'm still rebuilding my TPS environment so I can't check this myself right now, but here are the screenshots from the docs and they look pretty much identical: https://access.redhat.com/site/documentation/en-US/Red_Hat_Certificate_Sy... https://access.redhat.com/site/documentation/en-US/Red_Hat_Certificate_Sy... https://access.redhat.com/site/documentation/en-US/Red_Hat_Certificate_Sy... > Is there any case where those differences need be preserved? Even if there are differences, since they are the same resources (i.e. tokens) most of the visible attributes will be the same. If there are differences between the roles, it will be handled by returning the common attributes plus the attributes visible to that role only. When creating the REST data structure we combine all possible attributes.

> 2. You use PUT /tps/rest/tokens/X as the operation to modify tokens. > Usually REST semantics imply that the resource passed in to the PUT > operation essentially replace the resource at the server. > > Is that your intention? I'm guessing that the resource that would be > passed in is the token object returned in GET /tps/rest/tokens/foo. > It would help if you defined what a token resource was. Any REST resource is a partial representation of the actual object/service running on the server, meaning it may not have the full information to reconstruct the object on the server-side. So even if we PUT a resource it should be clear that we're not completely replacing the actual object on the server, but we're updating the resource with the attributes provided in the request. In case of token resource, the resource consists of the attributes returned by the GET operation which may not include everything a token has. When modifying the resource using PUT some of these attributes will be ignored because they are read-only (e.g. date created/modified), but we can still PUT a resource that we obtained earlier using GET back to the server to update some attributes.

> Or is your intention to send in something with an "action" parameter - > in which case, the correct operation is a POST? I use 'action' in the modify token operation because I couldn't find a better replacement for the 'question' parameter in the original op=do_token. Ideally it should be called 'status' but there's already another attribute with that name. Any suggestion? This 'action' should actually be returned by the GET operation too. I'll add it after we finalize the name.

> 3. Should we be worried about a XSS attack here for modifying the state > of the token? My guess is that we need to take advantage of the nonce > mechanism, in which case, token state modification will require two > steps. Yes, but similar to our discussion in the past, nonces or ETags can be added in a later phase without changing the interface. Since the modification is a PUT operation, and the request attributes are obtained using a GET operation done earlier, this is a two-step operation. Without the nonce or ETag the GET operation will be optional, and the client can construct the request from scratch. But later the PUT operation can require a nonce or ETag from a previous GET operation.

> 4. You should also note that we will be sending back BAD_REQUEST (401?) > when the token state transition is not permitted. That should fall under "Normal application errors will return HTTP 4xx return code." We can add more details as we implement it.

> 5. In the response to PUT /tokens/X, it is not necessary to return the > state of the token. Rather, you should return a URL pointer to the > newly modified resource. The same comment applies to the other PUT > operations as well. Since PUT operation in general doesn't create a new resource, and also the request is sent to the resource URL being modified, there is no need to return a new location because it will be identical. In this design the new location will only be returned on POST operations when adding a new resource, which in this case the new resource location will be different from the POST target (the resource collection).

> 6. Same question about XSS for add/remove token/ add/remove user. Similarly, nonces or ETags can be added later. As described earlier, modifying users is a two-step process too with GET and PUT. Add and remove operations do not require the client to do any other operation, but it can be made so without changing the interface. The DELETE can require a GET, but I'm not sure what operation should be required before add.

> 7. Note that the difference between op=view_activity, > op=view_activity_all etc. is in the types of activities that are > visible. This should be handled seamlessly in the logic used to select > activity records from the db. Yes, the interface will be the same, but depending on which roles you belong to you may get different results. > 8. For the configuration items - audit config, profiles, profile > mappings, connections, authenticators, I wonder if it makes things > clearer to put them under a config bucket .. like /tps/rest/config/audit > for example. I'd rather not do that. For now I'm using /tps/rest/config to store general settings. If there are resource-specific settings they should go to that resource subtree. For example, suppose later we want to provide interface to view audit logs, it will be stored under /tps/rest/audit/logs. The audit config can be stored in /tps/rest/audit, or maybe /tps/rest/audit/config. Imagine you're an admin looking at a web page showing the audit logs. If you want to change the audit settings you'd want to be able to click a button on that page directly. Having to go to config -> audit menu will be less intuitive. Not that this is relevant to determine the actual resource URL, but with the same principles the config should be located near the entries of that resource. It's also possible to provide two interfaces at both locations for the same resources, but there's no real need to do that now. For the other resources mentioned above, they are both the entries and the config. If we put them under config there wouldn't many (or any) things left at the top level. Even system users can also be considered a config, but unless there's a strong reason I prefer to keep them where they are now.

> 9. Is there a mapping between Profile and Profile Mapping operations and > the old operations? Please put that in so we can see whether we have > complete coverage. In particular, I do not see an operation to > approve/enable a profile. This is important because we have Common > Criteria requirements that two users are involved in the approval of a > profile. In our case, IIRC we implemented it such that an admin can > configure a profile but an agent must approve it. There should be, but they are not in the docs and I have not had a chance to test all possible operations in the UI. Once I get the TPS instance back running I'll continue this. But in general we can implement something similar to the cert requests: /tps/rest/profiles/{id}/approve (all approvers will use the same URL) /tps/rest/profiles/{id}/enable

> 10. In POST /tps/rest/profilemappings, you return the location of the > profile mappings as well as the contents of the profile mapping > resource. You should just return the location. In fact, its probably > better to just return locations in general. The client would then use > GET to fetch the details if needed. This comment applies to many of the > resources. In general I'd rather return the resource attributes in the POST response than requiring the client to do a separate GET later. This response acts as a receipt/acknowledgement for the add request, and it will return the actual attributes stored on the server before any other operation is done on it. This is rather important because sometimes a lazy UI won't do an extra GET because it assumes the attributes don't change, where in fact they could be normalized, ignored, or generated by the server. It also prevents a possible (although unlikely) attack that inserts an operation between the POST and GET.

> 11. In the PUT /tps/rest/config, we have requirements for having than > one user to approve changes. Admins make changes and agents approve > them if I recall correctly. You should look at the differences between > the agent and admin pages. I'll take a look again after I have the TPS instance back. Is there any documents or diagrams showing the workflow of all approval processes in TPS UI? Thanks.

On 6/28/2013 9:36 AM, Ade Lee wrote: >>> Or is your intention to send in something with an "action" parameter - >>> in which case, the correct operation is a POST? >> >> I use 'action' in the modify token operation because I couldn't find a >> better replacement for the 'question' parameter in the original >> op=do_token. Ideally it should be called 'status' but there's already >> another attribute with that name. Any suggestion? This 'action' should >> actually be returned by the GET operation too. I'll add it after we >> finalize the name. > > do_token?question=X is essentially an operation where you are attempting > to set the state of the token resource - for example, from "active" to > "lost". There is no need for an additional parameter. I added a Change Token State operation: http://pki.fedoraproject.org/wiki/TPS_REST_Interface#Change_Token_State As discussed, for now we'll call this parameter 'next state', which will update the status & reason depending on the current state of the token. >>> 3. Should we be worried about a XSS attack here for modifying the state >>> of the token? My guess is that we need to take advantage of the nonce >>> mechanism, in which case, token state modification will require two >>> steps. >> >> Yes, but similar to our discussion in the past, nonces or ETags can be >> added in a later phase without changing the interface. Since the >> modification is a PUT operation, and the request attributes are obtained >> using a GET operation done earlier, this is a two-step operation. >> Without the nonce or ETag the GET operation will be optional, and the >> client can construct the request from scratch. But later the PUT >> operation can require a nonce or ETag from a previous GET operation. > > OK, please put the nonce in the design. We dont want to forget it. I added two sections related to this: http://pki.fedoraproject.org/wiki/TPS_REST_Interface#Concurrency_Control http://pki.fedoraproject.org/wiki/TPS_REST_Interface#Vulnerabilities >>> 4. You should also note that we will be sending back BAD_REQUEST (401?) >>> when the token state transition is not permitted. >> >> That should fall under "Normal application errors will return HTTP 4xx >> return code." We can add more details as we implement it. > > Right, I'm just pointing this out because in this case, there is a state > machine which determines which operations are successful. From the > point of view of the API, doing a token transition is just an attempt to > set the state of the token differently, but unlike something simple like > just changing an attribute value, the operation is subject to the > results of a state machine. I noted that in the response of Change Token State: http://pki.fedoraproject.org/wiki/TPS_REST_Interface#Change_Token_State >>> 5. In the response to PUT /tokens/X, it is not necessary to return the >>> state of the token. Rather, you should return a URL pointer to the >>> newly modified resource. The same comment applies to the other PUT >>> operations as well. >> >> Since PUT operation in general doesn't create a new resource, and also >> the request is sent to the resource URL being modified, there is no need >> to return a new location because it will be identical. In this design >> the new location will only be returned on POST operations when adding a >> new resource, which in this case the new resource location will be >> different from the POST target (the resource collection). >> > Agreed. But in PUT /tokens/X, you return state information. This is > not necessary. You should probably return nothing - other than status. See explanation in #10. >>> 6. Same question about XSS for add/remove token/ add/remove user. >> >> Similarly, nonces or ETags can be added later. As described earlier, >> modifying users is a two-step process too with GET and PUT. Add and >> remove operations do not require the client to do any other operation, >> but it can be made so without changing the interface. The DELETE can >> require a GET, but I'm not sure what operation should be required before >> add. >> > Yeah - we need to think about ADD. Being able to add a privileged user > for example by XSS would be troubling. I think we're mixing up CSRF and XSS. Nonces will be useful to prevent CSRF. XSS can be prevented by encoding/escaping the values. If we implement a single-page Web UI (like IPA Web UI) the client can obtain a nonce during login, then the UI will keep the nonce in memory throughout the session and use it for all update requests including add (the nonce should really be called CSRF token). CSRF attack usually will open a new page, so it will not be able to know the nonce. If we implement a multi-page Web UI (like the current Dogtag UI), the add operation can be done in 2 steps. The first page will generate the nonce, then the second page will execute the operation. >>> 9. Is there a mapping between Profile and Profile Mapping operations and >>> the old operations? Please put that in so we can see whether we have >>> complete coverage. In particular, I do not see an operation to >>> approve/enable a profile. This is important because we have Common >>> Criteria requirements that two users are involved in the approval of a >>> profile. In our case, IIRC we implemented it such that an admin can >>> configure a profile but an agent must approve it. >> >> There should be, but they are not in the docs and I have not had a >> chance to test all possible operations in the UI. Once I get the TPS >> instance back running I'll continue this. But in general we can >> implement something similar to the cert requests: >> >> /tps/rest/profiles/{id}/approve (all approvers will use the same URL) >> /tps/rest/profiles/{id}/enable >> > Thats fine - lets make sure we get those operations in. I added the missing mappings. I also added a Change Profile State operation: http://pki.fedoraproject.org/wiki/TPS_REST_Interface#Change_Profile_State >>> 10. In POST /tps/rest/profilemappings, you return the location of the >>> profile mappings as well as the contents of the profile mapping >>> resource. You should just return the location. In fact, its probably >>> better to just return locations in general. The client would then use >>> GET to fetch the details if needed. This comment applies to many of the >>> resources. >> >> In general I'd rather return the resource attributes in the POST >> response than requiring the client to do a separate GET later. This >> response acts as a receipt/acknowledgement for the add request, and it >> will return the actual attributes stored on the server before any other >> operation is done on it. This is rather important because sometimes a >> lazy UI won't do an extra GET because it assumes the attributes don't >> change, where in fact they could be normalized, ignored, or generated by >> the server. It also prevents a possible (although unlikely) attack that >> inserts an operation between the POST and GET. > > I guess you are suggesting the same for PUT requests as above. > I'm not sure whether this is vaid reasoning or not. Just because you > see what has been posted (or PUT) in the server does not mean that this > data has not been changed the next time the data is accessed. There's never any guarantee that the data we get (either from PUT, POST, or even GET) has not been changed when we submit it for another update operation. The advantage of returning the data in PUT/POST response is we're pushing the updated data to the client's feet. > If anything, returning this data encourages lazy clients. On the contrary, by returning the data in the PUT/POST response the client will have little reason not to use it. If the client has no intention to get the new data anyway (either from PUT/POST response or by doing another GET) there's nothing we can do about it. But if the client wants to get the new data, using PUT/POST response is a much cheaper option. Compare the following code: user = users.update(user); with this: users.update(user); user = users.get(user.getId()); > The fact is > though that we will have to implement nonces and possibly etags, and so > GETs will become mandatory for any changes. Not necessarily. ETag is only an optimistic lock; it's not a guarantee that the data won't change. The PUT/POST response will also provide ETag. Also see comment #6, we may be able to reuse the nonce. If there's nobody else updating the same data, we could reuse the PUT/POST response and the ETag that came with it to make subsequent update. So we're still using a two-step update process, but the 2nd step of the first update may become the 1st step of the second update. See also the Concurrency Control section in the wiki page. Imagine this, you updated a user, then you went to do some other operations. Then you came back to this user and edited it again. If there's nobody else changing the user, then the new data and ETag that you got from the first update would still be valid. But if first update happened a long time ago, the client may decide according to the caching policy that the data has expired and then issue a GET to get a newer data and ETag. >>> 11. In the PUT /tps/rest/config, we have requirements for having than >>> one user to approve changes. Admins make changes and agents approve >>> them if I recall correctly. You should look at the differences between >>> the agent and admin pages. >> >> I'll take a look again after I have the TPS instance back. Is there any >> documents or diagrams showing the workflow of all approval processes in >> TPS UI? Thanks. >> > The CC docs are probably the best bet for that (other than looking at > the code). The only approval process I saw in the UI is for profiles, not for general config. Could you point me to a specific page in the docs? Thanks.

On 6/29/2013 2:16 PM, Ade Lee wrote: > One thing I just noticed though is the operation to add a token. > You have POST /tokens passing in the tokenId. > > POST /tokens is only used when creating a resource when the URL of the > resource is unknown -- ie. it will be specified by the server. An > example of this is POST /certrequests which returns the URL of the > request /certrequests/0xf00 where 0xf00 is the request ID as assigned by > the server. That is one way to use POST, but in general it doesn't always have to be that limited. Consider the ID as an optional parameter. You can specify it, but it's not specified the server could generate a new ID automatically (not that we want to do that for this case). For the client it would be easier to use the same operation to create a new entry, with or without the ID. The operation that will work for both cases is POST-ing into the collection. I don't think this is violating any REST guidelines. It's also consistent with the add operation for the existing user and group resources.

> In this case, the token ID is what is being passed in. So we know the > resultant URL when the resource is created. There are two options > instead: > > a) PUT /tokens/tokenID > b) POST /tokens/tokenID > > My preference is a) but of course the advantage of (b) is that there is > a distinction between an ADD and a modify operation. > > The same comment applies to all the ADD operations in the design. The problem with (a) is that without ETag there's a risk overwriting an existing token, and there's no warning or error message when that happens. While ETag is desired, it should be optional (can we guarantee all clients will support ETag?) and may not get implemented right away. If we POST to the collection the server can reject it if the entry already exists. Also, naturally an add operation is not idempotent, so we should not use PUT.

Option (b) is a little weird because usually we expect the resource would already exist when we POST. POST-ing to the collection is fine because the collection always exists.

The REST guidelines are precisely that - guidelines - and we can choose to ignore them at our discretion. But there are distinct advantages to following them. In particular, when doing so, anyone looking at the API knows exactly what is going on. By using PUT, its clear that the client is responsible for providing the resource ID. REST basically says to use the HTTP spec to determine what operation to use. To see what the REST guideline would be in this case, lets see what the HTTP spec says about PUT: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.6 The PUT method requests that the enclosed entity be stored under the supplied Request-URI. If the Request-URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server. If the Request-URI does not point to an existing resource, and that URI is capable of being defined as a new resource by the requesting user agent, the origin server can create the resource with that URI. If a new resource is created, the origin server MUST inform the user agent via the 201 (Created) response. If an existing resource is modified, either the 200 (OK) or 204 (No Content) response codes SHOULD be sent to indicate successful completion of the request.

The POST method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI in the Request-Line. POST is designed to allow a uniform method to cover the following functions: - Annotation of existing resources; - Posting a message to a bulletin board, newsgroup, mailing list, or similar group of articles; - Providing a block of data, such as the result of submitting a form, to a data-handling process; - Extending a database through an append operation. The actual function performed by the POST method is determined by the server and is usually dependent on the Request-URI. The posted entity is subordinate to that URI in the same way that a file is subordinate to a directory containing it, a news article is subordinate to a newsgroup to which it is posted, or a record is subordinate to a database.

The point you make about possibly overwriting an existing resource is valid -- but the answer there is etags. And as e will be writing the first clients for this, we can safely say that - yes - all clients will support etags.

> 1. Considering all the requirements, should CREATE and UPDATE be > handled by a single or separate methods? Because you only want admins to be able to create a user, and non-admins can update users, you should probably use the @Post method to create, and not use @Put for creation. @POST @Path("/users") create(); for @PUT @Path("/users/{id}") update(); you should throw a 404 if the user doesn't exist.

> Suppose the security requirements are the same for both operations, > would you design it differently? All depends whether or not I wanted to rely on the client to pick the URI or not. I usually use the @POST /users because it gives the server a lot of flexibility on what the returned URI (in the Location header) looks like.

Just because we may not implement etags in the first iteration does not mean that we should use non-standard API in our design.

> 3. Is it OK to require the use of ETag to distinguish CREATE and UPDATE > (requiring a GET operation before every UPDATE)? Or should ETag be > optional and used only if the system requires caching or concurrency > control? It is never OK to override the meaning of a header, HTTP method, or response code. One of the nice things about REST is that you have a solid foundation of how a service behaves without ever looking at the documentation of that service: This is because you know the HTTP protocol. ETag should only be used for conditional GET (when a cache is involved) or a conditional PUT, or POST (When concurrency is involved).

> I also added a Change Profile State operation: > http://pki.fedoraproject.org/wiki/TPS_REST_Interface#Change_Profile_State > The way this is written, it looks like you are just doing a POST to the /tps/rest/profiles/<ProfileID> and passing in "action" as a parameter. Thats not very RESTful at all. I think we want: /tps/rest/profiles/{id}/{action} where {action} is approve etc. This is also consistent with how we have done this for cert-requests etc. as well.

The URI in a POST request identifies the resource that will handle the enclosed entity. That resource might be a data-accepting process, a gateway to some other protocol, or a separate entity that accepts annotations.

On 7/1/2013 11:51 AM, Endi Sukma Dewata wrote: > From http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.5: >> The POST method is used to request that the origin server accept the >> entity enclosed in the request as a new subordinate of the resource >> identified by the Request-URI in the Request-Line. POST is designed >> to allow a uniform method to cover the following functions: >> >> - Annotation of existing resources; >> - Posting a message to a bulletin board, newsgroup, mailing list, >> or similar group of articles; >> - Providing a block of data, such as the result of submitting a >> form, to a data-handling process; >> - Extending a database through an append operation. >> >> The actual function performed by the POST method is determined by the >> server and is usually dependent on the Request-URI. The posted entity >> is subordinate to that URI in the same way that a file is subordinate >> to a directory containing it, a news article is subordinate to a >> newsgroup to which it is posted, or a record is subordinate to a >> database. > > Adding a new user or profile into a collection would be analogous to the > above examples. Just to clarify, when adding a file into a directory we know the file name already. Still, this is given as an example of a POST operation. So, knowing the ID ahead of time is irrelevant to picking PUT vs. POST to create a resource.