include more parts of old operator manual and add more details screenshots for published data

.github/mkdocs/mkdocs.yaml

@@ -12,18 +12,20 @@ nav:
       - Dashboard: login/Dashboard.md
     - Datasets:
       - datasets/index.md
-      - Register DOIs: datasets/Publishing.md
-    - Proposals: proposals.md
-    - Samples: samples.md
+      - Publishing data: datasets/Publishing.md
+      - Publishing data Advanced: datasets/PublishingAdvanced.md
+    - Proposals: 
+      - proposals/index.md
+    - Samples: samples/index.md
     - Instruments: instruments.md
     - Troubleshooting:
       - troubleshoot/index.md
 
   - SciCat Operator Guide: 
     - operator-manual/index.md
-    - sites/DESY/index.md
     - swagger/index.md
     - backendconfig/index.md
+    - backendconfig/authorization/index.md
     - backendconfig/dois.md
 
 

docs/backendconfig/authorization/authorization.md

@@ -0,0 +1,69 @@
+# Authorization
+### Permission settings or who can do what?
+
+SciCat backend v4.x relies on [CASL](https://casl.js.or) to manage permissions.  
+The default vanilla installation of the backend is configured with the permissions described and linked below.   
+To avoid confusion and clarify the terminology used below, the term _User_ indicates a normal authenticated user with no elevated permissions, while _Admin_ indicates any user who belongs to a group that it is listed in the environmental variable ADMIN_GROUPS.  
+By default ADMIN_GROUPS is set to groups: admin, ingestor, archivemanager.
+Special case is for deleting items in SciCat. Users with groups listed in DELETE_GROUPS, are allowed to perform delete. Default value is archivemanager.
+
+___IMPORTANT___ In v3.x, permissions were managed through roles. In v4.x, roles are not used, and they are converted to user groups.
+
+In the vanilla installation, the default functional accounts are assigned to groups as follow:
+- user: admin  
+  group: admin  
+
+- user: ingestor  
+  group: ingestor  
+
+- user: archiveManager  
+  group: archivemanager
+
+This allow for the flexibility required by many installations in different facilities with different needs.  
+
+
+## Group Lists available in Vanilla Configuration 
+The permissions in the vanilla installation provides a set of user groups which acquires specific set of permissions. In order to assign a set of permissions to a specific group of user, add such group to the correct list indicated below.
+
+| Configuration Group List | Description | CASL ability actions |
+| ------------------------ | ----------- | ------------------- |
+| _authenticated users_ | Authenticated users can view/access all datasets that belong to one of the groups they belong to |  DatasetReadOwn |
+| | Users can view attachments for datasets belonging to one of their group |  DatasetAttachmentReadOwn  |
+| | Users are allowed to view origdatablocks for datasets belonging to one of their group | DatasetOrigdatablockReadOwn |
+| | Users are allowed to view datablocks for datasets belonging to one of their group |  DatasetDatablockReadOwn |
+| | Users can view the logbook of the datasets that belong to one of their group | DatasetLogbookReadOwn |
+| | |
+| CREATE_DATASET_GROUPS | Users of the listed groups can create and modify datasets for any of the groups they belong to. At creation time, the system assignes a pid to the new datasets. If the user assigns one, the system will ignore it. |  DatasetCreateOwn , DatasetReadOwn , DatasetUpdateOwn |
+| | Users are allowed to perform all operations on attachments for datasets belonging to one of their group | DatasetAttachmentCreateOwn , DatasetAttachmentReadOwn , DatasetAtatchementUpdateOwn , DatasetAttachmentDeleteOwn  |
+| | Users are allowed to create and update origdatablocks for datasets belonging to one of their group | DatasetOrigdatablockCreateOwn , DatasetOrigdatablockReadOwn , DatasetOrigdatablockUpdateOwn |
+| | Users are allowed to create and update datablocks for datasets belonging to one of their group | DatasetDatablockCreateOwn , DatasetDatablockReadOwn , DatasetDatablockUpdateOwn |
+| | Users can view the logbook of the datasets that belong to one of their group | DatasetLogbookReadOwn |
+| | |
+| CREATE_DATASET_WITH_PID_GROUPS | Users of the listed groups can create and modify datasets for any of the groups they belong to. They are allowed to specify the dataset pid. If they decided not to specify a pid, the system will assign one. | DatasetCreateOwn , DatasetReadOwn , DatasetUpdateOwn |
+| | Users are allowed to perform all operations on attachments for datasets belonging to one of their group | DatasetAttachmentCreateOwn , DatasetAttachmentReadOwn , DatasetAtatchementUpdateOwn , DatasetAttachmentDeleteOwn  |
+| | Users are allowed to create and update origdatablocks for datasets belonging to one of their group | DatasetOrigdatablockCreateOwn , DatasetOrigdatablockReadOwn , DatasetOrigdatablockUpdateOwn |
+| | Users are allowed to create and update datablocks for datasets belonging to one of their group | DatasetDatablockCreateOwn , DatasetDatablockReadOwn , DatasetDatablockUpdateOwn |
+| | Users can view the logbook of the datasets that belong to one of their group | DatasetLogbookReadOwn |
+| | |
+| CREATE_DATASET_PRIVILEGED_GROUPS | Users of the listed groups can create datasets for any group, but can only modify datasets belong to one of the group they belong to. They are allowed to specify pids for new datasets. This settings are suggested for ingestion functional accounts | DatasetCreateAll , DatasetReadOwn , DatasetUpdateOwn |
+| | Users are allowed to perform all operations on attachments for datasets belonging to one of their group | DatasetAttachmentCreateOwn , DatasetAttachmentReadOwn , DatasetAtatchementUpdateOwn , DatasetAttachmentDeleteOwn  |
+| | Users are allowed to create origdatablocks for any datasets, but can only update them for datasets belonging to one of their group | DatasetOrigdatablockCreateAny , DatasetOrigdatablockReadOwn , DatasetOrigdatablockUpdateOwn |
+| | Users are allowed to create and update datablocks for datasets belonging to one of their group | DatasetDatablockCreateOwn , DatasetDatablockReadOwn , DatasetDatablockUpdateOwn |
+| | Users can view the logbook of the datasets that belong to one of their group | DatasetLogbookReadOwn |
+| | |
+| ADMIN_GROUPS | Users of the listed groups can create and modify datasets belonging to any group. They are allowed to specify the dataset's pid at creation time | DatasetCreateAny , DatasetReadAny , DatasetUpdateAny |
+| | Users are allowed to perform all operations on attachments for any datasets | DatasetAttachmentCreateAny , DatasetAttachmentReadAny , DatasetAtatchementUpdateAny , DatasetAttachmentDeleteAny  |
+| | Users are allowed to perform all operations on origdatablocks for any datasets, except delete | DatasetOrigdatablockCreateAny , DatasetOrigdatablockReadAny , DatasetOrigdatablockUpdateAny |
+| | Users are allowed to perform all operations on datablocks for any datasets, except delete | DatasetDatablockCreateAny , DatasetDatablockReadAny , DatasetDatablockUpdateAny |
+| | Users can view logbook for any datasets| DatasetLogbookReadAny | 
+| | |
+| DELETE_GROUPS | Users whose group is listed here are allowed to delete datasets, origdatablock or datablock | DatasetDeleteAny , DatasetOrigdatablockDeleteAny , DatasetDatablockDeleteAny |
+
+## Subsystems
+- [Datasets](./authorization_datasets.md)
+- [OrigDatablocks](./authorization_origdatablocks.md)
+- [Jobs](./authorization_jobs.md)
+- [Users](./authorization_users.md)
+
+___N.B.___: we know that many subsystems are still missing. We are working on reviewing the authorization model for each one of them and producing the relative documentation. We welcome any contribution.
+

docs/backendconfig/authorization/authorization_datasets.md

@@ -0,0 +1,155 @@
+# Datasets Authoorization
+## CASL ability actions
+This is the list of the permissions methods available for datasets and all their endpoints and more fine-grained instance authorization. 
+
+
+### Endpoint authorization
+1. DatasetCreate
+2. DatasetRead
+- DatasetUpdate
+- DatasetDelete
+- DatasetAttachmentCreate
+- DatasetAttachmentRead
+- DatasetAttachmentUpdate
+- DatasetAttachmentDelete
+- DatasetOrigdatablockCreate
+- DatasetOrigdatablockRead
+- DatasetOrigdatablockUpdate
+- DatasetOrigdatablockDelete
+- DatasetDatablockCreate
+- DatasetDatablockRead
+- DatasetDatablockUpdate
+- DatasetDatablockDelete
+- DatasetLogbookRead
+### Instance authorization
+1. DatasetCreateOwnerNoPid
+2. DatasetCreateOwnerWithPid
+- DatasetCreateAny
+- DatasetReadManyPublic
+- DatasetReadManyAccess
+- DatasetReadManyOwner
+- DatasetReadOnePublic
+- DatasetReadOneAccess
+- DatasetReadOneOwner
+- DatasetReadAny
+- DatasetUpdateOwner
+- DatasetUpdateAny
+- DetasetDeleteOwner
+- DatasetDeleteAny
+- DatasetAttachmentCreateOwner
+- DatasetAttachmentCreateAny
+- DatasetAttachmentReadPublic
+- DatasetAttachmentReadAccess
+- DatasetAttachmentReadOwner
+- DatasetAttachmentReadAny
+- DatasetAtatchementUpdateOwner
+- DatasetAtatchementUpdateAny
+- DatasetAttachmentDeleteOwner
+- DatasetAttachmentDeleteAny
+- DatasetOrigdatablockCreateOwner
+- DatasetOrigdatablockCreateAny
+- DatasetOrigdatablockReadPublic
+- DatasetOrigdatablockReadAccess
+- DatasetOrigdatablockReadOwner
+- DatasetOrigdatablockReadAny
+- DatasetOrigdatablockUpdateOwner
+- DatasetOrigdatablockUpdateAny
+- DatasetOrigdatablockDeleteAny
+- DatasetDatablockCreateOwner
+- DatasetDatablockCreateAny
+- DatasetDatablockReadPublic
+- DatasetDatablockReadAccess
+- DatasetDatablockReadOwner
+- DatasetDatablockReadAny
+- DatasetDatablockUpdateOwner
+- DatasetDatablockUpdateAny
+- DatasetDatablockDeleteOwner
+- DatasetDatablockDeleteAny
+- DatasetLogbookReadOwner
+- DatasetLogbookReadAny
+
+### Implementation
+How the different level of authorization translates in data condition applied byt he backend.
+
+- **Public**
+-  `isPublished = true`
+- **Access** (condition ar applied in logical _or_)
+- `isPublished = true`
+- `ownerGroup` is one of the groups that the user belongs
+- `accessGroups` are one of the groups that the user belongs
+- `sharedWith` contains the user's email
+- **Owner**  
+  - `ownerGroup` is one of the groups that the user belongs
+- **Any**
+- User can perform the action to any dataset
+
+
+### Priority
+```
+    DatasetCreate-->DatasetCreateOwnerNoPid;
+    DatasetCreateOwnerNoPid-->DatasetCreateOwnerWithPid;
+    DatasetCreateOwnerWithPid-->DatasetCreateAny;
+```
+```
+    DatasetRead-->DatasetReadManyPublic;
+    DatasetReadManyPublic-->DatasetReadManyAccess;
+    DatasetReadManyAccess-->DatasetReadManyOwner;
+    DatasetReadManyOwner-->DatasetReadAny;
+    DatasetRead-->DatasetReadOnePublic;
+    DatasetReadOnePublic-->DatasetReadOneAccess;
+    DatasetReadOneAccess-->DatasetReadOneOwner;
+    DatasetReadOneOwner-->DatasetReadAny;
+```
+```
+    DatasetUpdate-->DatasetUpdateOwner;
+    DatasetUpdateOwner-->DatasetUpdateAny;
+    DatasetDelete-->DatasetDeleteOwner;
+    DatasetDeleteOwner-->DatasetDeleteAny;
+```
+
+### Authorization table
+Note, merely for visibility reasons the table has been split. Hierarchically, `OrigDatablocks` and `Datablocks` belong to `Datasets`.
+#### Datasets
+| HTTP method | Endpoint | Endpoint Authorization | Anonymous | Authenticated User | Create Dataset Groups | Create Dataset with Pid Groups | Create Dataset Privileged Groups | Admin Groups | Delete Groups | Notes |
+| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |
+| POST | Datasets | _DatasetCreate_ | __no__ | __no__ | Owner, w/o PID<br/>_DatasetCreateOwnerNoPid_ | Owner, w/ PID<br/>_DatasetCreateOwnerWithPid_ | Any<br/>_DatasetCreateAny_ | Any<br/>_DatasetCreateAny_ | __no__ | 
+| POST | Datasets/isValid | _DatasetCreate_ | __no__ | __no__ | Owner, w/o PID<br/>_DatasetCreateOwnerNoPid_ | Owner, W/ PID<br/>_DatasetCreateOwnerWithPid_ | Any<br/>_DatasetCreateAny_ | Any<br/>_DatasetCreateAny_ | __no__ | 
+| GET | Datasets | _DatasetRead_ | Public<br/>_DatasetReadPublic_ | Has Access<br/>_DatasetReadAccess_ | Has Access<br/>_DatasetReadAccess_ | Has Access<br/>_DatasetReadAccess_ | Has Access<br/>_DatasetReadAccess_ | Any<br/>_DatasetReadyAny_ | __no__ | 
+| GET | Datasets/fullquery | _DatasetRead_ | Public<br/>_DatasetReadManyPublic_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Any<br/>_DatasetReadAny_ | __no__ | 
+| GET | Datasets/fullfacet | _DatasetRead_ | Public<br/>_DatasetReadManyPublic_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Any<br/>_DatasetReadAny_ | __no__ | 
+| GET | Datasets/metadataKeys | _DatasetRead_ | Public<br/>_DatasetReadManyPublic_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Any<br/>_DatasetReadAny_ | __no__ | 
+| GET | Datasets/count | _DatasetRead_ | Public<br/>_DatasetReadManyPublic_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Has Access<br/>_DatasetReadManyAccess_ | Any<br/>_DatasetReadAny_ | __no__ | 
+| GET | Datasets/findOne | _DatasetRead_ | Public<br/>_DatasetReadOnePublic_ | Has Access<br/>_DatasetReadOneAccess_ | Has Access<br/>_DatasetReadOneAccess_ | Has Access<br/>_DatasetReadOneAccess_ | Has Access<br/>_DatasetReadOneAccess_ | Any<br/>_DatasetReadAny_ | __no__ | 
+| GET | Datasets/_pid_ | _DatasetRead_ | Public<br/>_DatasetReadOnePublic_ | Has Access<br/>_DatasetReadOneAccess_ | Has Access<br/>_DatasetReadOneAccess_ | Has Access<br/>_DatasetReadOneAccess_ | Has Access<br/>_DatasetReadOneAccess_ | Any<br/>_DatasetReadAny_ | __no__ | 
+| PATCH | Datasets/_pid_ | _DatasetUpdate_ | __no__ | __no__ | Owner<br/>_DatasetUpdateOwner_ | Owner<br/>_DatasetUpdateOwner_ | Owner<br/>_DatasetUpdateOwner_ | Any<br/>_DatasetUpdateAny_ | __no__ | 
+| PUT | Datasets/_pid_ |  _DatasetUpdate_ |__no__ | __no__ | Owner<br/>_DatasetUpdateOwner_ | Owner<br/>_DatasetUpdateOwner_ | Owner<br/>_DatasetUpdateOwner_ | Any<br/>_DatasetUpdateAny_ | __no__ | 
+| POST | Datasets/_pid_/appendToArrayField |  _DatasetUpdate_ |__no__ | __no__ | Owner<br/>_DatasetUpdateOwner_ |  Owner<br/>_DatasetUpdateOwner_ | Owner<br/>_DatasetUpdateOwner_ | Any<br/>_DatasetUpdateAny_ | __no__ | 
+| | | | | | | | | |
+| DELETE | Datasets/_pid_ | _DatasetDelete_ | __no__ | __no__ | __no__ | __no__ | __no__ | __no__ | Any<br/>_DatasetDeleteAny_ | 
+| | | | | | | | | |
+| GET | Datasets/_pid_/thumbnail | _DatasetRead_ | Public<br/>_DatasetReadPublic_ | Has Access<br/>_DatasetReadAccess_ | Has Access<br/>_DatasetReadAccess_ | Has Access<br/>_DatasetReadAccess_ | Has Access<br/>_DatasetReadAccess_ | Any<br/>_DatasetReadAny_ | __no__ | 
+| | | | | | | | | |
+| POST | Datasets/_pid_/attachments | _DatasetAttachmentCreate_ | __no__ | __no__ | Owner<br/>_DatasetAttachmentCreateOwner_ | Owner<br/>_DatasetAttachmentCreateOwner_ | Any<br/>_DatasetAttachmentCreateAny_ | Any<br/>_DatasetAttachmentCreateAny_ | __no__ | 
+| GET | Datasets/_pid_/attachments | _DatasetAttachmemntRead_ | Public<br/>_DatasetAttachmentReadPublic_ | Has Access<br/>_DatasetAttachmentReadAccess_ | Has Access<br/>_DatasetAttachmentReadAccess_ | Has Access<br/>_DatasetAttachmentReadAccess_ | Has Access<br/>_DatasetAttachmentReadAccess_ | Any<br/>_DatasetAttachmentReadAny_ | __no__ | 
+| PUT | Datasets/_pid_/attachments/_aid_ | _DatasetAttachmemntUpdate_ | __no__ | __no__ | Owner<br/>_DatasetAttachmentUpdateOwner_ | Owner<br/>_DatasetAttachmentUpdateOwner_ | Owner<br/>_DatasetAttachmentUpdateOwner_ |  Any<br/>_DatasetAttachmentCreateAny_ | __no__ | 
+| DELETE | Datasets/_pid_/attachments/_aid_ | _DatasetAttachmemntDelete_ | __no__ | __no__ | Owner<br/>_DatasetAttachmentDeleteOwner_ | Owner<br/>_DatasetAttachmentDeleteOwner_ | Owner<br/>_DatasetAttachmentDeleteOwner_ | Any<br/>_DatasetAttachmentDeleteAny_ | __no__ | 
+
+#### OrigDatablock
+| HTTP method | Endpoint | Endpoint Authorization | Anonymous | Authenticated User | Create Dataset Groups | Create Dataset with Pid Groups | Create Dataset Privileged Groups | Admin Groups | Delete Groups | Notes |
+| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |
+| POST | Datasets/_pid_/origdatablocks | _DatasetOrigdatablocksCreate_ | __no__ | __no__ | Owner<br/>_DatasetOrigdatablockCreateOwner_ | Owner<br/>_DatasetOrigdatablockCreateOwner_ | Any<br/>_DatasetOrigdatablockCreateAny_ | Any<br/>_DatasetOrigdatablockCreateAny_ | __no__ | 
+| POST | Datasets/_pid_/origdatablocks/isValid | _DatasetOrigdatablocksCreate_ |  __no__ | __no__ | Owner<br/>_DatasetOrigdatablockCreateOwner_ | Owner<br/>_DatasetOrigdatablockCreateOwner_ | Any<br/>_DatasetOrigdatablockCreateAny_ | Any<br/>_DatasetOrigdatablockCreateAny_ | __no__ | 
+| GET | Datasets/_pid_/origdatablocks | _DatasetOrigdatablocksRead_ | Public<br/>_DatasetOrigdatablockReadPublic_ | Has Access<br/>_DatasetOrigdatablockReadOAccess_ | Has Access<br/>_DatasetOrigdatablockReadAccess_ | Has Access<br/>_DatasetOrigdatablockReadAccess_ | Has Access<br/>_DatasetOrigdatablockReadAccess_ | Any<br/>_DatasetOrigdatablockReadAny_ | __no__ | 
+| PATCH | Datasets/_pid_/origdatablocks/_oid_ | _DatasetOrigdatablocksUpdate_ | __no__ | __no__ | Owner<br/>_DatasetOrigdatablockUpdateOwner_ | Owner<br/>_DatasetOrigdatablockUpdateOwner_ | Owner<br/>_DatasetOrigdatablockUpdateOwner_ | Any<br/>_DatasetOrigdatablockCreateAny_ | __no__ |  |
+| DELETE | Datasets/_pid_/origdatablocks/_oid_ | _DatasetOrigdatablocksDelete_ | __no__ | __no__ | __no__ |  __no__ | __no__ |  __no__ | Any<br/>_DatasetOrigdatablockDeleteAny_ |  | 
+
+
+#### Datablocks
+| HTTP method | Endpoint | Endpoint Authorization | Anonymous | Authenticated User | Create Dataset Groups | Create Dataset with Pid Groups | Create Dataset Privileged Groups | Admin Groups | Delete Groups | Notes |
+| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |
+| POST | Datasets/_pid_/datablocks | _DatasetDatablocksCreate_ | __no__ | __no__ | Owner<br/>_DatasetDatablockCreateOwner_ | Owner<br/>_DatasetDatablockCreateOwner_ | Owner<br/>_DatasetDatablockCreateOwner_ | Any<br/>_DatasetDatablockCreateAny_ | __no__ |  |
+| GET | Datasets/_pid_/datablocks | _DatasetOrigdatablocksRead_ | Public<br/>_DatasetDatablockReadPublic_ | Has Access<br/>_DatasetDatablockReadAccess_ | Has Access<br/>_DatasetDatablockReadAccess_ | Has Access<br/>_DatasetDatablockReadAccess_ | Has Access<br/>_DatasetDatablockReadAccess_ | Any<br/>_DatasetDatablockReadAny_ | __no__ |  |
+| PATCH | Datasets/_pid_/datablocks/_oid_ | _DatasetDatablocksUpdate_ | __no__ | __no__ | Owner<br/>_DatasetDatablockUpdateOwner_ | Owner<br/>_DatasetDatablockUpdateOwner_ | Owner<br/>_DatasetDatablockUpdateOwner_ | Any<br/>_DatasetDatablockCreateAny_ | __no__ | | 
+| DELETE | Datasets/_pid_/datablocks/_oid_ | _DatasetDatablocksDelete_ | __no__ | __no__ | __no__ |  __no__ | __no__ | __no__ | Any<br/>_DatasetDatablockDeleteAny_ | 
+| | | | | | | | | |
+| GET | Datasets/_pid_/logbook | _DatasetLogbookRead_ | __no__ | Owner<br/>_DatasetLogbookReadOwner_ |  Owner<br/>_DatasetLogbookReadOwner_ | Owner<br/>_DatasetLogbookReadOwner_ | Owner<br/>_DatasetLogbookReadOwner_ |  Any<br/>_DatasetLogbookReadAny_ | __no__ | |