Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.0.0 nov 2022 updates #143

Merged
merged 11 commits into from
Nov 29, 2022
152 changes: 88 additions & 64 deletions docs/AMIviaSpreadsheets.md

Large diffs are not rendered by default.

44 changes: 30 additions & 14 deletions docs/ami_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,15 @@ tags:

[Archipelago Multi-Importer (AMI)](https://github.com/esmero/ami) is a module for batch/bulk/mass ingests of Archipelago digital objects (ADOs) and collections. AMI also enables you to perform batch administrative actions, such as updating, patching/revising, or deleting digital objects and collections. AMI's Solr Importer plugin can be used to create AMI ingests and migrating content from existing Solr-sourcable digital repositories (such as Islandora 7).

## Getting started with AMI

You can access AMI through the `AMI Sets` tab on the main Content page found at `/admin/content` or directly at `/amiset/list`.
## AMI Overview and Under-the-Hood Explanations

![AMI Sets List](images/ami/AMIsetsList.jpg)
*From the desk of [Diego Pino](https://github.com/DiegoPino)*

## AMI Overview and Pre-Release Notes - AMI 0.4.0 (Archipelago - 1.0.0)
AMI provides Tabulated data ingest for ADOs with customizable input plugins. Each Spreadsheet (or Google Spreadsheet) goes through a Configuration Multi-step setup and generates at the end an AMI Set. AMI Sets then can be enqueued or directly ingested, its generated Objects purged and reingested again, its source data (generated and enriched with UUIDS) CSV replaced, improved and uploaded again and ingested.

*From the desk of [Diego Pino](https://github.com/DiegoPino)*
### Learn More about Metadata in Archipelago and AMI

AMI provides Tabulated data ingest for ADOs with customizable input plugins. Each Spreadsheet (or Google Spreadsheet) goes through a Configuration Multi Step setup and generates at the end an AMI Set. AMI Sets then can be enqueued or directly ingested, its generated Objects purged and reingested again, its source data (generated and enriched with UUIDS) CSV replaced, improved and uploaded again and ingested.
Please review the [Metadata in Archipelago overview](metadatainarchipelago.md) to learn about Archipelago's unique approach to metadata and how this applies in the context of AMI set adminstration.

??? info "Click to read the full AMI 0.4.0 (Archipelago - 1.0.0) Pre-Release Notes."

Expand All @@ -39,20 +37,17 @@ AMI provides Tabulated data ingest for ADOs with customizable input plugins. Eac

You can access AMI through the `AMI Sets` tab on the main Content page found at `/admin/content` or directly at `/amiset/list`.

![AMI Sets List](images/ami/AMIsetsList.jpg)
![AMI Sets List Updated](images/AMIsetsList_updated_2022-11.jpg)

## Google Sheets API Configuration

If you plan on using the Google Sheets Importer option, you will need to [Configure the Google Sheets API](googleapi.md).

### Example Spreadsheet/CSV

This spreadsheet can be used to import a small set of Digital Objects using the same assets part of the [Two-Step Demo content ingest guide](archipelago-deployment-democontent.md) (these objects can also be found in the [Archipelago Deployment repository](https://github.com/esmero/archipelago-deployment)).
## Example Spreadsheet/CSV

- <https://docs.google.com/spreadsheets/d/10IAgDZ1_fGVd_2g1GyUYs0SrnkAfvVtTwukEaIEny5Y/edit?usp=sharing>
- To use this spreadsheet, you can either download as a CSV file to use with the Spreadsheet Importer, or make a copy of this using your [configured Google Sheets API account](googleapi.md).
Please refer to or use a fresh/new copy of the [Demo Archipelago Digital Objects (ADOs) spreadsheet](https://github.com/esmero/archipelago-deployment/blob/1.0.0/d8content/ami_set_entity_01.csv) to import a small set of Digital Objects, using the same assets part of the [One-Step Demo content ingest guide](archipelago-deployment-democontent.md).

### Example JSON template
## Example JSON template

This JSON template can be used during the Data Transformation (step 3) of your AMI Import. This particular template corresponds with the metadata elements found in the Default Descriptive Metadata and Default Digital Object Collection webforms shipped with Archipelago 1.0.0.

Expand All @@ -71,14 +66,30 @@ This JSON template can be used during the Data Transformation (step 3) of your A
"website_url": {{ data.website_url|json_encode|raw }},
"description": {{ data.description|json_encode|raw }},
"date_created": {{ data.date_created|json_encode|raw }},
"date_created_edtf": {{ data.date_created_edtf|json_encode|raw }},
"date_created_free": {{ data.date_created_free|json_encode|raw }},
"creator": {{ data.creator|json_encode|raw }},
"creator_lod": {{ data.creator_lod|json_encode|raw }},
"publisher": {{ data.publisher|json_encode|raw }},
"language": {{ data.language|json_encode|raw }},
"ismemberof": [],
"ispartof": [],
"sequence_id": {{ data.sequence_id|json_encode|raw }},
"owner": {{ data.owner|json_encode|raw }},
"local_identifier": {{ data.local_identifier|json_encode|raw }},
"related_item_host_title_info_title": {{ data.related_item_host_title_info_title|json_encode|raw }},
"related_item_host_display_label": {{ data.related_item_host_display_label|json_encode|raw }},
"related_item_host_type_of_resource": {{ data.related_item_host_type_of_resource|json_encode|raw }},
"related_item_host_local_identifier": {{ data.related_item_host_local_identifier|json_encode|raw }},
"related_item_note": {{ data.related_item_note|json_encode|raw }},
"related_item_host_location_url": {{ data.related_item_host_location_url|json_encode|raw }},
"note": {{ data.note|json_encode|raw }},
"physical_description_note_condition": {{ data.physical_description_note_condition|json_encode|raw }},
"note_publishinginfo": {{ data.note_publishinginfo|json_encode|raw }},
"physical_location": {{ data.physical_location|json_encode|raw }},
"physical_description_extent": {{ data.physical_description_extent|json_encode|raw }},
"date_published": {{ data.date_published|json_encode|raw }},
"date_embargo_lift": {{ data.date_embargo_lift|json_encode|raw }},
"rights_statements": {{ data.rights_statements|json_encode|raw }},
"rights": {{ data.rights|json_encode|raw }},
"subject_loc": {{ data.subject_loc|json_encode|raw }},
Expand All @@ -89,6 +100,11 @@ This JSON template can be used during the Data Transformation (step 3) of your A
"subject_wikidata": {{ data.subject_wikidata|json_encode|raw }},
"edm_agent": {{ data.edm_agent|json_encode|raw }},
"term_aat_getty": {{ data.term_aat_getty|json_encode|raw }},
"viaf": {{ data.viaf|json_encode|raw }},
"pubmed_mesh": {{ data.pubmed_mesh|json_encode|raw }},
"europeana_concepts": {{ data.europeana_concepts|json_encode|raw }},
"europeana_agents": {{ data.europeana_agents|json_encode|raw }},
"europeana_places": {{ data.europeana_places|json_encode|raw }},
"geographic_location": {{ data.geographic_location|json_encode|raw }},
"subjects_local_personal_names": {{ data.subjects_local_personal_names|json_encode|raw }},
"subjects_local": {{ data.subjects_locals|json_encode|raw }},
Expand Down
12 changes: 9 additions & 3 deletions docs/ami_spreadsheet_overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ tags:

There are multiple ways a spreadsheet/CSV file can be structured to work with AMI, depending on the data transformation and mapping you will be using.

- For most standard AMI ingests, each Row of your spreadsheet/CSV will correspond to a single Digital Object or Collection.
- For most standard AMI ingests, each Row of your spreadsheet/CSV will correspond to a single Digital Object, Creative Work Series (Compound Object Parent) or Collection.
- Columns in your spreadsheet/CSV can be mapped to different data (files) and metadata elements (label, description, subjects, etc.).

- It is recommended that different types of files are placed into separate columns--"images", "documents", "models", "videos", "audios", "text".
- It is recommended that different types of files are placed into separate columns--"images", "documents", "models", "videos", "audios", "texts".
- Filepaths can point to remote files, to existing files within your docker container, s3 (or other storage type/location that is accessible to Archipelago), and to paths within zip files.
- Example path for existing file within docker container:
`/var/www/html/d8content/myAMIimage.jpg`
Expand All @@ -22,7 +22,7 @@ There are multiple ways a spreadsheet/CSV file can be structured to work with AM
`https://dogsaregreat.edu/dogs.tiff`
- **Multiple files (of the same type) can be placed in a single cell, separated by a semicolon ( ; ).**
- For Digital Objects comprised of multiple types of files, such as an Oral History Interview with an audio file and a PDF transcript file, you can place different file types within different corresponding columns for the same Row.
- It is recommended that filepaths are copied/stored as plain (non hyperlinked) formatted text.
- It is recommended that filepaths are copied/stored as plain (non-hyperlinked) formatted text.

- **Every spreadsheet/CSV file should contain the following Columns:**
- `type`
Expand All @@ -33,13 +33,19 @@ There are multiple ways a spreadsheet/CSV file can be structured to work with AM
- this can be empty
- if empty, Archipelago will automatically generate UUIDs
- can be used with existing UUIDs during migrations
- **Soft-requirement** `sequence_id` for Creative Work Series (compound) children objects
- this is used to determine the sequence order for children objects within a Creative Work Series (compound) object
- should be an integer only (ie, '1' and not 'Page 1')
- if not present, the objects will present in the original ingest order
- we strongly recommend mapping the correct sequence using 'sequence_id'

- **Recommended Columns:**
- Files as defined above
- "images", "documents", "models", "videos", "audios", "text"
- .warc/.wacz files should be placed in a column "upload_associated_warcs"
- `ismemberof` and/or `ispartof` (and/or whatever predicate corresponds with the relationship you are mapping)
- these columns can be used to connect related objects using the object-to-object relationship that matches your needs
- in default Archipelago configurations, `ismemberof` is used for Collection Membership and `ispartof` is used for Parent-Child Object Relationships (so a Child ADO would reference the Parent ADO in `ispartof`)
- these columns can hold 3 types of values
- empty (no value)
- an integer to connect an object to another object's corresponding row in the same spreadsheet/CSV
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/AMIStep6_updated_2022-11.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/AMIqueueMgr_updated_2022-11.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/AMIreportOverview.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/AMIsetSuccess_updated_2022-11.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/AMIsetsList_updated_2022-11.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/AMIsetsPageReportOp.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
Binary file added docs/images/PluginImporterSelect_GoogleSheets.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/ami/AMIsetsList_updated.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading