Adds samples and further documentation on use

Signed-off-by: Christopher Grote <[email protected]>

.gitignore

@@ -21,7 +21,6 @@
 *.war
 *.nar
 *.ear
-*.zip
 *.tar.gz
 *.rar
 

README.md

@@ -6,8 +6,169 @@
 [Apache Atlas](https://atlas.apache.org) is an open source metadata repository. This connector provides an example
 implementation for interacting with a metadata repository through the open metadata standards of Egeria.
 
-Note that currently the connector implementation is a work-in-progress -- more details on the usage and implementation
-of the connector will follow once it is in a more ready-to-use state.
+Note that currently the implemented connector is read-only: it only implements those methods necessary to search, retrieve,
+and communicate metadata from Atlas out into the cohort -- it does *not* currently implement the ability to update Atlas
+based on events received from other members of the cohort. (This is due to some current limitations in Apache Atlas -- 
+see [open issues](https://github.com/odpi/egeria-connector-apache-atlas/issues?q=is%3Aissue+is%3Aopen+label%3Aexternal).)
+
+Furthermore, [only a subset of the overall Open Metadata Types are currently implemented](docs/mappings/README.md).
+
+## How it works
+
+The Apache Atlas Repository Connector works through a combination of the following:
+
+- Apache Atlas's REST API, itself abstracted through the Apache Atlas Java Client
+- Apache Atlas's embedded Apache Kafka event bus
+    - specifically the `ATLAS_ENTITIES` topic
+
+## Getting started
+
+### Enable Apache Atlas's events
+
+To start using the connector, you will need an Apache Atlas environment, ideally running version 2. You will need to
+first enable all event notifications (ie. including for relationships) in your Apache Atlas environment, by adding the
+following line to your `conf/atlas-application.properties` file:
+
+```properties
+atlas.notification.relationships.enabled=true
+```
+
+You will likely need to restart your environment after making this change.
+
+### Build connector and copy to OMAG Server Platform
+
+After building the connector project (`mvn clean install`) the connector is available as:
+
+```text
+distribution/target/egeria-connector-apache-atlas-package-VERSION.jar
+```
+
+Simply copy this file to a location where it can be run alongside the OMAG Server
+Platform from the Egeria core (in the example below, the file would be copied to
+`/lib/egeria-connector-apache-atlas-package-VERSION.jar`).
+
+### Configure security
+
+There are [multiple options to configure the security of your environment](docs/security/README.md) for this connector,
+but this must be done prior to starting up the connector itself (step below).
+
+For simple tests, if you can run your Apache Atlas environment with only its most basic security and without HTTPS, then
+there is nothing additional you need to configure for the connector.
+
+Alternatively, if you can still use basic authentication (username and password) but must run Apache Atlas via HTTPS,
+and you simply want to test things out, the simplest (but most insecure) option is to set the environment variable
+`STRICT_SSL` to `false` using something like the following prior to starting up the OMAG Server Platform:
+
+```bash
+export STRICT_SSL=false
+```
+
+Note that this will disable all certificate validation for SSL connections made between Egeria and your Apache Atlas
+environment, so is inherently insecure.
+
+### Startup the OMAG Server Platform
+
+You can startup the OMAG Server Platform with this connector ready-to-be-configured by running:
+
+```bash
+$ java -Dloader.path=/lib -jar server-chassis-spring-VERSION.jar
+```
+
+(This command will startup the OMAG Server Platform, including all libraries
+in the `/lib` directory as part of the classpath of the OMAG Server Platform.)
+
+### Configure this Egeria connector
+
+You will need to configure the OMAG Server Platform as follows (order is important) to make use of this Egeria connector.
+For example payloads and endpoints, see the [Postman samples](samples).
+
+1. Configure your event bus for Egeria, by POSTing a payload like the following:
+
+    ```json
+    {
+        "producer": {
+            "bootstrap.servers":"localhost:9092"
+        },
+        "consumer": {
+            "bootstrap.servers":"localhost:9092"
+        }
+    }
+    ```
+
+    to:
+
+    ```
+    POST http://localhost:8080/open-metadata/admin-services/users/{{user}}/servers/{{server}}/event-bus?connectorProvider=org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider&topicURLRoot=OMRSTopic
+    ```
+
+1. Configure the cohort, by POSTing something like the following:
+
+    ```
+    POST http://localhost:8080/open-metadata/admin-services/users/{{user}}/servers/{{server}}/cohorts/cocoCohort
+    ```
+
+1. Configure the Apache Atlas connector, by POSTing a payload like the following:
+
+    ```json
+    {
+        "apache.atlas.rest.url": "http://{{atlas_host}}:{{atlas_port}}",
+        "apache.atlas.username": "{{atlas_user}}",
+        "apache.atlas.password": "{{atlas_password}}"
+    }
+    ```
+
+    to:
+
+    ```
+    {{baseURL}}/open-metadata/admin-services/users/{{user}}/servers/{{server}}/local-repository/mode/repository-proxy/details?connectorProvider=org.odpi.egeria.connectors.apache.atlas.repositoryconnector.ApacheAtlasOMRSRepositoryConnectorProvider
+    ```
+
+    The payload should include the hostname and port of your Apache Atlas environment, and a `username` and `password`
+    through which the REST API can be accessed.
+
+    Note that you also need to provide the `connectorProvider` parameter, set to the name of the Apache Atlas
+    connectorProvider class (value as given above).
+
+1. Configure the event mapper for Apache Atlas, by POSTing something like the following:
+
+    ```
+    POST http://localhost:8080/open-metadata/admin-services/users/{{user}}/servers/{{server}}/local-repository/event-mapper-details?connectorProvider=org.odpi.egeria.connectors.apache.atlas.eventmapper.ApacheAtlasOMRSRepositoryEventMapperProvider&eventSource=my.atlas.host.com:9027
+    ```
+
+    The hostname provided at the end should be the host on which your Apache Atlas Kafka bus is running, and include
+    the appropriate port number for connecting to that bus. (By default, for the embedded Kafka bus, the port is `9027`.)
+
+1. The connector and event mapper should now be configured, and you should now be able
+    to start the instance by POSTing something like the following:
+
+   ```
+   POST http://localhost:8080/open-metadata/admin-services/users/{{user}}/servers/{{server}}/instance
+   ```
+
+After following these instructions, your Apache Atlas instance will be participating in the Egeria cohort. For those
+objects supported by the connector, most new instances or updates to existing instances should result in that metadata
+automatically being communicated out to the rest of the cohort.
+
+(Note: there are still some limitations with Apache Atlas that prevent this being true for _all_ types, eg. see 
+[Jira ATLAS-3312](https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3312?filter=reportedbyme))
+
+## Loading samples
+
+If you have a completely empty Apache Atlas environment, you may want to load some sample metadata to further explore.
+
+Samples are provided under [samples](samples):
+
+- `AtlasGlossary.zip` contains a sample glossary, set of categories and terms as defined in the [Coco Pharmaceuticals](https://github.com/odpi/egeria/tree/master/open-metadata-resources/open-metadata-deployment/sample-data/coco-pharmaceuticals)
+    set of samples.
+
+These can be loaded to the environment using the following command:
+
+```shell script
+$ curl -g -X POST -u <user>:<password> -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F [email protected] "http://<host>:<port>/api/atlas/admin/import"
+```
+
+For additional samples, eg. of Hadoop-native entity types, see the [Hortonworks Sanbox](https://www.cloudera.com/downloads/hortonworks-sandbox.html).
+
 
 ----
 License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/),

docs/README.md

@@ -0,0 +1,15 @@
+<!-- SPDX-License-Identifier: CC-BY-4.0 -->
+<!-- Copyright Contributors to the ODPi Egeria project. -->
+
+# Detailed documentation
+
+More detailed documentation related to the connector is contained in this directory.
+
+- [Implemented Apache Atlas -> OMRS type mappings](mappings)
+- [Extensions made to Apache Atlas to implement certain OMRS details](apache-atlas-extensions.md)
+- [Notes for developers](developers)
+
+
+----
+License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/),
+Copyright Contributors to the ODPi Egeria project.

docs/apache-atlas-extensions.md

@@ -0,0 +1,60 @@
+<!-- SPDX-License-Identifier: CC-BY-4.0 -->
+<!-- Copyright Contributors to the ODPi Egeria project. -->
+
+# Apache Atlas Extensions
+
+As part of implementing the Apache Atlas Repository Connector, extension have been developed through the software's
+type definition extensibility mechanisms.
+
+## Adds properties to Referenceable
+
+Apache Atlas's `Referenceable` entity definition is extended with the following properties:
+
+- `additionalProperties` - a `map<string, string>` to be able to capture any additional properties that might be
+    present beyond those defined up-front in a type definition
+
+(The version of the `Referenceable` EntityDef is incremented to `1.3`.)
+
+## Adds supertype to AtlasGlossary
+
+Apache Atlas's `AtlasGlossary` entity definition is extended to also have `Referenceable` as a supertype.
+
+This is to ensure, like in the open metadata types, glossaries include the properties present on `Referenceable`
+(like `qualifiedName` and `additionalProperties`) and that they should also be included when searching the repository
+for `Referenceable` entities (and all of its subtypes). (Note though that this does not work within Atlas, presumably
+due to it still extending `__internal` as well.)
+
+(The version of the `AtlasGlossary` EntityDef is incremented to `1.2`.)
+
+## Adds supertype to AtlasGlossaryCategory
+
+As above for `AtlasGlossary`.
+
+(The version of the `AtlasGlossaryCategory` EntityDef is incremented to `1.2`.)
+
+## Adds supertype to AtlasGlossaryTerm
+
+As above for `AtlasGlossary`.
+
+(The version of the `AtlasGlossaryTerm` EntityDef is incremented to `1.2`.)
+
+## How the extensions work
+
+The extensions themselves are part of the source code tree under `src/main/resources/ApacheAtlasNativeTypesPatch.json`,
+and are automatically deployed into the Apache Atlas environment during the initialization of the connector --
+specifically, when you call the:
+
+```
+POST http://localhost:8080/open-metadata/admin-services/users/{{user}}/servers/{{server}}/instance
+```
+
+API interface of the OMAG Server Platform that has been configured to connect to an Apache Atlas environment
+(see [Getting Started](../README.md)).
+
+Because the extensions are necessary for the connector to operate, if there are any errors or problems deploying the
+extensions you should be notified of these during the initialization: you will most likely receive an error `500`
+response and should consult the Egeria and Apache Atlas logs for further details.
+
+----
+License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/),
+Copyright Contributors to the ODPi Egeria project.

docs/developers/README.md

@@ -0,0 +1,23 @@
+<!-- SPDX-License-Identifier: CC-BY-4.0 -->
+<!-- Copyright Contributors to the ODPi Egeria project. -->
+
+# Developer notes
+
+Following are important notes for developers wishing to extend this connector.
+
+## JavaDoc generation
+
+The JavaDoc comments in the codebase make use of UTF-8 characters to represent different languages, and therefore when
+generating the JavaDocs it is important to make use of these additional command-line parameters:
+
+```
+-encoding utf8 -docencoding utf8 -charset utf8
+```
+
+In IntelliJ, for example, these can be entered into the "Other command line arguments:" box in the pop-up dialogue when
+generating the JavaDocs.
+
+
+----
+License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/),
+Copyright Contributors to the ODPi Egeria project.

docs/mappings/README.md

@@ -0,0 +1,67 @@
+<!-- SPDX-License-Identifier: CC-BY-4.0 -->
+<!-- Copyright Contributors to the ODPi Egeria project. -->
+
+# Implemented mappings
+
+The following types are currently mapped from Apache Atlas to OMRS. Note that there are currently limited
+mappings from OMRS types to Apache Atlas types as this connector is primarily read-only (primarily capable of
+propagating or retrieving information _from_ Apache Atlas, and not _to_ Apache Atlas).
+
+Hoping for a mapping that isn't there?
+
+- [Submit an issue](https://github.com/odpi/egeria-connector-apache-atlas/issues/new), or
+- Check out any of the linked files below for examples of what's needed to create a mapping,
+    and create your own (and feel free to submit a PR with the result!)
+
+## Enumerations
+
+Mappings for enumerations are defined directly in the
+[EnumDefMappings.json](../../adapter/src/main/resources/EnumDefMappings.json) file.
+
+This file is a simple array of JSON objects, each of which defines:
+
+- the name of the Apache Atlas enumeration (keyed by `atlas`)
+- the name of the Open Metadata Type enumeration (keyed by `omrs`)
+- an array of value mappings (keyed by `propertyMappings`)
+    - each element of which is a name-value pair between the Apache Atlas value (`atlas`) and the Open Metadata value (`omrs`)
+
+In addition to these mapped enumerations, any enumerations defined in the Open Metadata types that are not mapped will
+also automatically be created within Apache Atlas (using their same name and valid values as in the open metadata types
+themselves.)
+
+## Type definitions
+
+The mappings for both Entity and Relationship TypeDefs are defined directly in the
+[TypeDefMappings.json](../../adapter/src/main/resources/TypeDefMappings.json) file.
+
+Like the enumeration mappings, this file is a relatively simple array of JSON objects, each of which defines:
+
+- the name of the Apache Atlas type definition (keyed by `atlas`)
+- the name of the Open Metadata type definition (keyed by `omrs`)
+- an (optional) unique prefix value, for any Open Metadata types that do not exist as distinct entities in Apache Atlas
+    but for which we should generate an entity in Open Metadata (keyed by `prefix`)
+- an array of property mappings (keyed by `propertyMappings`):
+    - each element of which is a name-value pair between the Apache Atlas property name (`atlas`) and the Open Metadata
+        property name (`omrs`)
+- for relationship type definitions, an array of endpoint mappings (keyed by `endpointMappings`):
+    - which must always have 2 elements, one representing each endpoint of the relationship
+    - each containing the name of the Apache Atlas relationship endpoint (`atlas`) and the name of the Open Metadata
+        endpoint of the relationship (`omrs`)
+    - in the case of relationships for generated entities, may instead of the `atlas` endpoint (since there is not one)
+        specify the `prefix` for the generated entity for that end of the relationship
+
+For any Open Metadata entity types that do not exist in Apache Atlas, they will not be automatically created -- only
+those that are mapped through the file defined above will exist.
+
+## Classifications
+
+Because Apache Atlas does not ship with any out-of-the-box Classifications, but has a "Classification" concept, the
+connector automatically creates any open metadata Classifications that are applicable to the types in Apache Atlas as
+part of its cohort registration process. Therefore the mappings between the open metadata Classifications and these
+auto-created Classifications are one-to-one -- all names, properties, and endpoints are identical within Apache Atlas
+to what is defined in the open metadata Classification itself.
+
+
+----
+License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/),
+Copyright Contributors to the ODPi Egeria project.