Develop additional metadata elements for OAM Catalog #55

smit1678 · 2016-11-16T15:58:45Z

From other OAM improvements, we're going to look to implement some additional metadata elements for PacDID.

Next action: itemize the elements we need to add.

Recap of current:

Current metadata tracked

element	type	name	description
uuid	string	File	unique URI to file
projection	string	Projection	CRS of the datasource in full WKT format
bbox	array	Bounding Box	Pair of min and max coordinates in CRS units, (min_x, min_y, max_x, max_y)
footprint	string	Datasource footprint	WKT format, describing the actual footprint of the imagery
gsd	number	Ground Spatial Distance	Average ground spatial distance (resolution) of the datasource imagery, expressed in meters
file_size	number	File Size	File size on disk in bytes
acquisition_start	string	Acquisition Date Start	First date of acquisition in UTC (Combined date and time representation)
acquisition_end	string	Acquisition Date End	Last date of acquisition in UTC (Combined date and time representation) (optional)
title	string	Title	Human friendly title of the image
platform	string	Type of imagery	List of possible platform sources limited to satellite, aircraft, UAV, balloon, kite
provider	string	Imagery Provider	Provider/owner of the OIN bucket
contact	string	Contact	Name and email address of the data provider
properties	object	Properties	Additional properties about the image (optional)

smit1678 · 2016-12-12T10:40:28Z

Below is a recap of the future additions tracked in previous threads (https://github.com/hotosm/OpenAerialMap/tree/master/metadata, #31, #17) plus input received from the SPC and the PacDID project.

Future additions from our last review

element	type	name	description
license	string	License	Type of license for imagery
tags	string	Tags	User provided tags

Input from SPC/PacDID

element	type	name	description
bands	string	Bands available	Bands available or imaging bands
uav_type	string	Type of UAV	Type of UAV used for collection
operator	string	Operator of the UAV	Person or company that operated the UAV (could differ from provider)
gps	string	GPS Method Used	GPS method used -- autonomous or differential
abstract	string	Abstract	Short descriptor of the project or purpose of imagery

@cgiovando Additional metadata items for input before we select and finalize? Input on how we want to finalize? License and tag are candidates that we'll definitely want to include.

nbumbarger · 2016-12-14T18:27:22Z

Just wanted to update that we're planning to start work on this enhancement. Some aspects can't be developed until the list is finalized (form validation, for example), but generically scaffolding support for more metadata properties is something we can begin now.

smit1678 · 2016-12-15T02:32:46Z

From our thinking, we're leaning towards not having UAV-specific metadata items in the spec. bands could be an interesting addition. license and tags should be good to be included. The biggest question we're thinking about is how/if can we improve the core OIN spec as well as have an extended OAM version.

@mojodna Based on the mosaic work, is there anything we want to be thinking about for helping group items or connect between the Uploader and Catalog better?

nbumbarger · 2016-12-16T18:09:39Z

How would contributors input bands in the form? Would it be a set of checkboxes with some common ones? RE R G B NIR SWIR PAN? Or maybe Color Multispectral Panchromatic?

nbumbarger · 2016-12-16T22:22:12Z

@smit1678 - I'm going to need staging credentials and URLs for the catalog API in order to work on this feature next week. If anyone has access to these, could you get ahold of me on the Slack channel? As per the docs, the needed variables are:

OAM_DEBUG - Debug mode true or false (default)
AWS_SECRET_KEY_ID - AWS secret key id for reading OIN buckets
AWS_SECRET_ACCESS_KEY - AWS secret access key for reading OIN buckets
DBURI - MongoDB connection url
SECRET_TOKEN - The token used for post requests to /tms endpoint

cc @mojodna @danielfdsilva

smit1678 · 2016-12-19T18:06:18Z

To run this locally, you just need to set these yourself. Use your own AWS keys and then set the secret token. For our staging environment, you should have access through heroku.

…

_____________________________ From: Nick Bumbarger <[email protected]> Sent: Saturday, December 17, 2016 05:22 Subject: Re: [hotosm/OpenAerialMap] Develop additional metadata elements for OAM Catalog (#55) To: hotosm/OpenAerialMap <[email protected]> Cc: Nate Smith <[email protected]>, Mention <[email protected]> @smit1678 - I'm going to need staging credentials and URLs for the catalog API in order to work on this feature next week. If anyone has access to these, could you get ahold of me on the Slack channel? As per the docs, the needed variables are:OAM_DEBUG - Debug mode true or false (default)AWS_SECRET_KEY_ID - AWS secret key id for reading OIN bucketsAWS_SECRET_ACCESS_KEY - AWS secret access key for reading OIN bucketsDBURI - MongoDB connection urlSECRET_TOKEN - The token used for post requests to /tms endpoint cc @mojodna @danielfdsilva — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

nbumbarger · 2016-12-19T18:07:32Z

Thanks, @smit1678. I have it running.

mojodna · 2016-12-19T20:22:40Z

For the small-scale mosaicking (multiple images that are part of the same scene), we may want to add an optional "scene id" UUID (to group them together). That would also help people who've subdivided imagery and want to be able to associate it back together.

The the large-scale mosaicking (everything), gsd and acquisition dates are probably the main inputs. Having "quality" or "priority" might be helpful, but that's incredibly subjective.

nbumbarger · 2016-12-21T16:17:15Z

@mojodna I think the scene concept is implicit in the upload form, because it allows the user to add a list of images within a dataset but also allows for adding multiple datasets. It is true, however, that images submitted in a group within a dataset are not distinguished in the database. Are you proposing that the images in each submitted dataset be automatically assigned a unique group ID?

nbumbarger · 2016-12-21T16:24:49Z

@smit1678 @mojodna I added support for tags (optional) and license (required) to the form and catalog system (not yet merged); those seemed to be the highest priorities. Beyond the UUID question above, is there anything else we want to include at this time (quality, bands)? GSD is already generated, and acquisition dates are required with submission.

nbumbarger · 2016-12-21T19:27:55Z

In addition to adding a description of the new metadata properties to the API documentation, we also need to discuss where they should be exposed in the browser. For example, the license should probably be included in the image preview panel, but maybe we don't want to allow tags, considering that the user is currently able to attach an arbitrary number of them.

mojodna · 2016-12-28T22:39:53Z

Are you proposing that the images in each submitted dataset be automatically assigned a unique group ID?

Yes. The uploader is aware of relationships between images, but the rest of the toolchain isn't.

I'm tempted to advocate for the inclusion of bands, but we can table that until/if we decide that RGB(A) imagery is overly limiting.

We can also follow a philosophy of allowing common properties to emerge using well-known tags. This would require that metadata is mutable (in order to change them in response to consensus), which I'm not sure is the case right now.

@cgiovando recommended looking through OGC Earth Observation Metadata profile of Observations & Measurements to see if there's anything we're missing and to keep an eye out for future harmonization.

mojodna · 2016-12-29T23:17:16Z

Are you proposing that the images in each submitted dataset be automatically assigned a unique group ID?

Yes. The uploader is aware of relationships between images, but the rest of the toolchain isn't.

Further elaborating on this, the uploader should assign unique ids to each scene/dataset. Right now, the entire upload gets an id, as do the individual images, but there's no id that ties images to scenes within an upload.

E.g.: https://upload-api.openaerialmap.org/uploads/58655b07f91c99bd00e9c7ab

smit1678 · 2017-01-13T15:39:15Z

I started looking at the OGC EO profile, http://docs.opengeospatial.org/is/10-157r4/10-157r4.html, to get a sense of what/if we're missing. The spec covers much more than just optical and can be very mission/equipment specific.

I've tried to capture the Optical earth observation product information needed here: https://docs.google.com/spreadsheets/d/1yhX1cTfpa75wSKCDtRJTJ3rViw6exenTMXWsb1gpnnc/edit?usp=sharing.

Not mapped to OIN/OAM's current metadata spec. I think there will be ways for future harmonization. We seem to capture a subset of the information.

Couple items that we are missing that are specific to Optical products:

Details about cloud and snow cover
Details about sensor angle and other equipment specifics
Details about the resulting product - browse, mask, and product descriptions

smit1678 · 2017-01-13T16:27:45Z

Related to hotosm/oam-uploader-api#52 and looking at this further, it looks like one useful upgrade to the core OIN spec is creation_date and potentially modification_date. According to OGC, creation_date is:

creation date for the metadata item. When retrieved from a metadata catalogue, the creationDate is the date when the metadata item was ingested for the first time (i.e. inserted) in the catalogue.

This differs from acquisition_ time which is called phenomenonTime by OGC.

for upload IDs

Perhaps we can look at how OGC does it by using something like:

composedOf: Link to an EO product that is part of this EO product (e.g. a phr:DataStrip is composed of one or more phr:Scene)
subsetOf: Link to the “father” EO product (e.g. a phr:Scene is a subset of a phr:DataStrip)
linkedWith: Specify a link to another EO product (e.g. ERS1 and ERS2 interferometric pair)

mojodna · 2017-01-14T22:39:16Z

While working on re-indexing the HOT OIN bucket, I realized that the OIN metadata JSON should include something that identifies it as such (along with a version number), similar to how TileJSON does it:

{
  "...": "...",
  "tilejson": "2.1.0"
}

(The indexer currently attempts to treat all JSON files present in a bucket as OIN metadata, which is no longer true for the HOT bucket (there's footprint GeoJSON + TileJSON for the tiler); I have a temporary workaround that'll show up in a PR shortly, but it checks for uuid, which isn't a fully reliable check.)

mojodna · 2017-01-14T22:49:31Z

I'm using uploaded_at (for now) to signify when the imagery was updated (which is distinct from when it was ingested into the catalog).

Refs hotosm/OpenAerialMap#55

This skips JSON files that do not contain a `uuid` property. It would be better if there were an OIN-specific key/value to check for. Refs hotosm/OpenAerialMap#55

smit1678 · 2017-01-18T14:11:59Z

Ok, let's bring this home to a close. I started this in OIN: openimagerynetwork/oin-metadata-spec#14.

@mojodna @nbumbarger @cgiovando These changes look right and agreed upon?

Current OAM through Uploader:

{
    "uuid": "",
    "title": "",
    "projection": "",
    "bbox": [],
    "footprint": "",
    "gsd": ,
    "file_size": ,
    "acquisition_start": "",
    "acquisition_end": "",
    "platform": "",
    "provider": "",
    "contact": "",
    "properties": {
        "sensor": "",
        "thumbnail": "",
        "tms": ""
    }
}

New additions proposed to OAM through Uploader:

{
    "oin": "",              # OIN Version number. An update to OIN spec
    "uploaded_at" : "",     # date metdata uploaded into OIN. An update to OIN spec
    "uuid": "",
    "title": "",
    "projection": "",
    "bbox": [],
    "footprint": "",
    "gsd": ,
    "file_size": ,
    "acquisition_start": "",
    "acquisition_end": "",
    "platform": "",
    "provider": "",
    "contact": "",
    "creation_date": "",
    "properties": {
        "sensor": "",
        "thumbnail": "",
        "license": "",          # new addition, doesn't affect OIN spec
        "tags": "",             # new addition, doesn't affect OIN spec
        "tms": "",              
        "wmts": ""              # new addition, doesn't affect OIN spec
    }
}

mojodna · 2017-01-18T19:04:38Z

👍

1.1 for the OIN version?

smit1678 added this to the OAM Version 2 milestone Nov 16, 2016

This was referenced Dec 19, 2016

Additional metadata properties hotosm/oam-uploader-api#57

Merged

Feature/additional metadata hotosm/oam-uploader#54

Merged

mojodna added a commit to mojodna/oam-api that referenced this issue Jan 14, 2017

Add uploaded_at to metadata

2679d39

Refs hotosm/OpenAerialMap#55

smit1678 mentioned this issue Jan 18, 2017

Updates to OIN metadata spec openimagerynetwork/oin-metadata-spec#14

Open

smit1678 mentioned this issue Jan 18, 2017

WMTS endpoints #59

Closed

olafveerman assigned nbumbarger Feb 16, 2017

cgiovando added the v2 Features and ideas to be considered for v2 implementation label Apr 26, 2021

cgiovando unassigned nbumbarger Apr 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Develop additional metadata elements for OAM Catalog #55

Develop additional metadata elements for OAM Catalog #55

smit1678 commented Nov 16, 2016 •

edited

Loading

smit1678 commented Dec 12, 2016

nbumbarger commented Dec 14, 2016

smit1678 commented Dec 15, 2016

nbumbarger commented Dec 16, 2016

nbumbarger commented Dec 16, 2016

smit1678 commented Dec 19, 2016 via email

nbumbarger commented Dec 19, 2016

mojodna commented Dec 19, 2016 •

edited

Loading

nbumbarger commented Dec 21, 2016

nbumbarger commented Dec 21, 2016

nbumbarger commented Dec 21, 2016

mojodna commented Dec 28, 2016

mojodna commented Dec 29, 2016

smit1678 commented Jan 13, 2017

smit1678 commented Jan 13, 2017

mojodna commented Jan 14, 2017 •

edited

Loading

mojodna commented Jan 14, 2017

smit1678 commented Jan 18, 2017

mojodna commented Jan 18, 2017

Develop additional metadata elements for OAM Catalog #55

Develop additional metadata elements for OAM Catalog #55

Comments

smit1678 commented Nov 16, 2016 • edited Loading

Current metadata tracked

smit1678 commented Dec 12, 2016

Future additions from our last review

Input from SPC/PacDID

nbumbarger commented Dec 14, 2016

smit1678 commented Dec 15, 2016

nbumbarger commented Dec 16, 2016

nbumbarger commented Dec 16, 2016

smit1678 commented Dec 19, 2016 via email

nbumbarger commented Dec 19, 2016

mojodna commented Dec 19, 2016 • edited Loading

nbumbarger commented Dec 21, 2016

nbumbarger commented Dec 21, 2016

nbumbarger commented Dec 21, 2016

mojodna commented Dec 28, 2016

mojodna commented Dec 29, 2016

smit1678 commented Jan 13, 2017

smit1678 commented Jan 13, 2017

for upload IDs

mojodna commented Jan 14, 2017 • edited Loading

mojodna commented Jan 14, 2017

smit1678 commented Jan 18, 2017

Current OAM through Uploader:

New additions proposed to OAM through Uploader:

mojodna commented Jan 18, 2017

smit1678 commented Nov 16, 2016 •

edited

Loading

mojodna commented Dec 19, 2016 •

edited

Loading

mojodna commented Jan 14, 2017 •

edited

Loading