KEY_VALUE_STORE_SCHEMA.md

Key-value store schema file specification [work in progress]

This JSON file should contain schema for files stored in the key-value store, defining their name, format, or content type.

BEWARE: This is currently not implemented yet and subject to change.

Basic properties

Key-value store schema has two main use cases described in the following examples:

1. Some Actors such as Instagram scraper store multiple types of files into the key-value store. Let's say the scraper stores images and user pictures. So for each of these, we would define a prefix group and allow the user to list images from a single group in both the UI and API.

{
   "collections": {
      "screenshots": {
            "name": "Post images",
            "keyPrefixes": ["images-"],
            "contentTypes": ["image/jpeg", "image/png"]
       }
   }
}

2. Some Actor stores a specific record, and we want to ensure the content type to be HTML and embed it into the run view. A good example is monitoring Actor that generates HTML report that we would like to embed to run view for the user once the monitoring is finished.

{
    "collections": {
        "monitoringReport": {
            "name": "Monitoring report",
            "description": "HTML page containing monitoring results",
            "key": "REPORT",
            "contentTypes": ["text/html"]
        }
    }
}

For more information on how to create a key-value store with a schema, see DATASET_SCHEMA.json as the implementation and API will be the same.

Structure

{
    "actorKeyValueStoreSchemaVersion": 1,
    "name": "My Instagram backup",
    "description": "Backup of my Instagram account",
    
    "collections": {
        "postImages": {
            "name": "Post images",
            "description": "Contains all Instagram post images",
            "keyPrefix": "post-image-",
            "contentTypes": ["image/jpeg", "image/png"]
        },

        "profilePicture": {
            "name": "Profile picture",
            "key": "profile-picture",
            "contentTypes": ["image/*"] // Be able to enable all images or text types etc.
        }
    }
}

API implications

Enable user to list keys for specific record group:

https://api.apify.com/v2/key-value-stores/storeId/keys?recordGroup=postImages&exclusiveStartKey=xxx

In addition to this user will be able to list by prefix directly:

https://api.apify.com/v2/key-value-stores/storeId/keys?prefix=post-images-