[kobotoolbox] Optionally download and store `deployed_versions` of a form

[nicopace]

This functionality should be added to https://github.com/ConservationMetrics/gc-scripts-hub/blob/main/f/connectors/kobotoolbox/kobotoolbox_responses.py

Right now, this script imports the data and some metadata of the exported survey.
Most of the data from the survey can be found in the xlsform files that define the survey.

There needs to be a consideration in relation to the multiple versions a survey can have, so not only the last one needs to be downloaded, but all of them, and match the version of it to the version in the database.

This can also be used to derive from them the transformations that happened in between versions, like using diff between them.

[rudokemper]

Are you suggesting that we add an optional Windmill script input parameter to point to the location of an XLSForm file? (Note that it is not possible to upload a file when running or scheduling a script.)

Or is there a KoboToolbox API endpoint from where we can fetch an XLSform definition? I don't know this but I believe the metadata endpoint {server_base_url}/api/v2/assets/{form_id}/ provides the current XLSForm definition e.g. all of the translations, field and form properties.

Zooming out, what is the use case in storing the full versioned history of an XLSForm definition?

[nicopace]

Or is there a KoboToolbox API endpoint from where we can fetch an XLSform definition? I don't know this but I believe the metadata endpoint {server_base_url}/api/v2/assets/{form_id}/ provides the current XLSForm definition e.g. all of the translations, field and form properties.

There is an API for that:
https://kf.kobotoolbox.org/api/v2/assets/

Zooming out, what is the use case in storing the full versioned history of an XLSForm definition?

It is valuable to:

1. reconstruct the same survey on KoboToolbox or any other XLSForm-compatible survey platform
2. The diff between the xlsforms can be used to guess the transformations to the dataset that need to be done to understand the data.
3. It is the metadata for the data we collect... an essential resource to understand what the data is. (from an Open Data perspective)

[rudokemper]

Thanks. I understand your points. I've made a similar argument for getting as much metadata as possible from a CoMapeo project -- like the category icons and hex codes for point colors.

So, we already do dump all of the /assets/{form_id}/ metadata into a JSON file. For each Kobo form, we generate a {form_name}_metadata.json file which can be accessed using Filebrowser.

Is there a different endpoint that we should fetch for XLSForm version definition? If so, is it sufficient for that, too, to be dumped into a file? (Based on your points, it doesn't seem that this is the kind of thing that needs to be written to the database.)

Lastly, is this a priority for partner accompaniment?

[nicopace]

Thanks. I understand your points. I've made a similar argument for getting as much metadata as possible from a CoMapeo project -- like the category icons and hex codes for point colors.

Excellent!

So, we already do dump all of the /assets/{form_id}/ metadata into a JSON file. For each Kobo form, we generate a {form_name}_metadata.json file which can be accessed using Filebrowser.

Is there a different endpoint that we should fetch for XLSForm version definition? If so, is it sufficient for that, too, to be dumped into a file? (Based on your points, it doesn't seem that this is the kind of thing that needs to be written to the database.)

I have not dig that much.

Lastly, is this a priority for partner accompaniment?

No.
It came from me replicating the reporting capabilities of KoboToolbox in Metabase/Superset, that the information within the XLSForms are relevant.

But there is no urgency, as needs have been address through other means.

[rudokemper]

I just glanced at the /assets/{form_id}/ endpoint.

It looks like has an object storing reference to past versions:

  "deployed_versions": {
    "count": 34,
    "next": null,
    "previous": null,
    "results": [
      {
        "uid": "{uid}",
        "url": "https://kf.kobotoolbox.org/api/v2/assets/{form_id}/versions/{uid}/",
        "content_hash": "{content_hash}",
        "date_deployed": "{date}",
        "date_modified": "{date}"
      },
      {
        "uid": "{uid}",
        "url": "https://kf.kobotoolbox.org/api/v2/assets/{form_id}/versions/{uid}/",
        "content_hash": "{content_hash}",
        "date_deployed": "{date}",
        "date_modified": "{date}"
      },
      ...
    ]
  } 

So one way to solve this issue would be to download the contents of those deployed version URLs and save them in a deployed_versions/ directory.

I'd say that if we do this, we make this an optional param for the Kobo script, since this may not always be desirable for every project. (In my experience, a LOT of past versions of a Kobo form are experimentation / trial and error messing around with the formbuilder.)

But let's leave this on ice until it becomes a priority.

[nicopace]

The rationale behind fetching the XLSForm is that it contains a lot of metadata that describes the data that was downloaded:

• long descriptions of the fields
• whether they were mandatory or not
• a description of the general structure of the dataset

And this could be used to improve the experience in user-facing interfaces like:

• Superset or Metabase table metadata

[rudokemper]

We already download /assets/{form_id}/ which returns all of the things you mentioned. Please have a look at this endpoint on one of the KoboToolbox servers, or one of the JSON files for one of the community instances using Filebrowser.

What we don't do is download different deployed versions e.g. /assets/{form_id}/versions/{uid}/.

[nicopace]

The other usecase (not that someone had asked for this), is to create a new survey based on one that you have done in the past... the XLSForm might prove handy.

Will check the code you are mentioning.