support JQ for API response transformation

[newgene]

JMESPath is a query language for JSON, with libraries available in many languages (e.g. javascript and Python).

We can evaluate and consider to use it in api-respone-transform.js sub-module. Potentially remove or simplify the existing hard-coded transformers.

The overall benefits I believe this new feature can bring us:

• Allow to standardize the API response transformation in a well-defined language, and we can then put it into the SmartAPI metadata, instead of hard-coded API-specific code within the module.
• Simplify the complexity of the api-respone-transform.js sub-module, since the many transformation logic will be handled by JMESPath module.
• Support more complex transformation logic if needed.

[newgene]

jq can be another option, but jq module is more commandline-focused, while JMESPath is more suitable to be used in our modules.

[colleenXu]

how does this relate to the templating work (biothings/call-apis.js#30, biothings/call-apis.js#26, biothings/call-apis.js#31)?

Also is there any concern about SmartAPI x-bte annotation being complex / very code-like @andrewsu ?

[rjawesome]

@ericz1803 if you have other stuff to work on right now, I can take this issue.

[ericz1803]

Yeah you can go ahead and take it

[rjawesome]

I started work by trying it on one transformer on the use-jmes-path branch of api-response-transform. For now I'm trying JQ because it seems jmes path is missing some functions like string splitting.

[rjawesome]

I created some jq transformers for EBI, CTD, and OpenTarget. Is there any test queries I can use for any of these?

[andrewsu]

Next steps in this issue:

• @newgene to reach out to @rjawesome to discuss jq vs JMESPath
• finish migrating the rest of the transformers

Future issues:

• move jq code to SmartAPI metadata
• update documentation describing necessary formats
• update BTE to consume jq annotation

[colleenXu]

Replying to Rohan's comment:

• I think I removed CTD from BTE's list of APIs a while back due to an outdated SmartAPI yaml.
• I removed OpenTargets from BTE's list of APIs more recently because that API had been removed and replaced with a diff query-interface.

I think the EBI transformer is for EBI Proteins API? So the test-query I have for that API is this (from the comments in the x-bte section):

{
    "message": {
        "query_graph": {
            "edges": {
                "e01": {
                    "subject": "n0",
                    "object": "n1"
                }
            },
            "nodes": {
                "n0": {
                    "ids": ["UniProtKB:Q93099", "UniProtKB:Q9Y6P5"],
                    "categories": ["biolink:Gene"]
                },
                "n1": {
                    "categories": ["biolink:MolecularActivity"]
                }
            }
        }
    }
}

[colleenXu]

JQ draft PR biothings/api-respone-transform.js#38