Integrate with Luigi

[briandoconnor]

Show Luigi can query elasticsearch, find work, and trigger analysis.

[briandoconnor]

Working on feature branch:
feature/#46-luigi-integration-brian

Todo for Luigi workflow:

• pull out the bam file from the structure
• map to the file UUID from the storage system
• write a task to download it
• write a task to process the bam
• write a task to prepare metadata.json and upload it

Then:

• write a script to make randomized BAM uploads
• write a script to generate the index
• write a script to call the Luigi workflow... need to be careful that multiple Luigi jobs aren't launched.... maybe have a certain percentage of failure in this?
• write a simple ES query that generates a TSV of available reports (standin for portal)

Later:

• add in Consonance (lots of work)
• add in real RNASeq data and workflow (need Dockerized/Dockstore workflow)
• write Luigi decider for RNAseq

[kozbo]

What is the status on this issue?