-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
73 lines (73 loc) · 11.6 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
<!doctype html><html data-scroll=0 lang=en dir=ltr><head><meta name=generator content="Hugo 0.123.8"><meta charset=UTF-8><meta name=viewport content="width=device-width,initial-scale=1"><meta name=description content="Notice: This Documentation is in the process of being updated, some of the information may be out of date, or incorrect.
Introduction # The FAIR Data Pipeline is intended to enable tracking of provenance of FAIR (findable, accessible, interoperable and reusable) data used in epidemiological modelling. Pipeline APIs written in C, C++, FORTRAN, Java, Julia, Python and R can be called by modelling software for data ingestion. These interact with a local relational database storing metadata and the local filesystem, and are configured using a yaml file associated with the model run."><meta name=theme-color media="(prefers-color-scheme: light)" content="#ffffff"><meta name=theme-color media="(prefers-color-scheme: dark)" content="#343a40"><meta name=color-scheme content="light dark"><meta property="og:title" content="Introduction"><meta property="og:description" content><meta property="og:type" content="website"><meta property="og:url" content="https://www.fairdatapipeline.org/"><title>Introduction | FAIRDataPipeline</title>
<link rel=manifest href=/manifest.json><link rel=icon href=/favicon.png><link rel=stylesheet href=/book.min.78e0f2c3e0da3b6a0d761a0e01656154c8d89cc8742b7e517cbc86441aab779b.css integrity="sha256-eODyw+DaO2oNdhoOAWVhVMjYnMh0K35RfLyGRBqrd5s=" crossorigin=anonymous><script defer src=/sw.min.6f6f90fcb8eb1c49ec389838e6b801d0de19430b8e516902f8d75c3c8bd98739.js integrity="sha256-b2+Q/LjrHEnsOJg45rgB0N4ZQwuOUWkC+NdcPIvZhzk=" crossorigin=anonymous></script><link rel=alternate type=application/rss+xml href=https://www.fairdatapipeline.org/index.xml title=FAIRDataPipeline><script defer type=text/javascript src=/lunr.min.js></script><script defer src=/en.fancy-search.min.de24ecd18cde6f14e126fb9b38a62fea7ae1a6bbd4a1459a29f0ee6003d85ad2.js integrity="sha256-3iTs0YzebxThJvubOKYv6nrhprvUoUWaKfDuYAPYWtI="></script><script defer src=/scroll.js></script></head><body dir=ltr><input type=checkbox class="hidden toggle" id=menu-control>
<input type=checkbox class="hidden toggle" id=toc-control><div class=custom-header><div class="custom-flex container flex"><div class=custom-left><div class=custom-brand><h2 class=custom-brand-text><a href=/><img src=/logo.jpg alt=Logo><span>FAIRDataPipeline</span></a></h2></div></div><div class=custom-search-container><div class=search-top><div class=search><div class=search-input-wrap><input type=text id=search-input class=search-input tabindex=0 placeholder="Search FAIRDataPipeline" aria-label="Search FAIRDataPipeline" autocomplete=off>
<label for=search-input class=search-label><div class=search-svg><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentcolor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-search"><circle cx="11" cy="11" r="8"/><line x1="21" y1="21" x2="16.65" y2="16.65"/></svg></div></label></div><div id=search-results class=search-results></div></div></div></div></div></div><main class="container flex"><aside class=book-menu><div class=book-menu-content><nav><ul><li><a href=/docs/quick_start/>Quick Start</a><ul></ul></li><li><input type=checkbox id=section-44941259176b7b469cf043fda5bc0d2f class=toggle>
<label for=section-44941259176b7b469cf043fda5bc0d2f class="flex justify-between"><a href=/docs/cli/>FAIR CLI</a></label><ul><li><a href=/docs/cli/installation/>FAIR CLI Installation</a><ul></ul></li><li><a href=/docs/cli/cli_commands/>FAIR CLI Commands</a><ul><li><a href=/docs/cli/cli_commands/local/>CLI Local Command(s)</a><ul></ul></li><li><a href=/docs/cli/cli_commands/examples/example1/>DPAPI functions</a><ul></ul></li></ul></li></ul></li><li><input type=checkbox id=section-587e02d9e7deadfe2eb56765069ad4bb class=toggle>
<label for=section-587e02d9e7deadfe2eb56765069ad4bb class="flex justify-between"><a href=/docs/data_registry/>FAIR Data Registry</a></label><ul><li><input type=checkbox id=section-1e279a6587cffe062775a88592a46e74 class=toggle>
<label for=section-1e279a6587cffe062775a88592a46e74 class="flex justify-between"><a href=/docs/data_registry/local_registry/>Local Registry</a></label><ul><li><input type=checkbox id=section-b80603f175f75996389b0e6cf368f84f class=toggle>
<label for=section-b80603f175f75996389b0e6cf368f84f class="flex justify-between"><a href=/docs/data_registry/local_registry/installation/>Installation</a></label><ul></ul></li></ul></li><li><input type=checkbox id=section-e607823ea324fecf0cd179a24f6cf62f class=toggle>
<label for=section-e607823ea324fecf0cd179a24f6cf62f class="flex justify-between"><a href=/docs/data_registry/remote_registry/>Remote Registry</a></label><ul><li><a href=/docs/data_registry/remote_registry/installation/>Installation</a><ul></ul></li><li><a href=https://data.fairdatapipeline.org target=_blank rel=noopener>Example Remote Registry</a><ul></ul></li></ul></li><li><a href=/docs/data_registry/schema/>Schema Description</a></li><li><a href=/docs/data_registry/prov_report/>Provenance Report</a></li><li><a href=/docs/data_registry/ro_crate/>RO Crate Objects</a></li></ul></li><li><input type=checkbox id=section-f36fce4449df18f368b69f4c53a14a48 class=toggle>
<label for=section-f36fce4449df18f368b69f4c53a14a48 class="flex justify-between"><a href=/docs/api/>Modelling APIs</a></label><ul><li><a href=/docs/api/r/>R</a><ul></ul></li><li><a href=/docs/api/julia/>Julia</a><ul></ul></li><li><a href=/docs/api/python/>Python</a><ul></ul></li><li><input type=checkbox id=section-564032734992eab058204e48eb16adb0 class=toggle>
<label for=section-564032734992eab058204e48eb16adb0 class="flex justify-between"><a href=/docs/api/java/>Java</a></label><ul><li><a href=/docs/api/java/coderun/>Coderun</a></li><li><a href=/docs/api/java/issues/>Issues</a></li><li><a href=/docs/api/java/parameters/>Parameters</a></li><li><a href=/docs/api/java/hdf5/>Hdf5</a></li><li><a href=/docs/api/java/run/>Run</a></li><li><a href=/docs/api/java/debug/>Debug</a></li></ul></li><li><a href=/docs/api/c/>C</a><ul></ul></li><li><a href=/docs/api/cpp/>C++</a><ul></ul></li><li><a href=/docs/api/fortran/>FORTRAN</a><ul></ul></li></ul></li><li><a href=/docs/jargon_buster/>Jargon buster</a><ul></ul></li><li><a href=/docs/repository_status/>Repository Status</a><ul></ul></li></ul></nav><script>(function(){var e=document.querySelector("aside .book-menu-content");addEventListener("beforeunload",function(){localStorage.setItem("menu.scrollTop",e.scrollTop)}),e.scrollTop=localStorage.getItem("menu.scrollTop")})()</script></div></aside><div class=book-page><header class=book-header><div class="flex align-center justify-between"><label for=menu-control><img src=/svg/menu.svg class=book-icon alt=Menu>
</label><strong>Introduction</strong>
<label for=toc-control><img src=/svg/toc.svg class=book-icon alt="Table of Contents"></label></div><aside class="hidden clearfix"><nav id=TableOfContents><ul><li><a href=#introduction>Introduction</a><ul><li><a href=#running-models>Running Models</a></li><li><a href=#getting-data>Getting data</a></li></ul></li></ul></nav></aside></header><article class=markdown><p><span style=color:red><em>Notice: This Documentation is in the process of being updated, some of the information may be out of date, or incorrect.</em></span></p><h1 id=introduction>Introduction
<a class=anchor href=#introduction>#</a></h1><p>The FAIR Data Pipeline is intended to enable tracking of provenance of
<a href=https://doi.org/10.1038/sdata.2016.18>FAIR</a> (findable, accessible, interoperable and reusable) data used in epidemiological modelling. Pipeline APIs written in
<a href=/docs/api/c/>C</a>,
<a href=/docs/api/cpp/>C++</a>,
<a href=/docs/api/fortran/>FORTRAN</a>,
<a href=/docs/api/java/>Java</a>,
<a href=/docs/api/julia/>Julia</a>,
<a href=/docs/api/python/>Python</a> and
<a href=/docs/api/r/>R</a> can be called by modelling software for data ingestion. These interact with a local relational database storing metadata and the local filesystem, and are configured using a yaml file associated with the model run. Local files and metadata can be synchronised with a remote registry via a command line interface (
<a href=/docs/cli/><code>fair</code></a>).</p><p>The key benefits of using the FAIR Data Pipeline are:</p><ul><li>Opensource, all code is available on the
<a href=https://github.com/FAIRDataPipeline>FAIRDataPipeline GitHub</a></li><li>Data recorded in a FAIR fashion (metadata on all data and code open and available for inspection)</li><li>Provenance tracing allows model outputs to be traced to inputs and modelling code</li><li>Multiple language support</li><li>Designed to run on a broad range of platforms (including HPC, inside Safe Havens)</li><li>Designed to be set up and completed online (to down-/up-load data) and run offline (Safe Havens will require this)</li><li>Open metadata provides knowledge of or access to shared central data for specific domains (e.g. COVID-19 epidemiological modelling)</li></ul><h2 id=running-models>Running Models
<a class=anchor href=#running-models>#</a></h2><p>To use the FAIR Data Pipeline with a piece of modelling software, you must add a language specific Pipeline API as a dependency and interact with data registered in the pipeline via the methods it presents. Each model run must be configured using a
<a href=docs/interface/_index.md><code>config.yml</code></a> file which specifies inputs and outputs by metadata.</p><script src=/mermaid.min.js></script><script>mermaid.initialize({flowchart:{useMaxWidth:!0},theme:"default"})</script><p class=mermaid>graph LR;
subgraph CLI
fair
end
subgraph Local API
API[Pipeline API]
CY[config.yml]
end
subgraph localhost
LR[Registry]
FS[File Store]
end
subgraph Model
MC[Model code]
end
fair --> CY
CY --> API
fair --> MC
MC --> |read/write/link_*| API
API --> |read/write/link_*| LR
LR --> |read/link_*| API
API --> |read/link_*| MC
API --> |write_*| FS
FS --> |read_*| API
MC --> |"(from link_write)"| FS
FS --> |"(from link_read)"| MC</p><h2 id=getting-data>Getting data
<a class=anchor href=#getting-data>#</a></h2><p>The command line interface
<a href=/docs/cli/><code>fair</code></a> is used to download and upload data and metadata required for and produced by model runs.</p><p class=mermaid>graph LR;
subgraph Remote
RR
OS
URI
end
subgraph Local
LR
FS
end
RR[Remote Registry]-->|fair pull| LR[Local Registry]
LR-->|fair push| RR
RR-->OS(Managed Object Store)
RR-->URI(Arbitrary URI)
LR-->FS(Local Filesystem)
OS-->|fair pull| FS
URI-->|fair pull| FS
FS-->|fair push| OS
FS-->|fair push| URI</p></article><footer class=book-footer><div class="flex flex-wrap justify-between"><div><a class="flex align-center" href=https://github.com/FAIRDataPipeline/FDP_hugo/commit/00c5b6f84cc658142d0317b104aa88589e66344b title='Last modified by Ryan J Field | December 8, 2023' target=_blank rel=noopener><img src=/svg/calendar.svg class=book-icon alt=Calendar>
<span>December 8, 2023</span></a></div></div><script>(function(){function e(e){const t=window.getSelection(),n=document.createRange();n.selectNodeContents(e),t.removeAllRanges(),t.addRange(n)}document.querySelectorAll("pre code").forEach(t=>{t.addEventListener("click",function(){if(window.getSelection().toString())return;e(t.parentElement),navigator.clipboard&&navigator.clipboard.writeText(t.parentElement.textContent)})})})()</script></footer><div class=book-comments></div><label for=menu-control class="hidden book-menu-overlay"></label></div><aside class=book-toc><div class=book-toc-content><nav id=TableOfContents><ul><li><a href=#introduction>Introduction</a><ul><li><a href=#running-models>Running Models</a></li><li><a href=#getting-data>Getting data</a></li></ul></li></ul></nav></div></aside></main></body></html>