Skip to content

Commit d178ac1

Browse files
committed
Add publish data page
1 parent 0c715ee commit d178ac1

File tree

2 files changed

+35
-0
lines changed

2 files changed

+35
-0
lines changed

docs/mkdocs.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ nav:
6767
- Populate: compute/populate.md
6868
- Key Source: compute/key-source.md
6969
- Distributed Computing: compute/distributed.md
70+
- Publish Data: publish-data.md
7071
- Internals:
7172
- SQL Transpilation: internal/transpilation.md
7273
- Reproducibility:

docs/src/publish-data.md

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Publishing Data
2+
3+
DataJoint is a framework for building data pipelines that support rigorous flow of
4+
structured data between experimenters, data scientists, and computing agents *during*
5+
data acquisition and processing within a centralized project.
6+
Publishing final datasets for the outside world may require additional steps and
7+
conversion.
8+
9+
## Provide access to a DataJoint server
10+
11+
One approach for publishing data is to grant public access to an existing pipeline.
12+
Then public users will be able to query the data pipelines using DataJoint's query
13+
language and output interfaces just like any other users of the pipeline.
14+
For security, this may require synchronizing the data onto a separate read-only public
15+
server.
16+
17+
## Containerizing as a DataJoint pipeline
18+
19+
Containerization platforms such as [Docker](https://www.docker.com/) allow convenient
20+
distribution of environments including database services and data.
21+
It is convenient to publish DataJoint pipelines as a docker container that deploys the
22+
populated DataJoint pipeline.
23+
One example of publishing a DataJoint pipeline as a docker container is
24+
> Sinz, F., Ecker, A.S., Fahey, P., Walker, E., Cobos, E., Froudarakis, E., Yatsenko, D., Pitkow, Z., Reimer, J. and Tolias, A., 2018. Stimulus domain transfer in recurrent models for large scale cortical population prediction on video. In Advances in Neural Information Processing Systems (pp. 7198-7209). https://www.biorxiv.org/content/early/2018/10/25/452672
25+
26+
The code and the data can be found at https://github.com/sinzlab/Sinz2018_NIPS
27+
28+
## Exporting into a collection of files
29+
30+
Another option for publishing and archiving data is to export the data from the
31+
DataJoint pipeline into a collection of files.
32+
DataJoint provides features for exporting and importing sections of the pipeline.
33+
Several ongoing projects are implementing the capability to export from DataJoint
34+
pipelines into [Neurodata Without Borders](https://www.nwb.org/) files.

0 commit comments

Comments
 (0)