Skip to content

Commit d7e5cb6

Browse files
MitchellShiellMitchellShiell
authored
Song Dev Documentation (#857)
1 parent a81a8e4 commit d7e5cb6

38 files changed

+1977
-396
lines changed

‎CONTRIBUTING.md

+103
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# Contributing to Overture
2+
3+
We appreciate your interest in contributing to the Overture project. We are the Genome Informatics Software Engineering team from Ontario Institute for Cancer Research. At OICR we develop new software, databases and other necessary components to store, organize and compute over the large and complex datasets being generated by our cancer research programs. Embodying OICR's values of collaboration and community, we are firm believers in open-source and open-science. As such we strongly believe in the collective power of expertise and shared resources.
4+
5+
## Code of Conduct
6+
7+
By participating in this project, you are expected to abide by our [Code of Conduct](https://docs.overture.bio/community/code-of-conduct). Please take the time to read it carefully before contributing.
8+
9+
## Get Involved
10+
11+
**Getting Started:** Our primary platform for community support, feature requests, and general discussions is GitHub Discussions. This allows us to keep all conversations in one place and make them easily searchable for future reference.
12+
13+
14+
**Community Support:** Our primary platform for community support, feature requests, and general discussions is [GitHub Discussions](https://github.com/overture-stack/docs/discussions). This allows us to keep all conversations in one place and make them easily searchable for future reference.
15+
16+
- **Getting Help:** If you need assistance with Overture, please create a [new discussion in our support category](https://github.com/overture-stack/docs/discussions/categories/support).
17+
- Before creating a new discussion, please search existing discussions to see if your question has already been answered.
18+
- **Feature Requests & Proposals:** We love hearing your ideas for improving Overture! Before making a feature request, please check our [**feature roadmap**](https://github.com/orgs/overture-stack/projects/11/views/1) to see if your idea is already planned.
19+
- If your idea isn't on the roadmap, feel free to [**submit a feature request**](https://github.com/overture-stack/docs/discussions/categories/ideas) by creating a new discussion in our Ideas category
20+
- **Report a Potential Bug:** We use GitHub Issues primarily for tracking confirmed bugs and ticketing development tasks. If you come across a potential bug or issue, please first post it to our [**GitHub support discussion forum**](https://github.com/overture-stack/docs/discussions/categories/support).
21+
- This allows us to confirm the issue and gather more information if needed. If we determine that further development is required, we will create and tag you into a GitHub Issue from your discussion post.
22+
23+
## Our Development Process
24+
25+
We use GitHub issues and pull requests for communication related to code changes.
26+
27+
### Branch Organization
28+
29+
We use the following standard branches:
30+
31+
- `main` is for stable production code
32+
- `develop` is the integration branch for new features
33+
- `feature/<name>` for feature branches
34+
- `release/v<version>` for release branches
35+
- `hotfix/<name>` for hotfix branches
36+
37+
## Pull Requests
38+
39+
### Submitting a Pull Request
40+
41+
We welcome and encourage pull requests from the community. To submit a pull request, please follow these steps:
42+
43+
1. **Fork the Repository**: Fork the Overture repository on GitHub.
44+
2. **Clone Your Fork**: Clone your forked repository to your local machine.
45+
3. **Create a New Branch**: Create a new branch for your changes. Use lowercase and hyphens (e.g., `feature/user-authentication`). Include ticket/issue numbers when applicable (e.g., `feature/PROJ-123-user-authentication`).
46+
4. **Make Your Changes**: Implement your changes and commit them to your branch. Write clear, concise commit messages in present tense (e.g., "Add feature" not "Added feature"). Reference issue numbers in commits when applicable.
47+
5. **Push Your Changes**: Push your changes to your forked repository.
48+
6. **Submit a Pull Request**: Open a pull request against the main repository.
49+
50+
### Best Practices
51+
52+
1. **Keep PRs as small as possible:** Focus on one feature or bug fix per pull request. Break large changes into smaller, more manageable pieces making it easier for reviewers to understand and approve your changes.
53+
54+
2. **Use descriptive titles:** Start with a verb (e.g., "Add", "Fix", "Update", "Refactor"), briefly summarize the main purpose of the PR and include the issue number if applicable (e.g., "Fix user authentication bug (#123)").
55+
56+
3. **Describe how you tested it:** Explain the testing process you followed and mention any new automated tests you've added.
57+
58+
4. **Provide a clear description:** Explain the purpose of your changes and list the main modifications you've made. Mention any potential side effects or areas that might need extra attention.
59+
60+
5. **Link related issues:** Reference any related issues or pull requests. Use GitHub keywords to automatically link issues (e.g., "Closes #123", "Fixes #456").
61+
6. **Keep the PR's branch up-to-date:** Regularly rebase your branch on the latest main branch and resolve any merge conflicts promptly.
62+
63+
7. **Respond to feedback:** Be open to suggestions and willing to make changes. Address all comments from reviewers. If you disagree with a suggestion, explain your reasoning politely.
64+
65+
8. **Include documentation updates:** If your changes affect user-facing features, update or create and issue detailing the relevant changes need to the documentation. Where appropriate include inline comments for complex code sections.
66+
67+
10. **Be patient:** Reviewers will likely be unable to respond immediately. However, feel free to follow up politely if you haven't received feedback after a reasonable time.
68+
69+
### Using Draft Pull Requests
70+
71+
Draft Pull Requests are an excellent way to document work in progress and facilitate early feedback. Use them to:
72+
73+
- Organize your thoughts and process
74+
- Share early work and ideas with the team
75+
- Get feedback on implementation approaches before finalizing code
76+
- Track progress on long-running features
77+
78+
Guidelines for Draft Pull Requests:
79+
80+
1. **Creation**:
81+
- Open a pull request and select "Create draft pull request"
82+
- Clearly mark the title with [WIP] or [DRAFT] prefix
83+
2. **Description**:
84+
- Outline the current state of the work
85+
- List planned tasks or improvements
86+
- Highlight areas where feedback is specifically needed
87+
3. **Updates**:
88+
- Regularly update the description or provide comments following commits with progress notes
89+
- Use task lists (using `- [ ]` in Markdown) to track completion of sub-tasks
90+
4. **Collaboration**:
91+
- Encourage early feedback and discussion
92+
- Use the pull request comments for design discussions
93+
5. **Finalization**:
94+
- Complete all planned work and address feedback
95+
- Update tests and documentation
96+
- Click "Ready for review" to move out of draft state
97+
98+
### Merging a Pull Request
99+
100+
- Ensure all CI checks pass
101+
- Obtain the required number of approvals
102+
- Use the project's specified merge strategy (Typically squash and merge)
103+
- Delete the source branch after merging if no longer needed

‎README.md

+38-231
Original file line numberDiff line numberDiff line change
@@ -1,250 +1,57 @@
1-
<h1 align="center">Song</h1>
1+
# Song
22

3-
<p align="center">Quickly and reliably track genome metadata scattered across multiple Cloud storage systems.</p>
3+
Song functions as a file catalog system, tracking files and managing their metadata. To manage file transfers to and from object storage Song interacts with its required companion application, Score.
44

5-
<p align="center"><a href="http://www.overture.bio/products/song" target="_blank"><img alt="General Availability" title="General Availability" src="http://www.overture.bio/img/progress-horizontal-GA.svg" width="320" /></a></p>
5+
</br>
66

7-
[![Documentation Status](http://readthedocs.org/projects/song-docs/badge/?version=develop)](https://song-docs.readthedocs.io/en/develop/introduction.html)
8-
[![Slack](http://slack.overture.bio/badge.svg)](http://slack.overture.bio)
9-
10-
Project containing both the SONG microservice and CLI client.
11-
Both are written using JAVA 8 and Spring Boot.
7+
>
8+
> <div>
9+
> <img align="left" src="ov-logo.png" height="50"/>
10+
> </div>
11+
>
12+
> Song is part of [Overture](https://www.overture.bio/), a collection of open-source software microservices used to create platforms for researchers to organize and share genomics data.*
13+
>
14+
>
1215
1316
## Documentation
1417

15-
Explore documentation with the Song [Read the Docs](https://song-docs.readthedocs.io/en/develop/introduction.html).
16-
17-
## Build
18-
19-
```bash
20-
$ mvn clean package
21-
```
22-
23-
## Running
24-
25-
#### Command-line
26-
27-
The source can be built and run using maven.
28-
29-
```bash
30-
$ git clone [email protected]:icgc-dcc/SONG.git
31-
$ cd SONG/song-server
32-
$ mvn spring-boot:run -Dspring-boot.run.profiles=dev,test
33-
```
34-
35-
Both the server and client when compiled and built produce uber jars which can be run easily from the command line.
36-
37-
```bash
38-
$ java -jar song-server-0.1.1-SNAPSHOT.jar --spring.profiles.active=dev,test
39-
```
40-
41-
## Docker Based Developement
42-
43-
Several `make` targets are provided for locally deploying dependent services using docker.
44-
By using this, the developer will be able to replicate a live environment for song-server and song-client.
45-
It allows the user to develop locally, and test submissions, manifest creation, publishing, unpublishing and score uploads/downloads in an isolated environment.
46-
47-
There are 2 modes:
48-
49-
### 1. Developement Mode
50-
51-
The purpose of this mode is to decrease the wait time between building and testing against dependent services.
52-
This mode will run a `mvn package` if the distribution files are missing and copy them into a container for them to be run.
53-
This method allows for fast developement, since the `mvn package` step is handled on the **Docker host**.
54-
In addition, the debug ports `5005` and `5006` are exposed for both `song-client` and `song-server`, respectively, allowing developers to debug the docker containers.
55-
This mode can be enabled using the `DEMO_MODE=0` override. This is the default behaviour if the variable `DEMO_MODE` is not defined.
56-
57-
#### Debugging the song-client with IntelliJ
58-
59-
Since the JVM debug port is exposed by the `song-client` docker container, IntelliJ can **remotely debug** a running docker container.
60-
To do this, a **docker image run profile** must be created with the configuration outputted by the `make intellij-song-client-config` command, which will output a basic upload command, however it can be modified to be any song-client command.
61-
Then, a **remote debug profile** must be created, with the following config:
62-
63-
```
64-
Host: localhost
65-
Port: 5005
66-
Use module classpath: song-client
67-
```
68-
69-
and in the `Before launch: Activate tool window` section, click the `+` sign, and select `Launch docker before debug`.
70-
Then ensure the `Docker configuration` field is set to the name of the previously created **docker image run profile** and that `Custom Options` is set to `-p 5005:5005`. In order for the debugger to bind to the debug port in time,
71-
a delay needs to be introduced after starting the container. To do this, click the `+` sign again, and select `Launch docker before debug`, and select `Run External Tool` and a window will pop-up. Input the following:
72-
73-
```
74-
Name: Sleep for 5 seconds
75-
Program: /usr/bin/sleep
76-
Arguments: 5
77-
```
78-
79-
and click `OK`.
80-
81-
Finally, start debugging by simply running the **remote debug profile** and it will call the **docker image run profile** before launch.
82-
83-
#### Executing the dockerized song-client in developement mode
84-
85-
The script `./docker/tools/song-client-dev` takes the arguments runs the `song-client` service entry specified in the `docker-compose.yml` with them. For example, to ping the song server, run `./docker/tools/song-client ping`
86-
87-
#### Debugging the song-server with IntelliJ
88-
89-
Since the `song-server` is a server and exposes the 5006 debug port, configuration is much easier. First, start the server with `make clean start-song-server`. Then, create a **remote debug profile** in Intellij with the following configuration:
90-
91-
```
92-
Host: localhost
93-
Port: 5006
94-
Use module classpath: song-server
95-
```
96-
97-
and then run it in debug mode.
98-
99-
### 2. Demo Mode
100-
101-
The purpose of this mode is to demo the current `song-server` and `song-client` code by building it in **inside the Docker image**,
102-
as opposed to the **Docker host** as is done in Developement mode and then running the containers.
103-
This mode will not run `mvn package` on the Docker host, but instead inside the Docker container.
104-
This method is very slow, since maven will download dependencies every time a build is triggered, however creates a completely isolated environment for testing.
105-
This mode can be enabled using the `DEMO_MODE=1` make variable override. For example, to start the song-server, the following command would be run:
106-
107-
#### Executing the dockerized song-client in demo mode
108-
109-
The script `./docker/tools/song-client-demo` takes the arguments runs the `song-client` service entry specified in the `docker-compose.yml` with them. For example, to ping the song server, run `./docker/tools/song-client ping`
110-
111-
```bash
112-
make start-song-server DEMO_MODE=1
113-
```
114-
115-
For more information on the different targets, run `make help` or read the comments above each target for a description
116-
117-
## API
118-
119-
The server provides swagger docs documenting the API.
120-
121-
When running locally they can be accessed here: http://localhost:8080/swagger-ui.html
122-
123-
## Dockerhub Configuration
124-
125-
1. Edit build configurations by selecting the `Builds` tab at the top, then click `Build Configuration`
126-
2. Create a new build rule by clicking the `+` sign beside the `BUILD RULES` text
127-
3. Edit the configuration as follows:
128-
Source Type: `branch`
129-
Source: `develop`
130-
Docker Tag: `develop`
131-
Dockerfile Location: `Dockerfile`
132-
Build Context: `/`
133-
`Autobuild` is set to the ON position
134-
135-
`Build Caching` is set to the ON position
136-
137-
4. Then save the configuration
138-
139-
### Docker Song Client
140-
141-
The `song-client` is a CLI tool used for communicating with a `song-server`.
142-
143-
#### Building
144-
145-
Simply running `mvn clean package` will package the client into a `-dist.tar.gz` file.
146-
147-
#### Configuration
148-
149-
After unarchiving the distribution, it can be configured via the `./conf/application.yml` file. Alternatively, the client can be configured through environment variables, which take presedence over the `application.yml` config.
150-
For example, to run the `song-client config` command using environment variables with the same values as the `application.yml` configuration below:
151-
152-
```yaml
153-
client:
154-
serverUrl: http://localhost:8080
155-
studyId: ABC123-CA
156-
programName: sing
157-
debug: true
158-
accessToken: myAccessToken
159-
```
160-
161-
could be done via:
162-
163-
```bash
164-
CLIENT_SERVER_URL=http://localhost:8080 \
165-
CLIENT_STUDY_ID=ABC123-CA \
166-
CLIENT_PROGRAM_NAME=sing \
167-
CLIENT_DEBUG=true \
168-
CLIENT_ACCESS_TOKEN=myAccessToken \
169-
./bin/sing config
170-
```
171-
172-
#### Running the client locally
173-
174-
The `song-client` can be run using the `./bin/sing` script.
18+
Technical resources for those working with or contributing to the project are available from our official documentation site, the following content can also be read and updated within the `/docs` folder of this repository.
17519

176-
#### Running the client using Docker
20+
- **[Song Overview](https://docs.overture.bio/docs/core-software/Song/overview)**
21+
- [**Setting up the Development Enviornment**](https://docs.overture.bio/docs/core-software/Song/setup)
22+
- [**Common Usage Docs**](https://docs.overture.bio/docs/core-software/Song/setup)
17723

178-
Alternatively, the `song-client` can be run using docker. To run the dockerized client with the configurations above, the following command could be executed:
24+
## Development Environment
17925

180-
```bash
181-
docker run --rm \
182-
-e 'CLIENT_SERVER_URL=http://localhost:8080' \
183-
-e 'CLIENT_STUDY_ID=ABC123-CA' \
184-
-e 'CLIENT_PROGRAM_NAME=sing' \
185-
-e 'CLIENT_DEBUG=true' \
186-
-e 'CLIENT_ACCESS_TOKEN=myAccessToken' \
187-
overture/song-client:latest \
188-
sing config
189-
```
26+
- [Java 11 (OpenJDK)](https://openjdk.java.net/projects/jdk/11/)
27+
- [Maven 3.5+](https://maven.apache.org/) (or use provided wrapper)
28+
- [VS Code](https://code.visualstudio.com/) or preferred Java IDE
29+
- [Docker](https://www.docker.com/) Container platform
19030

191-
By default, the `song-client` is run as the root user. To run as a non-root user, add the switch `-u song` which will run the command as a predefined `song` user:
31+
## Support & Contributions
19232

193-
```bash
194-
docker run --rm \
195-
-u song \
196-
-e 'CLIENT_SERVER_URL=http://localhost:8080' \
197-
-e 'CLIENT_STUDY_ID=ABC123-CA' \
198-
-e 'CLIENT_PROGRAM_NAME=sing' \
199-
-e 'CLIENT_DEBUG=true' \
200-
-e 'CLIENT_ACCESS_TOKEN=myAccessToken' \
201-
overture/song-client:latest \
202-
sing config
203-
```
33+
- For support, feature requests, and bug reports, please see our [Support Guide](https://docs.overture.bio/community/support).
20434

205-
or run it as your current user:
35+
- For detailed information on how to contribute to this project, please see our [Contributing Guide](https://docs.overture.bio/docs/contribution).
20636

207-
```bash
208-
docker run --rm \
209-
-u $(id -u):$(id -g) \
210-
-e 'CLIENT_SERVER_URL=http://localhost:8080' \
211-
-e 'CLIENT_STUDY_ID=ABC123-CA' \
212-
-e 'CLIENT_PROGRAM_NAME=sing' \
213-
-e 'CLIENT_DEBUG=true' \
214-
-e 'CLIENT_ACCESS_TOKEN=myAccessToken' \
215-
overture/song-client:latest \
216-
sing config
217-
```
37+
## Related Software
21838

219-
Running as the host user is useful when the `song-client` needs to write to a mounted volume
39+
The Overture Platform includes the following Overture Components:
22040

221-
#### Outputting data from the song-client via Docker
41+
</br>
22242

223-
Some song-client commands (such as `sing manifest` and `sing export`) output a file to a path.
224-
When running the docker container, it maybe preferable to output the file to the docker host's filesystem, instead of the containers file system.
225-
To do this, a directory from the docker host must be mounted into the song-client docker container.
226-
227-
For example, the following command will generate a manifest file called `output-manifest.txt` in the directory `./mydir`:
43+
|Software|Description|
44+
|---|---|
45+
|[Score](https://github.com/overture-stack/score/)| Transfer data to and from any cloud-based storage system |
46+
|[Song](https://github.com/overture-stack/song/)| Catalog and manage metadata associated to file data spread across cloud storage systems |
47+
|[Maestro](https://github.com/overture-stack/maestro/)| Organizing your distributed data into a centralized Elasticsearch index |
48+
|[Arranger](https://github.com/overture-stack/arranger/)| A search API with reusable search UI components |
49+
|[Stage](https://github.com/overture-stack/stage)| A React-based web portal scaffolding |
50+
|[Lyric](https://github.com/overture-stack/lyric)| A model-agnostic, tabular data submission system |
51+
|[Lectern](https://github.com/overture-stack/lectern)| Schema Manager, designed to validate, store, and manage collections of data dictionaries. |
22852

229-
```bash
53+
If you'd like to get started using our platform [check out our quickstart guides](https://docs.overture.bio/guides/getting-started)
23054

231-
# Ensure the current user owns mydir inorder to write to it from within the container
232-
mkdir -p ./mydir
55+
## Funding Acknowledgement
23356

234-
docker run --rm \
235-
-u $(id -u):$(id -g) \
236-
-v $PWD/mydir:/data \
237-
-e 'CLIENT_SERVER_URL=http://localhost:8080' \
238-
-e 'CLIENT_STUDY_ID=ABC123-CA' \
239-
-e 'CLIENT_PROGRAM_NAME=sing' \
240-
-e 'CLIENT_DEBUG=true' \
241-
-e 'CLIENT_ACCESS_TOKEN=myAccessToken' \
242-
overture/song-client:latest \
243-
sing manifest -a someAnalysisId -d /data -f /data/output-manifest.txt
244-
```
245-
246-
### Notes
247-
248-
When running with the secure profile enabled, an oauth2 server is needed.
249-
250-
Test
57+
Overture is supported by grant #U24CA253529 from the National Cancer Institute at the US National Institutes of Health, and additional funding from Genome Canada, the Canada Foundation for Innovation, the Canadian Institutes of Health Research, Canarie, and the Ontario Institute for Cancer Research.

0 commit comments

Comments
 (0)