Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
name: Deploy documentation

on:
push:
branches: [main]
paths:
- "website/**"
- ".github/workflows/docs.yml"
pull_request:
paths:
- "website/**"
- ".github/workflows/docs.yml"
workflow_dispatch:

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: docs-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
name: Build Astro site
runs-on: ubuntu-latest
defaults:
run:
working-directory: website
steps:
- name: Check out repository
uses: actions/checkout@v6

- name: Configure GitHub Pages
if: github.event_name != 'pull_request'
uses: actions/configure-pages@v5
with:
enablement: true

- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
cache-dependency-path: website/package-lock.json

- name: Install dependencies
run: npm ci

- name: Build site
run: npm run build

- name: Upload Pages artifact
if: github.event_name != 'pull_request'
uses: actions/upload-pages-artifact@v3
with:
path: website/dist

deploy:
name: Deploy to GitHub Pages
if: github.event_name != 'pull_request'
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
156 changes: 101 additions & 55 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,74 +1,120 @@
### What is `seqproc`?
# seqproc

[![Fast CI](https://github.com/COMBINE-lab/seqproc/actions/workflows/actions.yml/badge.svg)](https://github.com/COMBINE-lab/seqproc/actions/workflows/actions.yml)
[![Comprehensive CI](https://github.com/COMBINE-lab/seqproc/actions/workflows/comprehensive.yml/badge.svg)](https://github.com/COMBINE-lab/seqproc/actions/workflows/comprehensive.yml)
[![Documentation](https://github.com/COMBINE-lab/seqproc/actions/workflows/docs.yml/badge.svg)](https://combine-lab.github.io/seqproc/)
[![License: BSD-3-Clause](https://img.shields.io/badge/license-BSD--3--Clause-blue.svg)](LICENSE)

`seqproc` is a performance-oriented FASTQ preprocessing engine for single-cell
and other structured sequencing data. A compact geometry describes where
barcodes, UMIs, biological reads, anchors, and discarded sequence occur;
`seqproc` compiles that geometry into a multithreaded transformation pipeline.

This keeps protocol logic out of ad hoc scripts while supporting fixed and
variable intervals, approximate matching, barcode correction, filtering,
orientation-aware processing, demultiplexing, ordered output, compressed I/O,
and versioned run summaries.

- **Documentation:** <https://combine-lab.github.io/seqproc/>
- **EFGDL language specification:** <https://efgdl-spec.readthedocs.io/>
- **Preprint:** <https://www.biorxiv.org/content/10.64898/2026.07.28.741211v1>
- **Reproducible paper analysis:** <https://github.com/COMBINE-lab/seqproc-paper-analysis>

## A first geometry

The following geometry describes the common 10x Chromium v2 layout: the first
FASTQ contains a 16-base cell barcode followed by a 10-base UMI, and the second
contains the biological read.

```efgdl
bc = b[16]
umi = u[10]
bio = r:

1{<bc><umi>}
2{<bio>}
-> 1{<bc><umi>} 2{<bio>}
```

`seqproc` is a program for interpreting and transforming sequencing data according to the extended fragment geometry description language (EFGDL). In fact, the `seqproc` executable is a rather thin wrapper around the underlying `seqproc` library, whose main purpose is to accept an input stream of reads (consisting of single-end or paired-end reads) and an EFGDL specification, and to transform the input reads into the desired output formation according to the specification.
Save it as `10x-v2.geom`, validate it, inspect the compiled representation, and
run it:

Thus the important information regarding `seqproc` actually pertains to the EFGDL, which has its own documentation. For detailed documentation describing EFDGL, please visit the EFGDL specification documentation [here](https://efgdl-spec.readthedocs.io/en/latest/)
```console
seqproc validate 10x-v2.geom
seqproc explain 10x-v2.geom
seqproc run --geom 10x-v2.geom \
--file1 reads_R1.fastq.gz --file2 reads_R2.fastq.gz \
--out1 processed_R1.fastq.gz --out2 processed_R2.fastq.gz \
--threads 8
```

### Using `seqproc`
Output paths should be supplied explicitly; an omitted primary output is
discarded rather than written to standard output. See the
[quick start](https://combine-lab.github.io/seqproc/getting-started/quick-start/)
and [command-line reference](https://combine-lab.github.io/seqproc/getting-started/command-line/)
for paired-end, compressed-I/O, demultiplexing, and reporting examples.

The CLI provides `validate`, `explain`, and `run` subcommands. The legacy
flag-only invocation remains available during the compatibility window.
## Install from source

Tagged binary releases are planned. During the pre-release phase, build the
pinned dependency set from source with Rust 1.88 or newer:

```console
seqproc validate protocol.geom
seqproc explain protocol.geom
seqproc run --geom protocol.geom --file1 reads_R1.fastq.gz \
--file2 reads_R2.fastq.gz --out1 clean_R1.fastq.gz \
--out2 clean_R2.fastq.gz --threads 8 --preserve-order
git clone https://github.com/COMBINE-lab/seqproc.git
cd seqproc
cargo build --release --locked
./target/release/seqproc --help
```

`--summary report.json` runs the same processing pipeline and emits the
versioned schema documented in [`schemas/`](schemas/). Statistics are disabled
unless a summary is requested. Summaries default to detailed statistics;
`--statistics-level basic` records input, output, and rejection totals without
read-length or per-match distributions, while `--statistics-level detailed`
also records read lengths, ordered match-stage attrition, edit-distance
distributions, and ambiguity-policy outcomes. Performance results should use
statistics-disabled runs as the headline measurement and report a separate
paired on/off overhead experiment.

### Compressed I/O

Gzip level 3 is the measured speed/size default. Two opt-in parallel output
backends cover different interoperability and performance requirements:

- `--parallel-gzip` compresses read batches on transform workers and emits a
concatenated multi-member gzip file. This is typically the fastest choice,
but consumers must support concatenated members.
- `--parallel-gzip-stream` emits one logical gzip member with dictionary
continuity across read batches. Its compression pool defaults to
`min(--threads, 4)`; use `--gzip-threads` and `--gzip-block-size` to tune it.

FASTQ input is parsed by `needletail`, which also provides transparent
decompression by default. `--accelerated-gzip-input` instead feeds `needletail`
from `rapidgzip-core`'s speculative decoder for regular `.gz` files while
retaining the same transformation graph. Decoder workers are created adaptively
up to the `--gzip-input-threads` ceiling; real-data profiling currently favors the
default ceiling of one worker per input when transform workers share a fixed
CPU allocation. `--gzip-input-chunk-size` controls decoded handoff chunks, with
a measured 256-KiB default. Plain input files are unchanged by the option.

Run `seqproc run --help` for the complete set of pipeline, ordering,
demultiplexing, and compressed-I/O options.

### Ambiguous barcode matches
## Ambiguous barcode matches

Equal-best matches against distinct whitelist or mapping entries use an
operation-specific default: filters accept set membership, while mapping
operations conservatively follow their no-match fallback. Override this with a
typed property annotation on the definition:
operations follow their no-match fallback. A geometry can select an explicit
policy:

```text
```efgdl
#[ambig_policy = accept]
bc3 = filter_within_dist(b[8], "barcodes.txt", 1)

#[ambig_policy = quality(min_delta = 2)]
bc = map_with_mismatch(b[8], "barcode-map.tsv", self, 1)
```

Supported values are `accept`, `no_match`, `first`, `random`, `quality`, and
`error`. `random(seed = N)` is deterministic across thread schedules;
`quality(min_delta = N)` selects the candidate whose mismatching positions
have the lowest summed Phred score and requires the specified advantage over
the runner-up. Exact duplicate input rows are removed before matching;
conflicting duplicate mapping rows are configuration errors.
Supported policies are `accept`, `no_match`, `first`, `random`, `quality`, and
`error`. The [ambiguity guide](https://combine-lab.github.io/seqproc/efgdl/annotations-and-ambiguity/)
documents their semantics and reproducibility guarantees.

## Development and reproducibility

Fast pull-request CI runs formatting, linting, core tests, and generated test
code using cached compiler outputs. Scheduled and release CI runs the complete
test, feature, benchmark-compilation, and sanitizer matrix.

```console
cargo fmt --all --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-targets --all-features
```

The documentation site requires Node.js 22.12 or newer and has its own locked
build:

```console
cd website
npm ci
npm run build
```

The JSON emitted by `--summary` follows the versioned schemas in
[`schemas/`](schemas/). Runtime statistics are disabled unless requested, so
headline performance measurements do not silently include instrumentation.

Please report bugs and feature requests through
[GitHub Issues](https://github.com/COMBINE-lab/seqproc/issues).

## Citation and license

Until a version of record is available, please cite the
[seqproc preprint](https://www.biorxiv.org/content/10.64898/2026.07.28.741211v1).
`seqproc` is distributed under the [BSD 3-Clause license](LICENSE).
37 changes: 0 additions & 37 deletions docs/index.md

This file was deleted.

17 changes: 0 additions & 17 deletions mkdocs.yml

This file was deleted.

3 changes: 3 additions & 0 deletions website/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
node_modules/
dist/
.astro/
24 changes: 24 additions & 0 deletions website/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# seqproc documentation site

This directory contains the Astro Starlight site published at
<https://combine-lab.github.io/seqproc/>.

Node.js 22.12 or newer is required. Install the locked dependencies and start a
local development server with:

```console
npm ci
npm run dev
```

Build the same static artifact used by GitHub Pages with:

```console
npm run build
```

Documentation pages live under `src/content/docs/`. Navigation is explicit in
`astro.config.mjs`; add a new page there as well as in the content tree.

The deployment workflow builds pull requests that modify `website/` and
deploys changes merged to `main` through GitHub Pages.
Loading
Loading