Pipelines¶

A pipeline file is not a script — it is a set of variable assignments and params_set calls that wrike_job.sh sources:

Variable	Required	Purpose
`NEXTFLOW_ARGS`	yes	Bash array; the full nextflow command line, already split
`PIPELINE_NAME`	yes	Exact version recorded on the Wrike task, e.g. `ampliseq_01`
`PARAMS_FILE`	no	Where `wrike_job.sh` writes the params file, e.g. `ampliseq_args.yaml`
`PARAMS_LOCKED`	no	Bash array; parameter names a requester may not override
`PRE_PROCESS_CMDS`	no	Bash array; commands run in order before nextflow, in the run directory
`POST_PROCESS_CMDS`	no	Bash array; commands run in order after nextflow succeeds. The last one publishes the results

PIPELINE_NAME upper-cased must name the file itself — ampliseq_01 → pipelines/AMPLISEQ_01.sh. That is how the run manifest records the version that ran rather than the shortcut that was asked for, which is what lets a rerun a year later reproduce it.

Steps on the progress page¶

Every pre-process command, the nextflow run, and every post-process command is a row on the live progress page, in the order they run. A command is listed under its script's name less the pipeline's prefix — taxprofiler_humann.sh is humann — and the nextflow run under the pipeline it runs, with nextflow's processes indented beneath it.

A step that drives a nextflow run of its own tees that run's console output to <name>.out in the run directory, and the page lists that run's processes under the step the same way. taxprofiler_humann.sh writes humann.out.

The last post-process command must be the one that publishes the results. The finished dashboard lands on the same S3 key the progress page is published to, so wrike_job.sh stops the page's watcher as that step begins, leaving a page that shows it under way. Every pipeline ends with its upload script.

Parameters¶

The pipeline declares its defaults with params_set, one call per parameter:

PARAMS_FILE="ampliseq_args.yaml"

params_reset
params_set input                 "ampliseq_samplesheet.tsv"
params_set outdir                "results"
params_set exclude_taxa          "mitochondria,chloroplast,Francisella"

wrike_job.sh writes the file after the pre-process stage, from three layers. Each overwrites only the keys it names, and later layers win:

the pipeline's own params_set defaults, including whatever it makes of the request form's answers
detected_params.yaml — anything a pre-process step measured from the data, such as the primers ampliseq_detect_region.sh found
rerun_params.yaml — a reproduced run's recorded parameters, which pin everything

Then every name in PARAMS_LOCKED is put back to the pipeline's own value, with a warning if something had changed it. Those name files the run itself creates: overriding input would point nextflow at a samplesheet nothing wrote, and overriding outdir would publish where the upload step does not look.

Values are scalars. true, false, null and numbers are written bare so nextflow types them; everything else is written as a quoted string. There are no YAML lists — nf-core takes a comma-separated string wherever this system needs more than one value.

The form's answers¶

A pipeline reads the request form with form_answer <key>, which returns an answer wrike_task_handler.sh has already checked against the list the form offers, or nothing when the question went unanswered.

Most fields are titled after the parameter they set — Ampliseq --dada_ref_taxonomy — so their key is the parameter name and applying them is a loop:

for AMPLISEQ_PARAM in dada_ref_taxonomy qiime_ref_taxonomy kraken2_ref_taxonomy exclude_taxa; do
    AMPLISEQ_ANSWER=$(form_answer "$AMPLISEQ_PARAM")

    if [[ -n "$AMPLISEQ_ANSWER" ]]; then
        params_set "$AMPLISEQ_PARAM" "$AMPLISEQ_ANSWER"
    fi
done

An unanswered question means the pipeline's own default, and that is the normal case: the form leaves an optional question off the task entirely when the requester takes Settings: Default, so there is nothing for the system to gate on and no Settings field to read.

The mapping lives in the pipeline, not in the handler, which knows no pipeline from another. Taxprofiler's one answer needs it — Human + PhiX becomes five parameters — and it is also what keeps answers versioned: this revision of ampliseq calls its primers primer_fwd and primer_rev where the last called them FW_primer and RV_primer, and only the pipeline file has to know which.

Each versioned pipeline pins its own nextflow arguments as well as its defaults. Nothing about the command line is defaulted by wrike_job.sh.

-r names a commit, not a tag or a branch. A branch moves by design and a tag can be moved by accident, and either would mean a rerun of a year-old run executing different code. The nf-core pipelines pin a SHA with a comment saying which release or branch it came from. BIOBAKERY runs a workflow from this repository and has no -r; see the biobakery pipeline.

Naming¶

Two names resolve to the same run, by chained source:

AMPLISEQ.sh  →  AMPLISEQ_01.sh      (the actual definition)

AMPLISEQ_01.sh is immutable once it has been used in production. To change parameters, add AMPLISEQ_02.sh and repoint AMPLISEQ.sh at it; old runs keep reproducing their original settings, and a rerun still names the version rather than the shortcut.

The request form's pipeline field is matched on its first word only — its options read ampliseq :: 16S full length or variable region amplicons — and case-insensitively, so ampliseq, Ampliseq and AMPLISEQ all resolve to pipelines/AMPLISEQ.sh. See the pipeline field.

Currently defined:

AMPLISEQ — nf-core/ampliseq against SILVA 138.2, pinned to a dev commit for its unreleased Oxford Nanopore support. One pipeline for every 16S library: the platform and the variable region are both measured from the reads rather than requested, and a samplesheet with no fastq_2 column is single-end. See the ampliseq pipeline.
TAXPROFILER — nf-core/taxprofiler 2.0.1 running kraken2, bracken, metaphlan and mOTUs over shotgun reads, with nonpareil measuring how much of each community was sequenced. Which host genome is depleted first is the form's "Host Removal" answer rather than a separate pipeline. See the taxprofiler pipeline.
BIOBAKERY — KneadData, MetaPhlAn 4.1.1 and HUMAnN 3.9 over shotgun reads, with mOTUs and nonpareil for diversity, Marker-MAGu for phages, and EsViritu for viruses when switched on, as a workflow in this repository, with HUMAnN and any add-on module switched on per pipeline version. Reads the same host answer as TAXPROFILER. See the biobakery pipeline.

Pipeline files are named in upper case. wrike_task_handler.sh uppercases whatever the user typed and looks for exactly that filename, so taxprofiler on the form resolves to pipelines/TAXPROFILER.sh and a lower-case file would never be found.

Reproducing an earlier run¶

Every run records its manifest as .manifest in run_state.json before nextflow starts, and the whole state file is published at nxf/<uid>/run_state.json alongside the results. It records the pipeline version, the nextflow command line, and every parameter as resolved — plus the sample count, which rides along because the published copy outlives the results it was published with: it is one of the few things an expired dashboard keeps.

It also records what a rerun years later would need and cannot look up then:

Key	What
`.manifest.code`	the commit `NEXTFLOW_DIR` was checked out at, the repository it came from, and the names of any tracked files changed since that commit
`.manifest.software`	the nextflow, Java and Apptainer versions outside the containers; the containers are pinned by tag in the modules, at that commit
`.manifest.databases`	every path under `NEXTFLOW_DB_DIR` a parameter names, or a file a parameter names lists (taxprofiler's database sheet), with its resolved path, file count and size, and the `<release>.manifest.json` its fetch or build script wrote — source URLs and checksums — copied in whole

A copy of the state file as it stood at upload is also in the results folder, so it is listed in the File Explorer and carried in the download zip.

A request that picks prev_run_id on the form and names a run in "Nextflow Previous Run ID" — or run --rerun <run_id> samples.txt — is handled by fetching that state file from S3, reading .manifest out of it, and using that in place of everything the request would otherwise decide, the host reference included. The requester supplies new samples; nothing else about the analysis changes. For ampliseq that also means region detection is skipped, since the primers are already fixed.

Runs published before the run directory moved to one state file carry the manifest as pipeline_manifest.json instead, which is tried second, so they can still be reproduced.

A run that never got as far as recording a manifest published none, and so cannot be reproduced; the request is rejected saying so.

Adding a pipeline¶

Write pipelines/<NAME>_01.sh setting the variables above, with PIPELINE_NAME="<name>_01".
Add pipelines/<NAME>.sh containing source "$NEXTFLOW_DIR/pipelines/<NAME>_01.sh".
Add <NAME> as an option to Wrike's "Nextflow Pipeline" field, and to WRIKE_FORM_ANSWERS.
Add docs/pipelines/<name>.md describing it, and list the page under Pipelines in mkdocs.yml.