Operating it

Two things run on a schedule of their own: the daemon, continuously, and the expiration pass, once a day under a systemd timer. Both are user units on one login node.

The daemon is the only long-lived process. It never returns, and handles INT/TERM cleanly, so it runs under systemd as a user unit — systemd/wrike-sqs-listener.service. Nothing here needs root, and lingering is what keeps the unit alive once you log out. Installation, the environment the unit has to carry in place of module load slurm, and what to check when it will not start are in the daemon.

systemctl --user status wrike-sqs-listener

log and warn write to stdout and stderr, so the journal is the daemon log:

journalctl --user -u wrike-sqs-listener -f

Restarting picks up edits to .env and to the files it sources, both read once at startup; a change to a handler script needs nothing, since the daemon executes it fresh per message:

systemctl --user restart wrike-sqs-listener

The daily pass that retires dashboards past their expiration date logs the same way, under its own unit:

systemctl --user list-timers wrike-expiration.timer

journalctl --user -u wrike-expiration --since "7 days ago"

Slurm output goes to $NEXTFLOW_DIR/log/job_<uid>_<jobid>.out and followup_<uid>_<jobid>.out — one file per run, since --job-name is the uid, and one file per job rather than two, since --output and --error name the same path and both streams interleave there. Per-run state lives in $NEXTFLOW_DIR/tmp/<uid>/, which is created as soon as a request is picked up and removed only when its job succeeds or its Wrike task goes away: a directory still present after a run ended means the run failed or the request was never accepted, and run_state.json and nextflow.log there say why — the state file's .status and .message carrying the account of it, and its .wrike.task_id naming the Wrike task it came from, since the uid does not lead back to one.

Every run that ends also leaves one line in $NEXTFLOW_DIR/log/run_history.tsv, written by wrike_followup.sh whether the run succeeded or failed: when it ended, its uid, the pipeline version, its outcome, how many samples and decimal gigabytes of FASTQ it was given, and the allocated cpu hours all of its Slurm jobs held. The FASTQ is measured as the run staged it in raw-sequences/, and the cpu time is read from sacct once every job of the run has finished — so it includes the steps after nextflow, HUMAnN among them, which the progress page's last clock does not. It is the record an estimate of how far along a running job is gets fitted to; keep the rows whose outcome is Completed.