Skip to contents

Resolve the default number of parallel workers for a pipeline stage

Source: R/default_n_workers.R
default_n_workers.Rd

Used as the default for `n_workers` across the heavy pipeline steps. Each step passes its own `stage` tag so worker counts can be tuned independently — a memory-heavy stage never inherits a box-wide setting meant for a light one. Resolution, in order:

  1. `getOption("swereg.n_workers.<stage>")`

  2. the `SWEREG_N_WORKERS_<STAGE>` environment variable (stage upper-cased)

  3. fallback `1L` (serial)

The default is **1 worker everywhere**; parallelism is opt-in, per stage. For example, to run the enrollment/IPW loop with 3 workers but keep the analysis loop serial, set `SWEREG_N_WORKERS_S1=3` and leave `SWEREG_N_WORKERS_S3` unset. Pass `n_workers` explicitly at the call site to override for a single run.

Usage

default_n_workers(stage = NULL)

Arguments

stage

Optional character stage tag (e.g. `"s1"`, `"s3"`, `"skeleton"`, `"rawbatch"`). When `NULL`, no per-stage override is consulted and the function returns `1L`.

Value

Integer worker count (>= 1).

Details

Stage tags currently used by the pipeline:

`"rawbatch"`

[RegistryStudy]'s `save_rawbatch()`

`"skeleton"`

[RegistryStudy]'s `process_skeletons()`

`"s1"`

[TTEPlan]'s `s1_generate_enrollments_and_ipw()` (~6 GB/worker)

`"s3"`

[TTEPlan]'s `s3_analyze()` — peaks ~20 GB/worker on large "vs none" panels, so keep at 1 unless you have the RAM headroom

Note: `s2_generate_analysis_files_and_ipcw_pp()` deliberately does NOT use this — it stays single-worker (`n_workers = 1L`) for per-ETT memory isolation.

The former box-wide `SWEREG_N_WORKERS` (and `getOption("swereg.n_workers")`) are retired: a single global knob could silently leak a high worker count into a heavy stage (3 x ~20 GB s3 workers -> OOM). If the deprecated `SWEREG_N_WORKERS` is still set, a one-time warning is emitted.