Skip to contents

R package · Long-running R work

Run long R work across processes or a scheduler.

Dispatch one function over many items, each in its own worker process, and take the return values or the written output files back. Work too long for one session goes to Slurm instead: batchit writes the job chain. A second call submits it.

Get started Browse the reference →

batchit logo

What’s inside

01

A fresh process per item

One function runs once per item, up to n_workers at a time. Three of the four dispatch functions start a brand-new R process per item, because process exit is what reclaims memory.

02

Declared outputs, committed atomically

Declare each item’s final output paths. batchit stages them beside their destinations, renames them into place, and writes the marker last. A failed or interrupted item never leaves a half-written file at a final path.

03

Or hand the work to a scheduler

slurm_it() describes one Slurm job. slurm_write() turns a list of them into one bash file per job, plus a submit.sh that chains them with –dependency=afterok. slurm_write() submits nothing. slurm_submit() runs that driver, so you read submit.sh first.