Package index • batchit

Dispatch a function across subprocesses

Run one function over many items, in worker processes: a fresh one per item for the first three, a reused pool for the fourth. Each takes fn, either a package_function() descriptor (hash-verified) or a bare closure for all but the streaming function. They differ in how each item’s data arrives and in what comes back, a return value or declared output files that batchit stages and then commits.

run(): Run a function once per item, in a fresh worker process, discarding the results
run_and_collect(): Run a function once per item, in a fresh worker process, and collect the results
run_and_write_files_atomically(): Run a function once per item, in a fresh worker process, and have batchit write its output files safely
stream_from_parent_and_write_files_atomically(): Like run_and_write_files_atomically(), but build each item lazily instead of all at once

Building a dispatch

Helpers used when constructing a dispatch: name the target function (hash-verified against what the worker loads), and, inside a staged_writer target, resolve where to write a declared output.

package_function(): Identify a function in an installed package, so a worker can run it
where_to_write_output(): Get the path to write one declared output to, inside a "staged_writer" function

Writing one file atomically

A standalone atomic qs2 writer, usable on its own with no dispatch: it writes to a temporary file beside the destination and renames it into place, so the destination is never a truncated file. Independent of the run_and_write_files_atomically() commit engine.

write_qs2_atomically(): Atomically write an object to a qs2 file

Running a chain of Slurm jobs

The route for work too long to sit in one R session. slurm_it() validates one job description. slurm_write() turns a list of them into one bash file per job, plus the submit.sh driver, and it submits nothing. slurm_submit() runs that driver and returns the job ids, named by stage. slurm_status() reports what the live queue and the accounting database say about those jobs. inside_slurm_job() reports whether this R process runs inside a Slurm job, which is what a self-submitting script needs to know.

slurm_it(): Describe one Slurm job
slurm_write(): Write a Slurm job chain
slurm_submit(): Submit a written Slurm job chain
slurm_status(): Report what Slurm says about a job chain
inside_slurm_job(): Is this R process inside a Slurm job?