Caching and resume: what invalidates what

A full run of this pipeline takes days, so the skeleton side of it skips work that it already did. It keeps three caches and one commit record. Each has different rules about what makes it stale.

A wrong rule fails silently. A stale cache that swereg does not discard gives a run that completes normally and reports nothing unusual. Part of the result then comes from data you replaced. This vignette lists what is cached, what invalidates it, and what none of it protects you from.

The TTE stages (s1 to s3) have no caching or resume logic of their own. s1 uses its work directory only to pass data between its internal sub-steps s1a to s1e. s1 clears that directory at the start of every call and deletes it again on success. s3_analyze() recomputes every targeted result on every call. A killed s1 restarts from zero, and an s3 crash during a run means a full recompute. These costs are accepted by design. This vignette therefore covers only the skeleton pipeline below s1, where all of the caching is.

Read vignette("skeleton-pipeline") first for how swereg builds skeletons, and vignette("tte-workflow") for what s1 to s4 do.

The layers

layer	what it avoids	invalidated by
1. Rawbatch staging	re-reading tens of GB of raw registry files	nothing automatic: swereg checks only that the files exist
2. Skeleton phases	rebuilding 2,000+ skeletons for a one-line change	the `body`/`formals` hash of a changed function, or a changed code registry fingerprint
3. Meta sidecars + derived artifacts	re-reading whole skeletons to answer provenance and population questions	rewritten with their batch
Skeleton manifest	(a commit record, not a cache)	cleared before every `$process_skeletons()`, recommitted only if the result validates

Layer 1: rawbatch staging

$save_rawbatch() writes the raw registries into per-batch .qs2 files. An existence check guards this step, not a hash. When the rawbatches and registrystudy.qs2 exist, a reclean skips staging and does not re-read the raw delivery.

This is a deliberate trade. Re-staging is expensive and the raw delivery is a fixed artifact, so the guard is almost always right. If the raw files change, nothing here detects it. You must delete the rawbatches (or the meta file) by hand.

study$delete_rawbatches()   # force a re-stage
study$delete_meta_file()

Layer 2: skeleton phases

$process_skeletons() does not rebuild skeletons. It replays the phases that changed. Each batch’s skeleton carries its own provenance, and the decision is per phase:

Phase 1 (framework). If the hash of framework_fn differs from the one stored on the skeleton, or no skeleton exists, swereg rebuilds the base from scratch. It resets every later phase with it.
Phase 1b (trim). The stored trim_fn_hash and phase_order join the framework hash in that same gate. Each of these rebuilds the base: a trim edit, a new trim, a removed trim, or a swereg version with a different phase order. The trim then runs on that fresh base, and on nothing else.
Phase 2 (codes). $sync_with_registry() diffs the stored entry fingerprints against the current ones. It drops the columns of every entry that left the registry and applies every entry that joined it. An untouched entry is kept.
Phase 3 (randvars). $sync_randvars() walks the registered functions in order and finds the divergence point: the first position where the name or hash differs from what the skeleton recorded. Everything from there on is dropped and replayed; everything before it is kept. If nothing diverges, the batch is untouched.

Each randvars hash folds in the framework hash, the trim identity and every code registry fingerprint. So any code registry edit moves every randvars hash, and the whole sequence replays against the freshly-applied code columns.

The rewind removes columns. It does not restore them. $sync_randvars() drops only the columns each step recorded itself as adding. It cannot restore a column that an earlier step overwrote. Nothing checks for that, so a step that changes an existing column breaks the replay silently.

swereg does check rows. A code entry or a randvars step that changes the row count stops the run. The error names the registration to edit. Only the phase-1b trim may delete rows.

The hash of a registered function is digest(list(body(fn), formals(fn))). This has two consequences.

A comment change does not trigger a replay. Documentation, changelog entries and comment reformatting do not change body(). You can fix a comment in a randvars file while a regeneration runs.

The hash covers code, not data. It fingerprints the registered functions. It says nothing about called helpers, package versions, globals, or the raw inputs. If you change the raw data and keep the code, every hash stays the same. Layer 2 then concludes that it has nothing to replay, and that conclusion is wrong.

The skeleton manifest below depends on this layer. It records which generation layer 2 produced, so a reader of the manifest sees every rebuild. It does not help in the case above. If layer 2 never rewrites the batch, there is no new generation to see. For changed raw data, you must invalidate layer 1 by hand.

The skeleton manifest (not a cache)

A manifest is the commit record of $process_skeletons(). It attests that, at commit time:

every batch’s provenance sidecar was readable
the set was complete and internally uniform
the set matched the study’s pipeline fingerprint

It does not prove that the skeleton bytes match their sidecars, that the raw inputs were current, or that any content is correct.

It lives in the skeleton_manifest field of RegistryStudy. $save_meta() writes it into registrystudy.qs2, so it needs no extra file.

study <- registrystudy_load("/path/to/meta/dir")
str(study$skeleton_manifest)
#> List of 7
#>  $ manifest_version: int 1
#>  $ committed_at    : POSIXct "2026-07-17 04:51:12"
#>  $ swereg_version  : chr "26.7.19"
#>  $ n_batches       : int 2194
#>  $ batches         : int [1:2194] 1 2 3 4 5 ...
#>  $ pipeline_hash   : chr "9e079c0876398c90"
#>  $ identity        : chr "73180d2cd7417e91"

pipeline_hash fingerprints the registered pipeline code. identity is a generation token derived from per-batch provenance metadata. It digests the ordered per-batch (batch, pipeline_hash, built_at) triples. Neither is a content hash of the skeleton data.

built_at makes identity a generation token and not a code token. When swereg rewrites a batch and its provenance, the built_at of that batch changes. identity then moves, although the code did not. An incremental run that skips every current batch rewrites nothing, so identity stays the same.

The commit protocol

swereg clears the manifest before it touches a single batch. It recommits the manifest at the end only if the finished dataset passes these checks:

Provenance readable for every batch.
Exactly one distinct pipeline_hash. More than one means that an interrupted replay left a mix of old and new skeletons.
That hash equals the study’s current $pipeline_hash(). A dataset can be fully self-consistent and still out of date.
Batch IDs are exactly seq_len(expected). A count cannot tell 1..N from 2..N+1. A first build interrupted at batch 272 leaves 272 mutually consistent skeletons, and a hash-only check accepts them. s1 then analyses 12% of the cohort and shows no sign of a problem.

The clear-first order makes a kill safe. A manifest that survived an interrupted run would vouch for skeletons it no longer describes. That is worse than no manifest.

batches controls only which batches a run processes. Validation always covers the whole directory, because s1 reads the whole directory. A subset run (batches = 1:10) therefore still commits if the resulting whole dataset validates. A full run (batches = NULL) changes only the failure behaviour. It raises an error, so a caller’s file-count gate cannot report success over a dataset that nothing will accept.

What none of this protects you from

Concurrency. Exactly one $process_skeletons() may run against a skeleton directory at a time. Nothing enforces this. Two writers can interleave as clear(A), clear(B), commit(A), then B replaces skeletons and dies. A’s manifest then vouches for B’s skeletons. Atomic writes cannot fix a logical race. Run one regeneration at a time.
Identity is not content. Nothing here digests skeleton bytes. If you edit, replace, restore from backup or corrupt a skeleton without regenerating its provenance, swereg may not detect it. The stale-sidecar case below is one instance of this general limit.
A skeleton replaced under its own sidecar. Validation reads sidecars, not skeletons, because opening all skeletons means reading gigabytes every run. Both writes are atomic, but they are separate writes. A crash between them can leave a new skeleton with a stale sidecar.
Untracked dependencies. A function-body fingerprint does not cover called helpers, package versions, globals, environment variables, external files, time, or randomness.
Raw data that changes without forcing a rewrite. Layer 1 is guarded on existence and layer 2 on code identity, so unchanged code over changed raw inputs invalidates nothing. Delete the rawbatches by hand.
Closure-captured state, anywhere. Functions are hashed by body and formals, so two closures that differ only in captured values get the same key. This applies to framework_fn, trim_fn and the randvars steps.
Phase-2 code registry entries are fingerprinted by their declared codes and arguments and by the body of the function that applies them, but not by anything that function calls.
Adversarial integrity. These are short cache identifiers, not cryptographic authenticity guarantees.
Scientific correctness. All of this checks identity and completeness. It says nothing about whether the contents are right.

Recovery: what to do

symptom	do this
Regeneration was killed	Run a full `$process_skeletons()` (`batches = NULL`) to completion. A cleared manifest after a kill is the protocol at work.
Raw delivery changed, code did not	Nothing detects this automatically. Run `$delete_rawbatches()`, `$delete_skeletons()`, `$delete_meta_file()`, then re-run the full generator.
“Is my skeleton dir mixed?”	Run `study$skeleton_pipeline_hashes()`. You want one distinct `pipeline_hash` and no `NA`s. Compare it to `study$pipeline_hash()` to see whether it is current as well as uniform.
s1’s work directory survived a killed run	Nothing. The next `$s1_generate_enrollments_and_ipw()` call clears it at start.

A worked example

This example comes from a production run on 2026-07-17. It shows layers 1 to 3 and the manifest commit.

A regeneration was killed the afternoon before. It left the skeleton directory mixed: 272 batches rebuilt with new code, 1,922 still on the old code.

                    before                      after
skeleton state      272  @ 9e079c08 (new)       2194 @ 9e079c08
                    1922 @ b4b325e5 (old)       1 distinct hash, 0 unreadable
manifest            NULL                        identity 73180d2cd7417e91

The rerun:

Layer 1 skipped raw staging (the rawbatches and the meta file existed).
Layers 2 and 3 reached 272/2194 in 56 seconds. The layer 3 meta sidecars showed the pipeline of those batches already current. Layer 2 skipped them without loading the skeleton, so the rerun kept the work of the killed run. Layer 2 then rebuilt the remaining 1,922 in 7h 35m.

The skeleton manifest validated and committed:

Skeleton manifest committed: 2194 batches, pipeline 9e079c0876398c90,
                             identity 73180d2cd7417e91