Manages the full skeleton pipeline lifecycle: portable batch directories, batch splitting, raw registry loading, the declarative code registry, and the orchestrated per-batch processing (framework -> trim -> codes -> randvars) that produces one Skeleton file per batch with incremental invalidation.
Portable Directory Resolution
Directories are stored as candidate path vectors and resolved lazily via CandidatePath active bindings. The first existing directory wins and is cached. If the cached path becomes invalid (e.g. after moving to a different machine), the binding automatically re-resolves from the candidate list.
Per-batch pipeline
$process_skeletons() runs four phases per batch, with per-phase
incremental invalidation so editing one step only re-runs what it
affects:
- Phase 1 – framework
A single user function registered via
$register_framework(fn), signature(batch_data, config), returns a fresh basedata.table(time grid + structural censoring). Full rebuild onbody(fn)/formals(fn)hash change.- Phase 1b – trim
At most one user function registered via
$register_trim(fn), signature(skeleton, batch_data, config), returning adata.table. It is the one declared place in the pipeline that MAY delete skeleton rows. It runs immediately after the framework and before the code registry, so every code entry sees the final row set. A change to the registered trim rebuilds the base, because a deletion cannot be rewound. It runs only on a rebuild, so it always reads a fresh base.- Phase 2 – codes
The declarative code registry, built via
$register_codes()(primary) and$register_derived_codes()(derived). Per-entry fingerprint diff: entries no longer present are dropped, new or modified entries are freshly applied. Derived entry fingerprints fold in their upstream primary fingerprints so upstream behavior edits cascade correctly.- Phase 3 – randvars
An ordered named list of user functions registered via
$register_randvars(name, fn), each signature(skeleton, batch_data, config). Divergence-point rewind-and-replay invalidation: the first step whose name or hash differs from the stored sequence triggers a drop of its columns and replay of it plus everything downstream of it. Add/remove/edit/reorder all handled uniformly.
Phase 2 runs BEFORE phase 3, so a phase-3 step MAY read a phase-2
column. $randvars_hashes() folds the code registry fingerprints into
every step's hash, so a registry edit replays every step against the
new columns. See the Skeleton class for the on-disk provenance
format.
The order is recorded on each Skeleton as phase_order. A skeleton
written under the old order carries NULL there, and
$process_skeletons() rebuilds it once.
Only phase 1b may delete rows. A phase-3 step that filters rows breaks
the rewind-and-replay contract, because rewind drops columns and cannot
restore rows. Move any such filter into $register_trim().
Code Registry
Primary entries are registered via $register_codes(), which declares
codes, the function to apply them (e.g. add_diagnoses, add_cods),
which rawbatch groups to use, and optional prefixing/combining.
Derived entries are registered via $register_derived_codes() and
OR together already-existing skeleton columns from upstream primary
entries – useful when the combined column needs to draw from
registrations that use DIFFERENT fns (something combine_as can't
express because it re-runs the same fn on rbound data).
See also
Skeleton for the per-batch on-disk format and provenance
fields; CandidatePath for the multi-host directory resolution
mechanism; add_diagnoses, add_cods, add_rx for common fn
choices in $register_codes().
Other skeleton_pipeline:
Skeleton
Public fields
group_namesCharacter vector. Names of rawbatch groups.
batch_sizeInteger. Number of IDs per batch.
seedInteger. Shuffle seed for reproducibility.
id_colCharacter. Person ID column name.
n_idsInteger. Total number of IDs across all batches.
n_batchesInteger. Number of batches.
batch_id_listList of ID vectors, one per batch.
batch_composition_digestCharacter or NULL. The
digest::digest()ofbatch_id_list, and ofmatched_set_rosterwhen it is set.$set_ids(),$set_batches()and$set_matched_sets()set it. A study saved by swereg 27.2.6 or earlier has none.matched_set_rosterList of data.tables, one per batch, or NULL. Each holds the roster rows of that batch, in roster row order.
$set_matched_sets()sets it.$set_ids()and$set_batches()set it to NULL.groups_savedCharacter vector of rawbatch groups saved to disk.
code_registryList of code registration entries, appended to by
$register_codes()and$register_derived_codes(). Primary entries (from$register_codes()) are plain lists with:codes, fn, fn_args, groups, combine_as, label. Derived entries (from$register_derived_codes()) are tagged withkind = "derived"and holdcodes, from, as, labelinstead – nofn, nogroups, no raw data access. The dispatcher.apply_code_entry_impl()branches on the entry'skindfield, defaulting to"primary"when absent.created_atPOSIXct. Timestamp when this study was created.
data_rawbatch_cpCandidatePath for the rawbatch directory.
data_skeleton_cpCandidatePath for the skeleton directory.
data_meta_cpCandidatePath for the metadata directory (holds
registrystudy.qs2). Defaults to the rawbatch directory for backward compatibility.data_raw_cpCandidatePath for the raw-registry directory, or NULL if not configured.
data_summaries_cpCandidatePath for the audit-track summaries directory (git-tracked TSV per full run), or NULL if the feature is not configured. When NULL,
$compute_summary()still writes the localsummary.qs2andstatus.txtbut skips the TSV.framework_fnFunction of signature
(batch_data, config)returning a fresh base skeletondata.table(phase 1). Set via$register_framework().$process_skeletons()re-runs this function per batch when its body/formals hash changes.trim_fnFunction of signature
(skeleton, batch_data, config)returning adata.table(phase 1b), or NULL. Set via$register_trim(). It is the one place in the pipeline that may delete skeleton rows.randvars_fnsNamed ordered list of phase-3 functions, each with signature
(skeleton, batch_data, config). Populated via$register_randvars(name, fn). Registration order is execution order.$process_skeletons()usesSkeleton$sync_randvars()'s divergence-point rewind-and-replay to apply changes incrementally.skeleton_manifestList, or NULL. The commit record for the skeleton dataset: proof that
$process_skeletons()ran to completion and produced a complete, uniformly-built set of batches, plus an identity for exactly which batches those were and what built them. Written toregistrystudy.qs2by$process_skeletons(), atomically, via$save_meta().NULL means "no trustworthy skeleton dataset".
$process_skeletons()clears it before doing any work and only re-commits it at the end if the result validates, so a killed or partially-failed run leaves NULL rather than a stale record vouching for skeletons it no longer describes. Downstream stages that need to know what they are reading (s1) must load this from disk rather than from an embedded study copy, which is frozen at the time the plan was saved.Fields:
committed_at,swereg_version,n_batches,batches(the exact sorted batch IDs – a count alone cannot tell batches 1..N from 2..N+1),identity_hash(shared by every batch) andidentity(a digest over the ordered per-batch (batch, identity_hash, saved_at) triples, so it moves when any batch is rebuilt even if the code did not change).population_by_specsList of character vectors. Each element declares one
byaggregation that will be pre-computed during$process_skeletons()and stored in each batch's meta sidecar, so that$population(by = ...)is a fast meta-only walk. Read back with$population(by = <one of the declared specs>).
Active bindings
data_rawbatch_dirCharacter (read-only). Resolved rawbatch directory for the current host. Lazily resolved from
self$data_rawbatch_cp.data_skeleton_dirCharacter (read-only). Resolved skeleton directory for the current host.
data_meta_dirCharacter (read-only). Resolved metadata directory for the current host (where
registrystudy.qs2lives).data_raw_dirCharacter or NULL (read-only). Resolved raw-registry directory, or NULL if not configured.
data_summaries_dirCharacter or NULL (read-only). Resolved audit-track summaries directory for the current host, or NULL if not configured.
skeleton_filesCharacter vector (read-only). Skeleton output file paths detected on disk. Scans
skeleton_diron each access.expected_skeleton_file_countInteger (read-only). Expected number of skeleton files (one per batch).
meta_fileCharacter. Path to the on-disk metadata file (
registrystudy.qs2) insidedata_meta_dir.summaryList or NULL (read-only). The
summary.qs2payload written by$process_skeletons()(per-column counts, registry-wide totals, build metadata). NULL with a one-line message if the file is missing.
Methods
RegistryStudy$save_rawbatch()
Save rawbatch files for one group.
Usage
RegistryStudy$save_rawbatch(
group,
data,
n_workers = default_n_workers("rawbatch")
)Arguments
groupCharacter. Group name (must be in group_names).
datadata.table or named list of data.tables.
n_workersInteger. Number of parallel writers (default 1L, serial). When > 1L, slices are written concurrently via mirai daemons; the 'mirai' package must be installed. Each splittable data.table gets
BIDadded in-place and is keyed on it (setkey(dt, BID)), so per-batch slices are O(log n) keyed lookups instead of O(n)%in%scans. RAM stays ~1x the input (no split materialisation).An ID MAY be in more than one batch (see
$set_batches()). Then each batch gets the rows of the IDs it holds, in input order, by one%in%scan per batch. That path adds noBIDcolumn and does not change the caller's tables.After
$set_matched_sets(), each batch gets every row of a person once for eachskeleton_idof that person in the batch. The ID column of each copy holds thatskeleton_id. The copies of one row are adjacent, and the rows keep their input order. The ID column takes the type ofskeleton_id. The other columns do not change. Before it writes the group, the method writes each missing"roster"file. A roster where everyskeleton_idequals the person ID takes the paths above.
RegistryStudy$load_skeleton()
Load a skeleton file for batch_number as a
Skeleton R6 object. Returns NULL if the file is missing
(caller rebuilds from scratch). Errors if the file on disk is
not a Skeleton R6 object (e.g. corrupted or from an
incompatible version of swereg).
Returns
A Skeleton, or NULL if the file is missing.
RegistryStudy$save_skeleton()
Save a Skeleton to this study's skeleton directory,
plus a small meta_%05d.qs2 sidecar capturing provenance hashes
and the per-batch code-check accumulator snapshot. Subsequent
$process_skeletons() runs read the meta first and skip loading
the heavy skeleton entirely when every hash still matches.
Skeleton is written first, then meta. A crash between the two leaves a stale meta on disk; the next run reads it, finds the hashes don't match the current pipeline, falls through to the slow path, and rewrites both.
This is the one site that computes the per-column code-entry
counts. sk$refresh_code_entry_counts() runs before either
file is written, so the skeleton file and the meta both carry
counts that describe the data being written.
Arguments
skA Skeleton to persist.
framework_removalsOptional
data.tablereporting what each framework step removed..process_one_batch()passes theframework_removalsattribute of the framework function's return on a rebuild. On a slow path that runs no framework, it passes the report the previous meta held.NULLwhen the caller holds no report. The table goes into the meta sidecar, which is where$compute_summary()reads it.
RegistryStudy$delete_skeletons()
Delete all skeleton output files (and their meta
sidecars, plus any cached population_*.qs2 and
summary.qs2 artefacts) from disk.
RegistryStudy$save_meta()
Save this study object as metadata. Captures the destination path first, then clears host-specific CandidatePath caches before writing, so the on-disk file never carries a resolved path from the saving host.
RegistryStudy$population()
Read a pre-computed population table for one of the
by specs declared at construction time via
population_by_specs.
Population tables are computed automatically at the end of
$process_skeletons() from the per-batch aggregations stored
in each meta sidecar, then written as
population_<spec>.qs2 in the skeleton directory. This
getter just reads that file.
RegistryStudy$new()
Create a new RegistryStudy object.
Usage
RegistryStudy$new(
data_rawbatch_dir,
group_names = c("lmed", "inpatient", "outpatient", "cancer", "dors", "other"),
data_skeleton_dir = data_rawbatch_dir,
data_meta_dir = data_rawbatch_dir,
data_raw_dir = NULL,
data_summaries_dir = NULL,
batch_size = 1000L,
seed = 4L,
id_col = "lopnr",
population_by_specs = list()
)Arguments
data_rawbatch_dirCharacter vector of candidate paths for rawbatch files. The first existing path is used; a single non-existing path is created automatically.
group_namesCharacter vector of rawbatch group names.
data_skeleton_dirCharacter vector of candidate paths for skeleton output. Defaults to same candidates as
data_rawbatch_dir.data_meta_dirCharacter vector of candidate paths for the metadata directory holding
registrystudy.qs2. Defaults to same candidates asdata_rawbatch_dir(backward compatible). Pass an explicit value – e.g. the parent of rawbatch – to keep the singleton control file out of the per-batch data directory.data_raw_dirCharacter vector of candidate paths for raw registry files (optional). NULL if raw data paths are managed externally.
data_summaries_dirOptional character vector of candidate paths for the audit-track summaries directory (typically inside the project git repo, e.g.
dev/summaries/). When NULL (default),$compute_summary()still writessummary.qs2andstatus.txtto the skeleton directory but skips the git-tracked TSV.batch_sizeInteger. Number of IDs per batch. Default: 1000L.
seedInteger. Shuffle seed.
id_colCharacter. Person ID column name.
population_by_specsOptional list of character vectors. Each element declares one
byaggregation pre-computed during$process_skeletons()and stored in each batch's meta sidecar for fast$population(by)access. Example:list(c("rd_age_continuous"), c("rd_age_continuous", "ri_is_amab")). Default: empty list.
RegistryStudy$check_version()
Check if this object's schema version matches the current class version. Errors if the object was saved with an older schema.
RegistryStudy$register_framework()
Register the framework function (phase 1). Called once
per batch at the start of $process_skeletons(), with signature
function(batch_data, config), returns a fresh data.table
containing the base time grid + censoring. Everything downstream
builds on this output. A change to the function body or formals
triggers a full rebuild of every batch on the next
$process_skeletons() run.
RegistryStudy$register_trim()
Register the trim function (phase 1b). A study may register at most one. It runs immediately after the framework and before the code registry, so every code entry sees the row set the trim leaves behind.
$register_trim() stops on a second call, naming the trim
already registered. $register_framework() overwrites in the
same situation. The two differ deliberately. A study has one
framework by construction. A second $register_trim() call is a
script that means to delete rows in two places.
This is the one declared place in the pipeline that MAY delete skeleton rows. Phase 3 MUST NOT: its rewind drops columns and cannot restore rows.
fn runs exactly once per rebuild, on a fresh base. It never
sees data it already trimmed, so it MAY delete a fixed count or
a fraction. An edit to a randvars step or to a code entry
rebuilds nothing and therefore re-runs no trim.
A change to body(fn) or formals(fn) rebuilds the base
skeleton of every batch. So does adding a trim to a study that
had none, and so does removing one.
RegistryStudy$register_randvars()
Register one phase-3 "random variables" step. Phase 3
is an ordered sequence of user-supplied functions; each call to
$register_randvars() appends one step to the end of the
sequence. Registration order is execution order at
$process_skeletons() time.
Signature of fn: function(skeleton, batch_data, config). It
mutates skeleton in place and must ONLY ADD columns (never
modifying or deleting existing ones – the drop-and-replay
tracking depends on this invariant).
Phase 3 runs after phase 2, so fn MAY read a code registry
column.
An edit to fn's body, under the same name, changes the step's
hash. It replays this step and every step after it. A change to
the framework function, or to any code registry entry, replays
the whole sequence: $randvars_hashes() folds both into every
step's hash.
fn MUST NOT change the row count.
Skeleton$sync_randvars() compares the row count before and
after every replayed step. It stops the run when the count
moves, and it names the step. Register a row filter with
$register_trim() instead. The trim runs on a fresh base,
before the code registry, so every later phase sees the rows it
leaves.
Nothing checks the add-only contract. Rewind drops the columns a step recorded. It cannot restore a column the step overwrote.
RegistryStudy$code_registry_fingerprints()
Return the xxhash64 fingerprint of every entry in
self$code_registry, in registry order.
Primary entries: fingerprint depends on
(codes, groups, fn_args, combine_as) and on the hash of the
entry's fn. Two primary entries produce the same fingerprint,
and are therefore treated as "the same entry" across runs, only
when all five agree. An edit to a registered code function's body
re-applies that entry. label is NOT included: it is
presentation, and editing it replays nothing.
Derived entries: fingerprint depends on (codes, from, as) PLUS
the fingerprints of every upstream primary entry whose output
prefix is referenced in from. This cascades invalidation when
an upstream primary's fn_args / groups / codes change,
without requiring the user to touch the derived entry. Computed
in a two-pass walk: primary fingerprints first, then derived
fingerprints using the already-computed upstream fingerprints.
Used by Skeleton$sync_with_registry() for incremental
per-entry add/drop.
RegistryStudy$randvars_hashes()
Return one hash per registered phase-3 step, named by step name and in registration order.
Each step's hash folds in four inputs. They are the step
function's own body and formals, the framework function's hash,
the trim function's identity, and the code registry fingerprint
set. Every step takes the same framework, trim and code registry
components. So a change to any of the three diverges at step 1,
and Skeleton$sync_randvars() replays the whole sequence.
The framework component is NA_character_ when no framework
function is registered. The trim component is
"__swereg_no_trim__" when no trim function is registered.
Two inputs are NOT covered. A change to either one replays nothing:
Whatever a registered function calls or reads. Each hash covers that function's own body and formals. It does not follow a call into a helper, and it does not read a variable the function captured from its environment.
The rawbatch data. Nothing hashes raw content, so new raw data alone replays nothing.
$pipeline_identity() and $process_skeletons() both call this.
$process_skeletons() passes the result to
Skeleton$sync_randvars(), which stores each step's hash in
Skeleton$randvars_state and compares it on the next run.
RegistryStudy$pipeline_identity()
This study's current pipeline identity: the five components that decide what a freshly-built skeleton looks like, each normalized so that two representations of the same pipeline compare equal.
The components are framework, trim, phase_order, randvars
and codes. codes is a SORTED SET of code-entry fingerprints,
because Skeleton$sync_with_registry() applies entries by set
difference: a re-applied entry moves to the end of the stored
registry while naming an unchanged pipeline. The other four stay
ordered, because for those the order is semantic.
identical(sk$pipeline_identity(), study$pipeline_identity()) is
necessary for a synced skeleton. It is not sufficient. Unequal
components mean the skeleton is definitely stale. Equal ones mean
only that nothing changed among those five inputs.
Two inputs sit outside the identity: the rawbatch data, and
whatever a registered function calls or reads from its
environment. A change to either one leaves the identity equal
over a stale skeleton. $randvars_hashes() says why.
.PHASE_ORDER is a package constant, so it never discriminates
between two studies. It discriminates on the SKELETON side, where
an old skeleton reads NULL.
RegistryStudy$adopt_runtime_state_from()
Copy runtime state (IDs, batch list, saved groups)
from another RegistryStudy into this one, WITHOUT touching
config fields (group_names, code_registry, directory candidates,
framework/randvars registration, schema version, etc.).
Use case: in run_generic_create_datasets_v2.R, the generator
script constructs a fresh study every run with the current
in-memory config, then on re-runs calls
$adopt_runtime_state_from(qs2_read(self$meta_file)) to pick up
batch ids and saved-group state without silently adopting a
stale code registry or group name list.
RegistryStudy$register_codes()
Register code definitions for the code registry.
Each call declares codes, the function to apply them, which batch data
groups to use, and optional prefixing/combining. Appends to
self$code_registry.
Usage
RegistryStudy$register_codes(
codes,
fn,
groups,
fn_args = list(),
combine_as = NULL,
label = NULL
)Arguments
codesNamed list of code vectors (e.g. ICD-10, ATC, operation codes).
fnFunction to call (e.g.
add_diagnoses,add_rx).groupsNamed list mapping prefixes to group names. Unnamed elements get no prefix. Each element is a character vector of group names to rbindlist before calling
fn.fn_argsNamed list of extra arguments to pass to
fn(e.g.list(source = "atc")).combine_asCharacter or NULL. If non-NULL, also run
fnon all groups combined, using this as the prefix.labelCharacter. Human-readable label for describe_codes() output. Defaults to deparse(substitute(fn)). Presentation only: it is NOT part of the entry's fingerprint, so editing it replays nothing.
The call STOPS on either of two conflicts. It stops when
groupsnames a group that is not in the study'sgroup_names, because$load_rawbatch()reads one file per name there. It stops when the entry would generate a column an already-registered entry generates, because$drop_code_entry()removes exactly the columns an entry declares.
RegistryStudy$register_derived_codes()
Register a derived code entry: one that doesn't read rawbatch data, but instead ORs together already-existing skeleton columns from earlier primary entries.
For each name <nm> in codes, a new column <as>_<nm> is
written as Reduce("|", list(get("<from[1]>_<nm>"), ...)). The
codes list pattern values are ignored at apply time but DO
participate in the fingerprint, so editing the code list
triggers replay. The fingerprint also folds in the fingerprints
of every upstream primary entry whose output prefix appears in
from, so upstream behavior edits (e.g. cod_type on an
add_cods primary) cascade into derived replay automatically.
The derived entry runs in registration order during phase-2 sync, so any primary registrations whose output columns it references MUST be registered BEFORE this call.
Arguments
codesNamed list. Keys name the output columns' suffixes; the pattern values are ignored at apply time.
fromCharacter vector of source prefixes (e.g.
c("os", "dorsu", "dorsm")).asCharacter scalar: the output column prefix.
The call STOPS when the entry would generate a column an already-registered entry generates. Every entry owns its output columns alone.
RegistryStudy$apply_codes_to_skeleton()
Apply all registered codes to a skeleton data.table.
Thin loop over self$code_registry that delegates per-entry work
to the file-level .apply_code_entry_impl() helper. Kept for
backwards-compatible "apply everything at once" callers; the
incremental code-registry sync inside the Skeleton R6 class
calls .apply_code_entry_impl() directly on one entry at a time.
The method reserves the column slots the whole registry needs, then
applies each entry. It writes the result back to skeleton where
the caller passed a variable. Where the caller passed an expression
there is no binding to write to, so use the return value.
RegistryStudy$set_batches()
Set the batch composition directly, one ID vector per
batch, in the given order. Use it to keep a matched set in one
batch, which $set_ids() cannot do because it shuffles the IDs.
An ID MAY be in more than one batch. $save_rawbatch() then
writes every row of that ID into each batch that holds it. The
population and summary counts sum over the batches, so they count
a person in two batches twice.
The method stops when rawbatch or skeleton files are on disk and the new composition differs from the recorded one. A study with no recorded composition, for example one saved by swereg 27.2.6, records the new one and continues.
RegistryStudy$set_matched_sets()
Set the batch composition from a roster of matched sets. Each set stays whole in one batch. Each membership gets its own skeleton ID and its own copy of the person's rawbatch rows.
The method packs the sets into batches in order of first
appearance of set_id, sets_per_batch sets per batch. The
batches hold the skeleton_id values of their roster rows, as
$set_batches() would set them. The guard of $set_batches()
applies, and the roster is part of the recorded composition. So a
changed roster stops while rawbatch or skeleton files are on disk.
$save_rawbatch() then writes each rawbatch row of a person once
for each skeleton_id that person has in the batch. The copy
carries that skeleton_id in the ID column, and its other columns
do not change. $save_rawbatch() also writes the roster rows of
each batch as the rawbatch group "roster", with the person ID and
the skeleton_id unchanged. Every phase function reads them as
batch_data[["roster"]].
A roster where every skeleton_id equals the person ID gives the
same rawbatch files as $set_batches() on the same IDs, plus the
"roster" files.
Arguments
rosterA data.table with one row per membership and the columns
id_col(the person ID in the rawbatch data),set_id,role(character) andskeleton_id. Oneskeleton_idMUST map to one person ID. Askeleton_idMUST NOT repeat within one batch, and MAY repeat across batches. Other columns go into the"roster"files unchanged.sets_per_batchA positive whole number. The number of sets in each batch. The last batch MAY hold fewer.
RegistryStudy$load_skeleton_meta()
Read the meta_%05d.qs2 sidecar for one batch.
Returns NULL if missing or unreadable (treated as cache miss
by the fast path in .process_one_batch()).
RegistryStudy$skeleton_pipeline_hashes()
Summary of per-batch pipeline hashes across all
currently-persisted skeleton files in self$data_skeleton_dir.
Use this to spot batches out of sync with each other or with
self$pipeline_identity().
Files that are not valid Skeleton R6 objects (e.g. unreadable
or corrupted) surface as rows with NA identity_hash,
NA framework_fn_hash, NA trim_fn_hash and NA
phase_order.
Arguments
filesOptional character vector of skeleton file paths. When
NULL(default) everyskeleton_NNNNN.qs2inself$data_skeleton_diris read. Pass the subset a consumer is about to read to scope the answer to those files.
Returns
A data.table with columns: batch, identity_hash,
identity, framework_fn_hash, trim_fn_hash, phase_order,
n_randvars, n_code_entries, saved_at. identity is a list column
holding each batch's normalized pipeline identity, so a caller can
name the component that differs rather than quote a digest.
phase_order is the stored character vector collapsed with
" -> ", so one batch is one row.
RegistryStudy$assert_skeletons_consistent()
Assert that every persisted skeleton file has the same pipeline identity AND that it matches this study's current pipeline identity. Errors loudly with an actionable message if not, naming the component that differs.
Runs as the pre-flight check inside
tteplan_from_spec_and_registrystudy(), so partial-rebuild
stragglers or config drift never silently flow into a TTE plan.
RegistryStudy$process_skeletons()
Orchestrate the skeleton pipeline per batch.
Reads self$framework_fn (phase 1), self$trim_fn (phase 1b),
self$code_registry (phase 2), and self$randvars_fns
(phase 3) from the study. Applies them via the incremental logic
on Skeleton. Exact per-batch work:
Load existing skeleton via
self$load_skeleton(i). Rebuild the base from scratch when the file is missing. Rebuild it too whenframework_fn_hash,trim_fn_hashorphase_orderdoes not match the study's current identity. A rebuild callsself$framework_fn(batch_data, self), wraps the result in a fresh Skeleton, and resets phases 2 and 3. (Phase 1.) 1b. On a rebuild only, callself$trim_fn(sk$data, batch_data, self)and rebindsk$datato what it returns. Skipped when no trim is registered. (Phase 1b.)Call
sk$sync_with_registry()withself$code_registry_fingerprints(). Entries present on disk but not in the current registry are dropped (via.entry_columns()on the stored descriptor); entries present in the current registry but not on disk are applied fresh. (Phase 2.)Call
sk$sync_randvars()with the current orderedself$randvars_fnsand their body/formals hashes. Divergence- point rewind-and-replay semantics drop and re-run the affected phase-3 steps only. A step MAY read a phase-2 column, because phase 2 already ran. (Phase 3.)Save via
self$save_skeleton(sk).
batch_data is loaded lazily – exactly once per batch, by
whichever phase needs it first. If no phase needs it (everything
already in sync), the rawbatch read is skipped entirely and the
per-batch work is just load then save.
Usage
RegistryStudy$process_skeletons(
batches = NULL,
n_workers = default_n_workers("skeleton"),
...
)Arguments
batchesInteger vector of batch indices to process, or
NULL(default) for all batches inself$batch_id_list.n_workersInteger. Number of parallel workers (1 = sequential). When
> 1, each batch runs in a fresh worker subprocess via the generic batch runner....Additional arguments (unused; reserved for future use).
Examples
if (FALSE) { # \dontrun{
study <- RegistryStudy$new(
data_rawbatch_dir = c("/linux/.../rawbatch/", "C:/win/.../rawbatch/"),
data_skeleton_dir = c("/linux/.../skeleton/", "C:/win/.../skeleton/"),
data_raw_dir = c("/linux/.../raw/", "C:/win/.../raw/"),
group_names = c("lmed", "inpatient", "outpatient", "cancer", "dors")
)
# Phase 1: framework (structural time grid + censoring)
study$register_framework(my_framework_fn)
# Phase 1b: trim (the only place rows may be deleted)
study$register_trim(my_trim_fn)
# Phase 2: codes. Primary entries first, derived entries after.
study$register_codes(
codes = list(e11 = c("E11"), vte = c("I26", "I80")),
fn = swereg::add_diagnoses,
groups = list(ov = "outpatient", sv = "inpatient"),
combine_as = "os"
)
study$register_codes(
codes = list(e11 = c("E11"), vte = c("I26", "I80")),
fn = swereg::add_cods,
fn_args = list(cod_type = "underlying"),
groups = list(dorsu = "dors")
)
study$register_codes(
codes = list(e11 = c("E11"), vte = c("I26", "I80")),
fn = swereg::add_cods,
fn_args = list(cod_type = "multiple"),
groups = list(dorsm = "dors")
)
# Build osd_e11 = os_e11 | dorsu_e11 | dorsm_e11 (same codes list
# shared by reference so an edit in one place cascades to all four)
study$register_derived_codes(
codes = list(e11 = c("E11"), vte = c("I26", "I80")),
from = c("os", "dorsu", "dorsm"),
as = "osd"
)
# Phase 3: randvars (ordered user steps; order = execution order).
# They run after the code registry, so they may read its columns.
study$register_randvars("demographics", my_demographics_fn)
study$register_randvars("exposure", my_exposure_fn)
study$set_ids(ids)
study$save_rawbatch("lmed", lmed_data)
study$describe_codes()
study$process_skeletons(n_workers = 4L)
# Per-batch provenance and cross-batch consistency check
sk <- study$load_skeleton(1L)
identical(sk$pipeline_identity(), study$pipeline_identity()) # FALSE => stale
study$assert_skeletons_consistent() # errors on mixed state
} # }