Preventing Temporal Leakage in EHR Prediction Studies

A laboratory result can carry a specimen time of 09:20 and still be invisible to a clinician at noon. If a retrospective model receives the result at 09:20, its apparent ability to predict an afternoon event may come from information that did not yet exist at the decision point. A causal attention mask cannot repair that error: the future value has already been placed in the model’s past.

The remedy is to construct each input as an as-of snapshot: the information the intended user could have accessed at a specified prediction time. This article develops that contract for a hypothetical ward-to-ICU transfer experiment, from source timestamps and corrections through model comparison and audit tests. The example is synthetic; it makes no claim about clinical performance. The same method applies whether the final model is a regression, a tree, or a sequence transformer.

Specify the decision before extracting features

Suppose a research team wants to study whether an inpatient ward model could flag an unplanned ICU transfer in the next six hours. Define an issue time, t, at noon each day while the patient is on a ward and eligible for a decision. The inputs are records accessible at t. The outcome is an unplanned transfer with an effective transfer time in (t, t + 6 hours]. Whether that endpoint is clinically useful is a separate question for clinicians; it is only a concrete experiment here.

This formulation forces several decisions that a generic “predict deterioration” label hides. Is the patient still on the ward at noon? Are planned transfers excluded by a rule fixed before looking at model predictions? What happens if discharge or death occurs before six hours elapse? Is noon a real intervention opportunity, or merely a convenient sampling point? If a team instead samples only moments immediately before observed transfers, cases and controls have different origins, and the evaluation no longer represents the proposed noon workflow. A systematic review of temporal EHR prediction studies found substantial variation in how observation and prediction windows were defined and reported; those definitions are part of the scientific method, not a formatting detail.

Write a prediction contract before building a table:

Contract element	Choice in this illustrative study	What must be checked
Eligible population	Current ward admissions at the noon trigger	Eligibility can be established using information known at noon.
Input boundary	Values accessible no later than noon	Source-specific availability timestamps, not just clinical event times.
Outcome window	Unplanned ICU transfer after noon through 18:00	Transfer definition, ascertainment, and competing events.
Prediction cadence	At most one noon prediction per eligible admission-day	Repeated observations from one patient must remain linked in analysis.
Intended use	Research evaluation of a possible alert	Clinical action, threshold, and prospective validation remain unspecified.

The outcome may be determined from a later, complete record. That is legitimate for labelling the noon example. It does not make the later information legitimate as an input. Keep these two data products separate. In particular, do not use the eventual diagnosis, discharge disposition, or transfer order status as a noon predictor merely because it is attached to the same admission.

Four clocks, one patient

An EHR record can have at least four relevant times: when an event happened, when someone entered or validated it, when a result became accessible in the system used for prediction, and when the extract was created or revised. These are different questions. A clinically meaningful event time can be valuable for modelling elapsed time, but the value must first pass an availability test.

The distinction is explicit in MIMIC-IV’s laboratory table documentation: charttime for a lab usually reflects specimen acquisition, while storetime is when the measurement became available in the laboratory system. The ICU chart-events documentation similarly distinguishes observation time from manual input or validation time. Neither field should be treated as a universal measure of what a particular clinician saw on a particular screen; the local interface, feed, and workflow still need checking. A recent EHR extraction and preparation guide identifies time-sensitive feature engineering and timestamp interpretation as central to avoiding retrospective temporal leaks.

Consider this synthetic admission on 8 January. The numbers and times are invented solely to expose the boundary.

Source record	Clinical event time	Available to the prediction system	Belongs in the noon input?
Vital observation: stable	08:00	08:04	Yes.
Lactate order	09:10	09:10	Yes, as an order, not as a result.
Specimen collected	09:20	09:25	Yes, if the collection notice is visible.
Vital observation: low pressure	11:45	11:47	Yes.
Vital observation: lower pressure	11:50	12:07	No; it was entered after noon.
Lactate result: 4.1	09:20 specimen time	12:18 result time	No; backdating it to the specimen time would leak the value.
ICU transfer	13:05	13:06	No; it is the later outcome in this example.

The two timelines diverge at the prediction boundary:

Clinical event time:  09:20 specimen → 11:50 vital → 12:00 prediction → 13:05 transfer
Information arrival:  09:25 collection notice → 12:00 prediction → 12:07 vital entry → 12:18 lab value
                                                     ↑ unavailable at prediction ↑

At noon, the legitimate sequence contains the order and collection, but not the numeric result. Keeping the order can be appropriate because ordering a test is itself an observed care action. Its interpretation depends on the task: an intervention or order may also be a close proxy for the intended outcome, even when correctly timed. A framework for label leakage in healthcare ML argues that prediction cadence, user perspective, and intended application affect whether such a feature is appropriate. Clinical reviewers should decide whether the proposed alert would still be useful after that action has occurred.

The 11:50 vital sign shows a second failure mode. Filtering only on measurement time would admit it, although the ward system did not have it at noon. Conversely, replacing all event times with entry times throws away useful information about when the physiology was observed. Preserve both: filter by availability, then represent the accepted event’s clinical time and its delay if those variables are relevant. A missing availability timestamp is an uncertainty to investigate, not proof that the value was available immediately.

Build an as-of ledger, not a final-state admission row

For each source type, record a mapping from source fields to the concepts below. The mapping should be signed off by someone who understands that system’s workflow; the same name can mean different things across lab, medication, charting, and billing feeds.

Field in the research ledger	Purpose
patient_id, encounter_id, source_event_id	Identify a person, admission, and source record without using them as predictive features by default.
event_type, event_time	Describe what happened and when it is said to have happened.
available_at	Earliest defensible time this version and value could have been used by the intended prediction system.
recorded_at, revision_id	Reconstruct corrections and late entries rather than silently using the final version.
value, unit, status	Preserve the content and state of the record, including preliminary or cancelled results.
source_table, source_row_id	Trace every derived feature back to the extract for manual review.

The core operation is simple: for a prediction at t, select versions with available_at <= t, keep the latest visible version of each source event, then reject any whose event time is after t where the task requires past events. This order matters if a correction changes the event time. Only then calculate features or serialize an event sequence. A date-only value needs an explicit policy: treating an unspecified calendar date as midnight can make it appear hours too early. Times must be normalized consistently, including daylight-saving transitions and source time zones. Do not impute an unknown availability time with the clinical event time simply because it makes the join easy.

The annex contains an illustrative SQL-style pattern for this version-aware selection.

If a source exposes only the current value and no history of revisions, the past version cannot be recovered by sorting the final extract. A query executed today may show a corrected value that was absent at noon months ago. Use an audit log, a historical snapshot, or a justified conservative exclusion for those fields. Explicitly report which feature families cannot be reconstructed and how many candidate predictions they affect. If the missing history is material to the experiment, the retrospective study cannot answer the proposed as-of question as stated.

The ledger should also separate different information states. For a lab, an order at 09:10, a specimen collection at 09:20, a preliminary result, and a final result are distinct signals with distinct availability times. Collapsing them into one row can attach the final numeric value to the earliest state. For a note, the note date, authored time, signed time, and later addendum can likewise differ. For diagnosis codes, a code associated with an encounter may have been assigned only at discharge or during billing. Source-specific evidence, not a blanket column-name rule, determines whether any one of these is usable at noon.

Test the time boundary as data, not as an attention setting

Before training, produce one small, hand-inspectable snapshot like the table above for each major source type. Then make the construction code prove several invariants on a synthetic fixture. The annex contains the complete Python 3.10+ audit and its observed output: it selects the four noon-visible events, checks that a later correction leaves the noon snapshot unchanged, and checks a visible correction that moves an event beyond noon. Those tests establish properties of the fixture and code path, not the correctness of a hospital extract.

1. Cutoff test. For every included predictor version, assert available_at <= issue_time. Also assert that no feature draws from an event or interval that starts after the issue time.
2. Backdated-result test. Give a lab result a pre-noon collection time and a post-noon availability time. The noon feature table must exclude its value while retaining a separately available order or collection event.
3. Future-perturbation test. Add or change records and corrections that become available after noon. Rebuilding the noon input must return byte-identical features or an intentionally equivalent representation. This is stronger than checking a few timestamps after feature engineering.
4. Replay test. Where source snapshots permit it, compare a feature vector built from the entire admission with an as-of cutoff to one built by replaying only updates available by that cutoff. Differences reveal retrospective transformations, latest-state joins, or undocumented corrections.
5. Lineage test. For every derived feature, retain the contributing source row IDs and effective times. A rolling minimum or “latest lab” feature is only as safe as its least well-timed contributor.

Run these tests after joins and feature engineering as well as on the raw ledger. A filter on the event table can be undone by a later join to a final admission summary. Look especially at “ever received treatment,” length of stay, discharge diagnosis, whole-encounter counts, note text updated after signing, and outcome-related status flags. An impressive model score is a reason to inspect these fields, not evidence that they are safe.

There is also a subtler choice at the exact boundary. Decide whether records stamped exactly at t were available before the prediction job executed. If that cannot be established, use a strict earlier-than rule or a fixed operational delay, and record it. An overly optimistic equality rule can create tiny but systematic leaks when laboratory and batch systems round timestamps to the same minute.

Keep the label and evaluation aligned with noon

Once the as-of features are fixed, construct the six-hour label from transfer records using a separate, later extraction path. Define the transfer event and its effective time independently of predictor construction. Admissions already in ICU at noon are not negative examples. A planned transfer may require a prespecified exclusion or a separate outcome. Patients discharged before 18:00 may have incomplete time at risk; whether to censor, exclude, or use a competing-event analysis depends on the clinical question. Record the counts for each decision. An empty “no transfer” label is not automatically a valid control if follow-up is missing.

The sampling scheme matters as much as the label definition. If a patient contributes five noon predictions, those five rows are correlated. If a model will be run daily, evaluation should preserve that cadence and report both patient or admission counts and prediction counts. Do not choose the best-looking prediction time retrospectively for each positive patient. A positive patient’s noon before transfer and a negative patient’s admission noon should be generated by the same eligibility rule. Otherwise the model may distinguish sampling mechanisms rather than future transfer risk.

Split the study according to its intended generalization claim. For a model intended for later deployment at the same hospital, reserve a later calendar period untouched while developing the model on earlier patients. If the study claims performance on unseen people, prevent one patient’s admissions from crossing development and test partitions. Both constraints may need to hold, and a temporal boundary can make the sample smaller. Fit vocabularies, normalizers, imputation values, feature selection, and thresholds on the development period only. A valid as-of snapshot can still be contaminated by a test-set-fitted transformation. The scikit-learn guidance on data leakage gives the general train-only fitting rule.

Choose a reference model that gets the same noon information. A logistic model using recent vital summaries, visible lab results, order indicators, missingness, and elapsed time is a credible first comparator. A tree model can test nonlinear interactions without a sequence encoder. Include a simple clinical score only if its inputs and intended population actually match the task. Compare models on the same prediction points and frozen outcome definition. If an event-order model improves discrimination, ask whether the gain comes from order, richer feature access, greater capacity, or subtle leakage. An aggregate “all labs in the previous six hours” baseline must use result availability times too.

Discrimination is only part of the result. Report calibration, event prevalence, the number of alerts at proposed thresholds, sensitivity at a clinically relevant alert burden, and lead time from prediction to transfer. Set thresholds on validation data before touching the held-out period. Estimate uncertainty at the patient or admission level rather than pretending overlapping daily predictions are independent. Inspect missingness, delayed-entry rates, source-system changes, and performance by ward and time period. A later-period drop can be a data-flow change as well as a modelling failure. None of these retrospective checks establishes that an alert improves care; that requires its own clinical and prospective evaluation.

Where causal sequence modelling fits

An ordered record can capture a pattern an aggregate discards: a lab order, followed by a newly visible vital sign, followed later by a result is different from the same three items in another order. A causal model can condition a next-event prediction only on preceding positions. That is useful for testing whether event order adds information, provided the positions themselves were created from as-of-available records. Causal masking constrains attention inside a model window; it does not determine whether the underlying values were available to the user at the issue time.

Sequifier’s README describes a tabular sequence format with sequenceId, itemPosition, and one or more feature columns. Its preprocessing guide documents windowing and splitting, including max_target_offset: 1 for causal next-item training; its training guide documents training_objective: causal, context_length, input columns, and target columns. These are useful implementation pieces after the clinical source ledger and noon eligibility rules have been built.

For a next-event experiment, assign a sequence to each admission or to a patient history according to the research question. Order sequence positions by information arrival, normally available_at, rather than by clinical event time. In the example, the 12:18 lactate result comes after the noon prefix even though its specimen was collected at 09:20. Keep that 09:20 time as an attribute of the result, and derive any elapsed-time feature only when the result becomes visible. A correction is a new arrival or a new as-of snapshot version; never silently replace an earlier training prefix with the final corrected value. If several values share one availability timestamp, group them into one step or define a defensible simultaneous-event representation. An arbitrary tie-breaker should not let one same-time value predict another as if it arrived earlier.

After building that arrival-ordered stream, assign itemPosition. A causal next-event setup can use a stored width of context_length + 1, max_target_offset: 1, and matching one-step target settings. Audit decoded windows to confirm that the target event is to the right of its allowed context. A separate as-of snapshot is still needed for each noon prediction: truncating a full arrival stream at noon must produce exactly the records available then. Sequifier’s inference guide describes aligned outputs and the target-position interpretation of causal embeddings; it does not establish the clinical availability rule for the source data.

For example, a preprocessor could give the synthetic admission the following illustrative Sequifier input order. The stream represents information arrivals, not a clinical-time sort:

sequenceId	itemPosition	event	Arrival
101	0	stable_vital	08:04
101	1	lactate_order	09:10
101	2	specimen_notice	09:25
101	3	low_pressure_vital	11:47
101	4	lower_pressure_vital	12:07
101	5	lactate_result	12:18
101	6	icu_transfer	13:06

The noon prefix ends at position 3. In a next-event exercise, position 4 may be a legitimate future target for the prefix, but it cannot be an input to the noon prediction. Nor does predicting position 4 answer the six-hour transfer question. Keep the arrival timestamp and original clinical timestamp in the research ledger even if only the categorical event column is passed to a first Sequifier run; they are needed to audit the prefix and the clinical endpoint.

The annex’s Sequifier preprocessing check runs this seven-row synthetic stream through sequifier preprocess. Its first stored window puts positions 0–3 before position 4, as the documented causal next-item layout requires. This verifies the handoff and window alignment only; no model was trained or evaluated.

Next recorded event and “unplanned ICU transfer within six hours” are different targets. Event streams are irregular: the next event may happen in a minute or a day, and a lab result may intervene before transfer. A successful next-event experiment is therefore not evidence of six-hour transfer prediction. A researcher could study whether causal history representations help a separately specified six-hour classifier, but that requires an explicit training and evaluation design. The cited Sequifier guides do not make the EHR extraction, availability audit, clinical endpoint, or prospective validation automatic. For an initial study, the simpler as-of baseline may be the better model; the sequence comparison is justified only if it adds useful evidence on the same valid snapshots.

A publishable methods record

A reader should be able to reproduce the information boundary even without the underlying patient data. Preserve the exact issue-time and eligibility rules; source-by-source timestamp semantics; handling of missing times, corrections, and date-only fields; label and censoring definitions; patient and calendar splits; transformation fit scope; snapshot audit cases; and the denominators behind every metric. Show a timeline for at least one case and one control, with the visible input separated from later label evidence. The EHR data-preparation guide recommends manual verification of sampled admissions alongside automated checks; this is particularly important where an extract’s timestamps do not fully describe the clinical workflow.

If the hospital cannot reconstruct when a material predictor became available, report that limitation and narrow the experiment. No architecture can infer a trustworthy historical snapshot from a final-state table alone. Conversely, once the snapshot contract is credible, causal sequence modelling becomes a meaningful comparison: it tests whether ordered, legitimately visible history helps with a precisely defined task. The first technical milestone is therefore not a model score. It is being able to answer, for every input value: what did the user know at noon, and how do we know?

Annex: As-of query and audit fixture

The examples below are implementation aids for the method described above. The query is illustrative pseudocode; the Python fixture uses only invented records and was executed.

Version-aware as-of query

Here is illustrative SQL-style pseudocode, with the source-to-ledger mapping and version semantics supplied by the research team:

WITH visible_versions AS (
  SELECT p.prediction_id, p.issue_time, e.*,
         ROW_NUMBER() OVER (
           PARTITION BY p.prediction_id, e.source_event_id
           ORDER BY e.available_at DESC, e.revision_id DESC
         ) AS version_rank
  FROM prediction_points AS p
  JOIN event_ledger AS e
    ON e.encounter_id = p.encounter_id
   AND e.available_at <= p.issue_time
)
SELECT *
FROM visible_versions
WHERE version_rank = 1
  AND event_time <= issue_time;

This pattern assumes that the research team has mapped source records to the ledger and established the meaning of each revision. It is not a query that can be run against an arbitrary hospital schema.

Executed synthetic audit

Save the following complete Python 3.10+ program as asof_demo.py and run python3 asof_demo.py. It uses only invented records and checks the noon cutoff, a later correction, and a visible correction that moves an event beyond noon.

"""Synthetic, standard-library fixture for an EHR as-of feature audit.

Run: python3 asof_demo.py
This checks data construction only; it is not a clinical experiment.
"""

from datetime import datetime


def at(clock: str) -> datetime:
    return datetime.fromisoformat(f"2026-01-08T{clock}:00")


def asof_events(rows: list[dict], cutoff: datetime) -> list[dict]:
    """Select the latest visible version per source event, then order it."""
    visible = [row for row in rows if row["available_at"] <= cutoff]
    latest = {}
    for row in visible:
        key = row["event_id"]
        if key not in latest or row["available_at"] > latest[key]["available_at"]:
            latest[key] = row
    return sorted(
        (row for row in latest.values() if row["event_time"] <= cutoff),
        key=lambda row: (row["available_at"], row["event_time"], row["event_id"]),
    )


def main() -> None:
    rows = [
        {"event_id": "v1", "event_time": at("08:00"), "available_at": at("08:04"), "kind": "vital", "value": "stable"},
        {"event_id": "o1", "event_time": at("09:10"), "available_at": at("09:10"), "kind": "lab_order", "value": "lactate"},
        {"event_id": "s1", "event_time": at("09:20"), "available_at": at("09:25"), "kind": "specimen", "value": "collected"},
        {"event_id": "v2", "event_time": at("11:45"), "available_at": at("11:47"), "kind": "vital", "value": "low_pressure"},
        {"event_id": "v3", "event_time": at("11:50"), "available_at": at("12:07"), "kind": "vital", "value": "lower_pressure"},
        {"event_id": "l1", "event_time": at("09:20"), "available_at": at("12:18"), "kind": "lab_result", "value": "4.1"},
        {"event_id": "t1", "event_time": at("13:05"), "available_at": at("13:06"), "kind": "icu_transfer", "value": "yes"},
    ]
    cutoff = at("12:00")
    expected = ["v1", "o1", "s1", "v2"]
    snapshot = asof_events(rows, cutoff)
    assert [row["event_id"] for row in snapshot] == expected

    # A later correction to a previously visible row must not rewrite noon.
    corrected = rows + [
        {"event_id": "v2", "event_time": at("11:45"), "available_at": at("14:00"), "kind": "vital", "value": "normal_pressure"}
    ]
    assert asof_events(corrected, cutoff) == snapshot

    # Select the latest visible version before checking its event time.
    moved = rows + [
        {"event_id": "m1", "event_time": at("11:00"), "available_at": at("11:01"), "kind": "procedure", "value": "scheduled"},
        {"event_id": "m1", "event_time": at("12:30"), "available_at": at("11:55"), "kind": "procedure", "value": "rescheduled"},
    ]
    assert asof_events(moved, cutoff) == snapshot

    # The result value, despite its 09:20 collection time, is absent at noon.
    assert "l1" not in [row["event_id"] for row in snapshot]
    print("As-of snapshot:", [(r["event_id"], r["kind"], r["value"]) for r in snapshot])
    print("PASS: cutoff, future-correction, and revised-time invariants")


if __name__ == "__main__":
    main()

The executed program printed:

As-of snapshot: [('v1', 'vital', 'stable'), ('o1', 'lab_order', 'lactate'), ('s1', 'specimen', 'collected'), ('v2', 'vital', 'low_pressure')]
PASS: cutoff, future-correction, and revised-time invariants

Sequifier preprocessing handoff

The as-of audit above establishes what may enter a noon input. Sequifier receives the arrival-ordered rows after that rule has been applied. For a small preprocessing smoke test, save this synthetic input as data/events.csv in a Sequifier project:

sequenceId,itemPosition,event
101,0,stable_vital
101,1,lactate_order
101,2,specimen_notice
101,3,low_pressure_vital
101,4,lower_pressure_vital
101,5,lactate_result
101,6,icu_transfer

Save this documented preprocessing configuration as configs/preprocess.yaml:

project_root: .
preprocessing_data_path: data/events.csv
read_format: csv
write_format: parquet
selected_columns: [event]
categorical_columns: [event]
window_length: 5
max_target_offset: 1
window_stride: 1
split_method: within_sequence
split_ratios: [0.8, 0.1, 0.1]
normalize_on_all_data: false
seed: 1010

Run sequifier preprocess --config-path configs/preprocess.yaml from the project root. In the executed smoke test, Sequifier completed preprocessing and the sole split-0 window decoded to stable_vital, lactate_order, specimen_notice, low_pressure_vital, lower_pressure_vital, with no left padding. Under a four-position causal context and one-position target, the first four events are the context available at noon and lower_pressure_vital is the next-event target. The separate verification script printed:

Window context: ['stable_vital', 'lactate_order', 'specimen_notice', 'low_pressure_vital']
Next-event target: lower_pressure_vital
PASS: Sequifier window places the noon prefix before the next event

This is a format and windowing check on one invented admission. Its tiny chronological split leaves later event categories unseen in training, so it cannot support a meaningful model comparison. It also does not train a Sequifier model or predict the six-hour ICU-transfer endpoint. For a real experiment, use audited as-of event streams from many admissions, split patients and calendar time according to the study design, and compare a configured causal model with baselines on the same noon predictions.