How to Choose Context Length for Time-Series Transformers

How much history should a time-series transformer see for a one-hour-ahead forecast? An hourly series can carry two kinds of evidence at once. Last week’s value at the same hour may explain a repeating load cycle. The last three observations may show that a machine has just changed state. A short history can see the change but miss the earlier cycle; a long history can contain both, yet may spend most of its capacity on irrelevant or obsolete observations. The useful context length is the shortest history that improves the decision you actually make, on forecast origins the competing lengths can all serve, at an acceptable cost. It is an empirical choice, not a property of the transformer alone.

This guide sets up that comparison. Its running example is a hypothetical hourly load series with daily and weekly patterns and occasional abrupt jumps. The dates and readings in that worked example are synthetic; a separate measured-data check appears below. The same logic applies to other multivariate sequences, provided the prediction time, target, and meaning of a step are made explicit.

Define what one prediction is allowed to know

Suppose a forecasting service runs after hour t has closed. It receives measured load y_t, temperature, operating mode, and other observations available by that time, then predicts load for hour t+1. A context of L rows contains positions t-L+1 through t; the target is y_{t+1}. One step means one hour only if the data have been put on a regular hourly grid. If sensors report irregularly, 168 events need not represent a week. Resample with an explicit missing-data policy, or make elapsed time and gaps part of the representation.

This contract prevents a common category error: forecasting an abrupt event before it happens differs from detecting one after its first measurement. A short context may be excellent for identifying a jump already in y_t while giving little warning of a genuinely unanticipated jump at t+1. Do not use a detection result as evidence of forecast skill. Likewise, a one-hour-ahead result says nothing by itself about a 24-hour horizon. Horizon, issue time, target, and permissible inputs are part of the task, and context length should be selected separately when they change.

For the example, construct an as-of table with one row per site and hour:

Field	Meaning at hour t	Availability rule
sequenceId	Site or uninterrupted operating history	Stable across the history used by the model.
itemPosition	Consecutive hourly index	Increases by one after a deliberate grid and gap policy.
load	Observed load	Use only values received by the prediction issue time.
ambient_temperature	Measured temperature	Do not replace it with a later corrected reading in a historical replay.
operating_mode	State logged at that hour	Check whether state changes are recorded before or after the load measurement.
hour_of_week	Derived calendar category	Known without looking at future load; make daylight-saving handling explicit.

sequenceId and itemPosition describe order; the other fields are candidate inputs. If a site is shut down or its meter is replaced, decide whether the later rows are one continuous history or a new sequence. A model may otherwise learn a false dependency across a reset. Keep timestamps in the raw audit data even if the training table uses positions. A fixed UTC grid plus local calendar features is one defensible way to avoid duplicated or missing local clock hours, though the choice must match the operation being forecast.

For one-hour-ahead prediction, a 24-row window reaches the corresponding hour yesterday: y_{t+1-24} is inside the window. A 168-row window reaches the corresponding hour last week: y_{t+1-168} is inside it. These are coverage thresholds, not reasons to assume either window wins. A calendar feature can encode the weekly phase without requiring an entire week of raw measurements, while a weekly naive forecast needs that observed historical value. Whether older load still predicts the future after recent readings, weather, and calendar information are available is the question to test.

Two forecast origins, the same targets

Consider one synthetic site with an uninterrupted hourly UTC record. At each origin below, the 24-hour and 168-hour candidates forecast the same next-hour value. The example readings make the information boundary visible; no model has been fitted to them.

Forecast origin and withheld target	What both contexts can see	What only the 168-hour context can see
Monday 17 August 2026, 09:00; forecast 10:00 load, later observed as 104	The 24-hour window spans Sunday 16 August 10:00 through Monday 17 August 09:00. It includes Sunday’s 10:00 load of 86 and the current load of 99.	The 168-hour window starts Monday 10 August 10:00. It includes the entire 24-hour window and the previous Monday 10:00 load of 102.
Tuesday 18 August 2026, 09:00; forecast 10:00 load, later observed as 137	The 24-hour window spans Monday 17 August 10:00 through Tuesday 18 August 09:00. It includes Monday’s 10:00 load of 104, an operating-mode change logged Tuesday at 08:00, and the current load of 132.	The 168-hour window starts Tuesday 11 August 10:00. It includes the entire 24-hour window and the previous Tuesday 10:00 load of 96, from before the mode change.

The short window cannot inspect the previous week’s matching hour at either origin. Both windows can see the mode change at the second origin; neither can see the 137 target before making that forecast. This second row tests how a model responds after an observed change, not whether it could predict the change before it happened. Scoring both candidates on these same two targets would isolate the extra visible history, although two examples are far too few to select a model.

Why more history can help, and why it can hurt

Longer context expands the set of dependencies a model could use. It does not prove that the fitted model uses them. For a stable weekly cycle, the previous matching weekday may be informative. For an abrupt operating-mode change, a long average over the old regime can be misleading. Old readings may also become less comparable after calibration changes, outages, promotions, or shifts in the mix of sites. Attention can in principle downweight them, but it must learn to do so from finite training data.

The cost also depends on the representation and architecture. For ordinary full self-attention, the number of attention scores for one layer and head grows approximately as L²: moving from 168 to 672 positions gives 16 times as many position pairs, before considering other model operations. That is a scaling calculation, not a measured training-time ratio. Memory, kernels, batching, model width, padding, and hardware determine actual runtime. Temporal patching can reduce the number of tokens for a given raw history; the PatchTST paper studies that design. Changing from hourly tokens to patches changes more than context length, so it deserves a separate architecture comparison.

Evidence against a universal “longer is better” rule is not merely theoretical. The LTSF-Linear study reports that longer look-back windows did not consistently improve the transformer variants it evaluated and that simple linear baselines were strong on its benchmarks. PatchTST reports gains from a different representation that makes longer look-back practical. Adaptive look-back with wavelets is another proposed approach, but changing the window separately for each example changes the method being evaluated. Neither fixed nor adaptive results settle the choice for this site’s data or for an abrupt-event slice. The defensible conclusion is to test context length with credible non-transformer baselines and a fixed evaluation contract.

Build a comparison that isolates context length

Start with a small grid tied to identifiable timescales. For an hourly series, L = 24, 168, 336, 672 tests one day, one week, two weeks, and four weeks. If engineering constraints make 672 impossible, record the feasible maximum rather than silently replacing it. If the task involves minute-level spikes, add a shorter candidate such as 6 or 12 hours. If the cycle is monthly, four weeks of hourly data may still be too short to represent it. These values are hypotheses about the data, not recommended defaults.

The unit of comparison is a forecast origin: site, issue time t, and observed target at t+1. Build a manifest of origins before fitting the candidate models. For the primary matched comparison, include an origin only if it has the required 672 valid preceding hours and a target, or define one common padding policy in advance. Feed every candidate the same origin and target; truncate only the visible history. Otherwise the 24-hour model may be scored on many new, short-lived sites while the 672-hour model is scored mainly on established sites. A difference in average error could then reflect the evaluated population rather than history length.

This matched set answers, “When all four lengths are available, which one should we use?” It does not answer, “How often can each model make a forecast?” Report that separately: the fraction of real deployment origins with enough history, the fraction needing padding, and their error under the intended fallback policy. A short-context model may be operationally preferable because it covers recently commissioned sites even when its matched-set accuracy is slightly worse.

Keep the rest of the experiment controlled. Give each transformer the same input columns, target, forecast horizon, backbone family, loss definition, optimisation budget, and evaluation origins. A positional encoding or architecture capacity must support the longest candidate; verify this in the actual configuration. Record whether the same target positions were used for training as well as evaluation. If longer contexts reduce the number of eligible training positions, you can either train all lengths on their common eligible positions for an isolated context study, or let each use every position it can serve for a deployment-oriented study. Both are useful, but they answer different questions. Do not mix their results in one ranking.

Preprocessing stride needs the same attention. Overlapping windows increase the number of examples, but they do not necessarily increase the number of distinct targets contributing to loss. Count target positions and padded positions for each candidate, not just stored windows or raw rows. With a one-step target, a wide window and a coarse stride can expose hundreds of inputs while supervising only its final target. A smaller stride buys denser supervision at storage and compute cost.

Comparison control	Matched-context experiment	Deployment check
Forecast origins	Same site-hour-target tuples for every L	All origins the candidate can serve, with the stated fallback.
Inputs	Same fields and as-of rules; only history truncated	Inputs exactly as they arrive in service.
Training targets	Prefer a common target-position manifest	Allow realistic coverage, then report the difference.
Metrics	Paired errors at identical origins	Error plus coverage, padding, and latency.
Decision	Whether extra history adds information	Whether the chosen design is usable in operation.

Use splits that respect time and identity

Put the final evaluation period after the training period. Use an intervening validation period to choose context length and tune other settings; reserve the final period for one locked comparison. Rolling forecast origins within a period are appropriate because each forecast uses only information available before its own target. Forecasting: Principles and Practice describes this rolling-origin logic. If there are many sites, keep the same calendar cutoffs across sites so that a shared shock or season change appears in the correct partition.

History can cross a split boundary in a legitimate forecast: the first validation target may use observations from the end of training, because those observations were known at its issue time. The target itself and later observations may not enter its input. Some pipelines instead impose a gap; that is appropriate when data arrive late, labels are revised, or the serving contract forbids recent readings. State the gap in hours, and apply it to every candidate. A blanket rule that no input window may touch an earlier split can make long-context evaluation unnecessarily sparse and answer a different question from production.

Fit transforms on training data only. That includes numerical scaling, categorical maps, imputation parameters, thresholds for event slices, and any learned calendar or site embeddings. Do not calculate a “normal” load level from the complete series and feed it into earlier forecasts. Audit each input’s availability time, not just its timestamp: a retrospective operating-mode correction may be stamped t but have arrived after t+1.

Evaluate more than one cutoff if the data permit. A context that works in one summer month may lose to a shorter context after a maintenance change or seasonal transition. Refit using only data before each cutoff, choose configurations on that cutoff’s validation interval, and compare paired errors on its later test interval. Adjacent hourly errors within one site are correlated. For uncertainty, group resampling by site and/or contiguous time block; treating each overlapping window as an independent observation makes intervals too narrow.

Give the transformer something worth beating

A longer sequence model is useful only if it beats cheaper ways to use the same information. For the running example, evaluate at least these forecasters on the identical target origins:

1. Persistence: predict y_t. This tests how much of the task is simply inertia and is especially informative around jumps.
2. Seasonal naive: predict the corresponding hour yesterday and, where available, last week. It establishes how much daily or weekly repetition can be captured without fitting a model.
3. Calendar and recent-history model: fit a regularised linear or tree-based model using recent lags, calendar fields, temperature, and mode. This tests whether engineered summaries replace hundreds of raw tokens.
4. Sequence transformer: compare the specified context grid under a common training and evaluation contract. A recurrent model or patch-based model is a reasonable additional comparator if sequence architecture itself is at issue.

Be fair about information. A 24-hour transformer with no calendar field and a regression baseline with hour_of_week answer different feature questions. Either provide the same known features or label the comparison as a system-level comparison. Likewise, a seasonal naive model that reaches back one week has access to an old observation the 24-hour raw-context transformer lacks. That is acceptable as a practical baseline, but make the asymmetry visible.

The primary metric for numeric load might be mean absolute error, averaged first within site and then across sites so that a high-volume site does not dominate by default. Also report a demand-weighted metric if the business cost genuinely scales with load. If the decision concerns capacity breaches, add an operational metric that matches the threshold decision; do not infer threshold performance from MAE alone. Select thresholds on validation data, including any alarm or action threshold, and hold them fixed on test.

Inspect the abrupt-event slice without leaking it

Define a jump from the observed next-hour change, for example |y_{t+1} - y_t| above the 95th percentile of absolute changes in the training period. The percentile is illustrative; choose it with domain input and freeze it before test. Because y_{t+1} defines the slice, the slice label is available only for analysis after the forecast, never as a feature at issue time. Report the number of jump origins, their mean absolute error, and signed error separately from ordinary hours. Break jumps into upward and downward changes if their costs differ.

This slice measures forecast behaviour around large changes. It does not establish that jumps were predictable. If many are caused by unobserved interventions, all methods may fail. Compare errors immediately before and after documented mode changes, outages, or missing telemetry; check whether the model misses onset but tracks the new level once it is observed. A long-context model could win ordinary hours by following the weekly pattern and still react too slowly to a new regime. That trade-off should be visible rather than buried in one aggregate score.

A measured-data check

To show what the table looks like with actual scores, we ran a small benchmark on the UCI Appliances Energy Prediction dataset. Its 19,735 ten-minute records describe one low-energy building. Summing complete groups of six readings gives 3,289 consecutive hourly appliance-energy observations, from 11 January to 27 May 2016. The target is the next hour’s appliance energy in Wh. Inputs are past appliance energy, lighting energy, indoor temperature and humidity, and calendar phase derived from the recorded hour. We excluded the dataset’s interpolated outdoor weather fields because their availability at each historical issue time is less clear. This is one building, without a recorded operating-mode variable; it is separate from the hypothetical two-origin example above.

We used a chronological 60/20/20 split. All fitted methods trained on the same 1,301 target positions with at least 672 preceding hours; 658 later positions were used for validation and 658 still later positions for the table. Every row below uses those same 658 test targets. The calendar-and-lag model was a fixed, untuned gradient-boosted tree using recent lags, 24- and 168-hour lags and means, the latest recorded sensors, and the known target-hour calendar phase. The transformer rows were trained and inferred with Sequifier v3.1.1.0 in its Conda environment: four causal models with the same two-layer, 16-dimensional backbone, two attention heads, one training seed, and four epochs. Each run changed the raw history length and compatible stored-window width; Sequifier selected its checkpoint by validation mean squared error. Its real-valued inputs were normalized using training-split statistics. The engineered baseline has an explicit future calendar feature that the transformers did not receive; its comparison with them is a practical system comparison, while the four transformer rows isolate the length change within one architecture.

The jump threshold was set once from training data: an observed next-hour change greater than 1,060 Wh, the training period’s 95th percentile of absolute hourly changes. It identified 20 test targets; the other 638 are ordinary hours. The slice label was never an input. MAE columns are in Wh. “Coverage of deployment origins” is a cold-start proxy: among all 3,288 possible one-hour-ahead origins in this single recorded history, it is the fraction with enough unpadded history for that method. All 658 scored test origins have enough history for every method. Latency is the p95 of 658 warmed, single-origin, in-memory CPU calls on one eight-core arm64 Mac, excluding file I/O and hourly aggregation. Transformer timings call Sequifier’s PyTorch inference API with compilation disabled; baseline timings call their own Python implementations. Peak memory is each fresh scoring worker’s process-resident high-water mark in MiB, including its runtime and in-memory data. Runtime and library overhead therefore differ between the Sequifier and baseline rows. The direct arithmetic baselines were below the timing method’s useful hundredth-millisecond precision.

Candidate	Same-origin MAE	Jump MAE / count	Ordinary MAE / count	Coverage of deployment origins	p95 inference latency	Peak memory
Persistence	198.8 Wh	1,394.0 Wh / 20	161.4 Wh / 638	100.0%	<0.01 ms	29.5 MiB
Weekly naive	248.4 Wh	865.0 Wh / 20	229.0 Wh / 638	94.9%	<0.01 ms	28.9 MiB
Calendar and lag model	170.5 Wh	905.4 Wh / 20	147.5 Wh / 638	94.9%	10.18 ms	183.6 MiB
Transformer, 24 hours	203.4 Wh	971.1 Wh / 20	179.3 Wh / 638	99.3%	0.72 ms	469.5 MiB
Transformer, 168 hours	196.7 Wh	950.6 Wh / 20	173.1 Wh / 638	94.9%	0.89 ms	489.6 MiB
Transformer, 336 hours	197.4 Wh	935.6 Wh / 20	174.3 Wh / 638	89.8%	1.24 ms	518.8 MiB
Transformer, 672 hours	202.6 Wh	943.9 Wh / 20	179.3 Wh / 638	79.6%	2.98 ms	588.1 MiB

On these targets, the calendar-and-lag model has the lowest overall MAE. The 168-hour Sequifier transformer is the best of the four tested contexts by overall MAE, while the 336-hour run has the lowest jump-slice MAE among them. Weekly naive is worst overall but has the lowest jump-slice MAE of all seven candidates, illustrating why the slice changes the interpretation. The tree’s 10.18 ms latency reflects this unbatched scikit-learn call and should not be read as an inherent disadvantage of tree models. Only 20 jumps were observed in test, and there is one site, one temporal cutoff, one model size, one training seed, and four training epochs. The figures cannot establish a generally optimal length, stable jump performance, or a general ranking of Sequifier against these baselines. They show how the evaluation exposes trade-offs that a single aggregate score would hide.

Include training time and stored-window size in the experiment record even if they are not the final service bottleneck. The context choice is often a Pareto decision: a small gain on ordinary hours may not justify a large increase in latency, or it may be worthwhile if the expensive hours matter most. Decide the acceptable latency and memory budget before reading test results.

Map the experiment into Sequifier

The benchmark above used Sequifier v3.1.1.0 for the transformer arm. Its README describes multivariate tabular sequences and a configurable causal objective. The preprocessing guide and training guide distinguish the stored window_length from the model’s global_training.context_length. For this one-hour-ahead task, the tested physical width was L + 1: L input rows and one held-out target row, with preprocessing max_target_offset: 1 and training target_offset: 1. Thus L = 168 used a stored width of 169; L = 672 used 673. This window arithmetic defines the input contract; it does not imply that either length will perform best elsewhere.

Prepare one audited source table and one split policy. For shared calendar boundaries across sites, Sequifier documents split_method: value_cutoff with a split_column and ordered split_values; a row equal to a cutoff enters the later split. A timestamp split column can be retained for traceability but cannot be a model input or target. Derive hour_of_week separately and check that every feature was available at the forecast issue time. For each candidate, use the same feature and target definitions, set the context and compatible stored width, then inspect generated windows and their target positions. Keep the backbone and decoder settings comparable, while ensuring max_context_length can accommodate the intended input. Sequifier’s preprocessing window_stride controls stored windows; training’s global_training.window_stride controls additional model views inside a longer stored window. They are different settings. The preprocessing guide warns that with a one-position prediction length, a coarse stored stride can make target supervision sparse. Check the actual count of loss positions before treating runs as matched.

For dense validation and test coverage, the guide documents prediction_aligned_splits. With a one-step target and one alignment step, the conceptual setting is to align validation and test split indices, set prediction_length: 1, and keep target_offset equal to max_target_offset. The inference guide says aligned inference uses window_stride: null and labels outputs by their target itemPosition. Join those identifiers back to the original observations and to the shared origin manifest; verify a few joins by hand. Do not assume that equal raw split ratios automatically produce equal scored positions across context lengths, particularly near sequence starts and cutoffs.

Sequifier can also export causal embeddings. That supports a related but separate question: does a longer history produce a representation that transfers to a downstream task? Train the causal models, freeze each one, extract embeddings at matched origins, and fit the same small downstream probe on the same training labels. Evaluate the probe on later, untouched labels and compare it with a lag-feature baseline. For prediction-aligned splits with target_offset: 1, an embedding labeled by row N represents the context before row N; it does not contain that row’s value, as the preprocessing guide specifies. That alignment is valuable for leakage control. A useful embedding on one downstream task still does not demonstrate general foundation-model transfer; broader reuse needs evaluation across tasks and datasets. The distinction is discussed in our foundation-model explainer.

Keep the product’s role proportionate. Sequifier handles sequence windowing, causal training, and inference or embedding export under a documented configuration. It does not choose a scientifically correct context length for the application, repair timestamp semantics, supply the baseline models, or make the jump slice meaningful. Those remain experiment-design decisions.

Choose a length from evidence, not from the longest curve

Use validation results to identify the smallest candidate whose matched-origin performance is materially better for the operational objective. Require the result to survive the event slice, relevant site groups, and more than one temporal cutoff when possible. Then check whether its coverage, p95 latency, and memory meet service constraints. Lock the choice and evaluate it once on the final period. If 168 and 672 are effectively tied within the uncertainty of the test, prefer the less costly or more widely available context unless there is a domain reason to pay for four weeks of history.

A few outcomes would change the recommendation. If weekly naive wins, the cycle may be simple enough that raw long-context attention adds little. If 24-hour context with calendar features matches 672 hours, old measurements may be redundant. If 672 hours wins only on stable sites and loses after operating-mode changes, consider a regime-aware model or a fallback rather than one universal window. If no approach improves jump forecasts, seek earlier causal signals or redefine the task as detection once new measurements arrive. Context length is then not the limiting factor.

The experiment should end with an explicit record: target and issue-time contract, candidate lengths, eligible-origin manifest, split cutoffs, as-of feature rules, target-position counts, metrics by slice, paired uncertainty, and resource costs. That record is more useful than a lone “best length.” It tells the next engineer which history was actually available, which task the model solved, and whether the extra context earned its place.