Forecasting electricity prices in a volatile market
A case study · Australian National Electricity Market
Mohammed Osman Gani · Zhipeng He · Chun Ouyang · Sara Khalifa
Energy Transition Centre & School of Information Systems, Queensland University of Technology
Act I · Volatility
NEM prices at a glance
A week of half-hourly Regional Reference Prices in Queensland sets out three structural conditions that any forecaster has to handle. Each appears in a different hour of the day, and each carries different downstream stakes.
↗ Midday negatives (solar oversupply)
↗ Evening spike (ramp + peak demand)
↗ Direction-flip hours
Act I · Volatility
One spike, two views
On a single evening, the settled price climbed past $1,100, sixteen times the week's typical level. AEMO's predispatch forecast, published 24 hours earlier, placed that same half-hour near the market price cap, an over-forecast of roughly an order of magnitude, and held that level after the settled price had already fallen back. The failure is one of magnitude and persistence rather than detection.
The size of the vertical gap on that hour is the forecast value at stake for every battery, retailer, and trader acting on the official forward path.
Act II · Diagnosis
Where models actually fail
A single average error number hides where forecast errors actually live. Decomposed across the 48 half-hour intervals of the day, three structural failure windows appear — and every model studied falls into all three.
Evening volatility window
Midday negative-price window
Direction-flip hours
- LSTM
- CNN-LSTM
- Transformer
- TimeMixer
- TimeXer
Act II · Diagnosis
The leaderboard surprise
Selecting a Region, Horizon, and Metric changes the ranking of the models. The benchmark's principal results show: the standard deep-learning baselines (LSTM, CNN-LSTM and Transformer) rank among the top three in nearly every setting, and none generalises across the market. Architectural sophistication alone therefore does not transfer to this market, which points toward volatility-aware modelling rather than further architectural sophistication.
The three marked models are the architectures EPF studies already rely on. Switch region, horizon, or metric: they hold the top ranks nearly every time, and the newer benchmark models re-shuffle below them. Test-split values.
- #1TransformerTransformer · standard in EPF54.94A$/MWh
- #2LSTMRNN · standard in EPF57.88A$/MWh
- #3MambaSSM58.70A$/MWh
- #4CNN-LSTMHybrid · standard in EPF59.03A$/MWh
- #5TimeXerTransformer65.85A$/MWh
- #6TimesNetCNN75.25A$/MWh
- #7iTransformerTransformer75.47A$/MWh
- #8DLinearLinear77.64A$/MWh
- #9TimeMixerMLP88.81A$/MWh
Act II · Diagnosis
The extreme prices are where it collapses
Restricted to the spike and crash prices, day-ahead MAE inflates roughly five-fold for every baseline and recent time-series model. Neither regime is solved by newer architectures.
Day-ahead (24 h) horizon.
Bridge
Spikes and crashes break things differently
The two regimes are not similar. Spike prices run past $15,000 while crashes never fall below −$200 and each arrives by a different mechanism. Spikes are scarcity: evening demand climbs just as solar falls away. Crashes are oversupply: midday demand sinks toward a solar supply that holds firm. A correction designed for one direction is not the correction needed for the other.
Demand climbs as solar falls away → price spikes
Scarcity — the demand–solar gap blows out into the evening peak.
Demand sinks toward steady solar → price goes negative
Oversupply — solar covers most of a shrinking midday load.
Act III · Treatment
A tail residual on any base
The treatment does not replace the forecaster. It wraps one — frozen. A CNN encoder reads the raw input window for local pre-event patterns; an adapter conditions a FiLM layer on the base model’s own representation, making the features tail-sensitive; two non-negative residual heads — one upward, one downward — propose asymmetric corrections; and a horizon-wise soft gate decides per step how much of each to apply. Training is two-stage: the base is fitted first and frozen, so the correction module learns only the tail residuals and calm hours stay on the base forecast.
Act III · Treatment
Asymmetric correction in action
Scrub across a single 24-hour day-ahead window that carries both failure modes: a midday dip into negative prices and an evening spike. On normal hours the gate keeps p(normal) near one and the corrected output sits on the base forecast. Around the spike, p(spike) rises and the spike head — non-negative by construction — adds magnitude; around the dip, p(crash) rises and the crash head pushes the forecast back below zero. The corrections recover most of each event, never all of it.
Horizon-wise gate — regime weight per step
Applied correctionper step
Hour 18.0 · base 513 → GTR 1419 (actual 1787) · 0.83 · 0.02
Act III · Treatment
It works
Across every base architecture studied and across QLD and NSW, the tail-residual treatment concentrates its gain on the spike and crash regimes — where the diagnosis said the gap lived. Crash-regime error falls by 12% to 52% depending on the base model, and the weaker the base architecture on the tails, the more it gains. Normal-regime accuracy is never sacrificed: it improves too, by a smaller margin. Every comparison is significant at p < 0.001 under a Diebold–Mariano test with small-sample correction. Toggle base and region to inspect.
Regimes follow the paper: spike = actual price at or above the 95th percentile, crash = at or below the 5th, the rest normal. Mean absolute error in A$/MWh on the day-ahead horizon.
Bar values are MAE in A$/MWh — lower is better.
Open threads
Where the work goes next
- Multivariate inputs. Demand, renewable output, and interconnector flows likely carry the structural signals current price-only models cannot recover from a univariate history.
- Crash dynamics specifically. Negative pricing is the regime that shifts fastest as renewable share grows; modelling it on its own terms — not as a mirrored spike — is the next direction.
- Drift under regime change. The 2023–2025 data window already covers post-COVID and post-5-min-settlement; the NEM keeps evolving, and any deployed model needs an explicit drift response.
The forecasting layer feeds AI-driven battery orchestration; tail accuracy is what unlocks the operational value that project depends on.
Benchmark, tail, and per-regime figures are the papers' reported values; the intraday and mechanism panels remain illustrative until those exports land. Code released alongside publication. ← Research · Contact the group