Failing without losing anything
Log written and replicated before processing, a single primary, replay on the secondary, outputs never published twice.
Lesson 79 min
By the end of this lesson
- Explain why the log is replicated before an entry is processed
- Calculate how long a replay takes from the log size and the rate
- Explain the risk of two machines that both believe they are primary
In this lesson3
A machine will fail during a session: the only question is when. On that day, no acknowledged order must disappear, no execution must be published twice, and trading must resume quickly. What you will learn here: with a log written before processing and replicated, a secondary replays the inputs and reaches exactly the primary's state.
What a recovery must achieve
In 2024 the BVMAC launched the equipment of a replication site for the "real-time duplication of market data" from a main site to a secondary site, in order to "minimise the risks of loss of market data" (tender no. 016/09/BVMAC/DSI/07-24). Its general rules provide backup trading stations on its premises, switching members to the backup site and, if needed, suspending or postponing the session (RG BVMAC 2025, art. 29 and 137). No figure for the delay is given.
Elsewhere, figures exist. In Europe, a venue must be able to resume trading within about two hours of an incident, with data loss "close to zero" (Delegated Regulation (EU) 2017/584, art. 15). For post-trade infrastructures, the international principles also aim for critical systems to resume within two hours, from a secondary site, and state that backup systems should ideally start processing immediately (CPMI-IOSCO, PFMI, principle 17).
Failover
Moving processing from a failed primary machine to a secondary machine, which rebuilds the state by replaying the replicated log and becomes the only primary.
Four rules
- Write before processing. The sequencer writes the input to the log, and copies it to the secondary, before the engine processes it (lesson 4).
- Acknowledge only after replication. The member receives the acknowledgement only once the input is on two sites. An acknowledged input therefore cannot be lost.
- Only one primary. The secondary takes over only once it is sure the old primary no longer publishes. Two primaries numbering on their own produce two truths.
- Do not publish twice. Each output carries a number; after the replay, only the outputs beyond the last published number go out.
Why wait for replication before acknowledging an order?
A crash, then a replay
The animation logs six orders; the primary dies after writing and replicating the fifth, before publishing its result. The states and their fingerprints are computed by @criee/marche.
Replicated log
Primary · active
processed up to #2
state c5c9cbdd
Secondary
processed up to #0
state -
log → disk + secondary, then processing
Write first, process second
The LMAX architecture described in 2011 goes further: several processors handle every input in parallel and only one output is used, which allows almost immediate failover; a full restart, nightly snapshot plus the day's replay, took less than a minute there (Martin Fowler, "The LMAX Architecture"). It is a company's account, not a standard.
import { Carnet, type Execution, type Ordre } from '@criee/marche'
interface Entree { seq: number; ordre: Omit<Ordre, 'rang'> }
interface Sortie { seq: number; executions: Execution[] }
/** Rebuilds the state and returns only the outputs the market has not seen yet. */
function reprendre(journal: Entree[], dernierPublie: number): { carnet: Carnet; aPublier: Sortie[] } {
const carnet = new Carnet(10000)
const aPublier: Sortie[] = []
for (const e of [...journal].sort((a, b) => a.seq - b.seq)) {
const executions = carnet.passer(e.ordre)
if (e.seq > dernierPublie) aPublier.push({ seq: e.seq, executions })
}
return { carnet, aPublier }
}
The book is rebuilt in full, even for inputs already published: what counts is their effect on the state, not their message.
How long does the replay take?Fictional
Fictional figures. At 2 pm, the session log holds 1,200,000 entries. The engine replays 400,000 per second.
- Without a snapshot: 1,200,000 ÷ 400,000 = 3 seconds of replay.
- With a snapshot taken at entry 1,000,000: 200,000 entries remain, i.e. 200,000 ÷ 400,000 = 0.5 second, plus loading the snapshot.
- Detection: if the primary is declared dead after 3 seconds without a heartbeat, recovery takes about 3 + 0.5 = 3.5 seconds, far from the two hours of the European text.
The main risk is not the replay time but detection that is too quick and fails over from a primary that is merely slow: rule 3 then applies.
Writing for the number of entries in the log, for the entry at which the snapshot was taken ( without a snapshot) and for the replay rate in entries per second, the replay time , not counting loading the snapshot, is:
The developer track details the log format and the trap of the snapshot that forgets the rank counter (Logging and replaying a session).
The last published number is 4; the log holds entries 1 to 5. After the replay, what does the secondary publish?
Your turn
Read next: CPMI-OICV, Principes pour les infrastructures de marchés financiers (2012)
A question? Ask your teacher, or an AI assistant, giving it the title of this lesson.