Skip to content
Criée

Failing without losing anything

Log written and replicated before processing, a single primary, replay on the secondary, outputs never published twice.

Lesson 79 min

By the end of this lesson

  • Explain why the log is replicated before an entry is processed
  • Calculate how long a replay takes from the log size and the rate
  • Explain the risk of two machines that both believe they are primary
In this lesson3

A machine will fail during a session: the only question is when. On that day, no acknowledged order must disappear, no execution must be published twice, and trading must resume quickly. What you will learn here: with a log written before processing and replicated, a secondary replays the inputs and reaches exactly the primary's state.

What a recovery must achieve

In 2024 the BVMAC launched the equipment of a replication site for the "real-time duplication of market data" from a main site to a secondary site, in order to "minimise the risks of loss of market data" (tender no. 016/09/BVMAC/DSI/07-24). Its general rules provide backup trading stations on its premises, switching members to the backup site and, if needed, suspending or postponing the session (RG BVMAC 2025, art. 29 and 137). No figure for the delay is given.

Elsewhere, figures exist. In Europe, a venue must be able to resume trading within about two hours of an incident, with data loss "close to zero" (Delegated Regulation (EU) 2017/584, art. 15). For post-trade infrastructures, the international principles also aim for critical systems to resume within two hours, from a secondary site, and state that backup systems should ideally start processing immediately (CPMI-IOSCO, PFMI, principle 17).

Failover

Moving processing from a failed primary machine to a secondary machine, which rebuilds the state by replaying the replicated log and becomes the only primary.

Four rules

  1. Write before processing. The sequencer writes the input to the log, and copies it to the secondary, before the engine processes it (lesson 4).
  2. Acknowledge only after replication. The member receives the acknowledgement only once the input is on two sites. An acknowledged input therefore cannot be lost.
  3. Only one primary. The secondary takes over only once it is sure the old primary no longer publishes. Two primaries numbering on their own produce two truths.
  4. Do not publish twice. Each output carries a number; after the replay, only the outputs beyond the last published number go out.

Why wait for replication before acknowledging an order?

A crash, then a replay

The animation logs six orders; the primary dies after writing and replicating the fifth, before publishing its result. The states and their fingerprints are computed by @criee/marche.

A crash, then a replay1/7

Replicated log

#1v1 sell 40 @ 10,100
#2v2 sell 30 @ 10,050
#3…
#4…
#5…
#6…

Primary · active

processed up to #2

state c5c9cbdd

Secondary

processed up to #0

state -

log → disk + secondary, then processing

Write first, process second

Each entry is written to the log and copied to the secondary before the primary processes it. Two sells enter the book.

The LMAX architecture described in 2011 goes further: several processors handle every input in parallel and only one output is used, which allows almost immediate failover; a full restart, nightly snapshot plus the day's replay, took less than a minute there (Martin Fowler, "The LMAX Architecture"). It is a company's account, not a standard.

import { Carnet, type Execution, type Ordre } from '@criee/marche'

interface Entree { seq: number; ordre: Omit<Ordre, 'rang'> }
interface Sortie { seq: number; executions: Execution[] }

/** Rebuilds the state and returns only the outputs the market has not seen yet. */
function reprendre(journal: Entree[], dernierPublie: number): { carnet: Carnet; aPublier: Sortie[] } {
  const carnet = new Carnet(10000)
  const aPublier: Sortie[] = []
  for (const e of [...journal].sort((a, b) => a.seq - b.seq)) {
    const executions = carnet.passer(e.ordre)
    if (e.seq > dernierPublie) aPublier.push({ seq: e.seq, executions })
  }
  return { carnet, aPublier }
}

The book is rebuilt in full, even for inputs already published: what counts is their effect on the state, not their message.

How long does the replay take?Fictional

Fictional figures. At 2 pm, the session log holds 1,200,000 entries. The engine replays 400,000 per second.

  1. Without a snapshot: 1,200,000 ÷ 400,000 = 3 seconds of replay.
  2. With a snapshot taken at entry 1,000,000: 200,000 entries remain, i.e. 200,000 ÷ 400,000 = 0.5 second, plus loading the snapshot.
  3. Detection: if the primary is declared dead after 3 seconds without a heartbeat, recovery takes about 3 + 0.5 = 3.5 seconds, far from the two hours of the European text.

The main risk is not the replay time but detection that is too quick and fails over from a primary that is merely slow: rule 3 then applies.

Writing NN for the number of entries in the log, II for the entry at which the snapshot was taken (I=0I = 0 without a snapshot) and vv for the replay rate in entries per second, the replay time tt, not counting loading the snapshot, is:

t=N−Ivt = \frac{N - I}{v}

The developer track details the log format and the trap of the snapshot that forgets the rank counter (Logging and replaying a session).

The last published number is 4; the log holds entries 1 to 5. After the replay, what does the secondary publish?

Your turn

The log is replicated before processing so that...

1The log is replicated before processing so that...

The European delegated regulation aims for trading to resume within...

2The European delegated regulation aims for trading to resume within...

Log of 900,000 entries, replay at 300,000 per second. The full replay takes...

3Log of 900,000 entries, replay at 300,000 per second. The full replay takes...

Two machines believe they are primary at the same time. The risk is...

4Two machines believe they are primary at the same time. The risk is...

Read next: CPMI-OICV, Principes pour les infrastructures de marchés financiers (2012)

A question? Ask your teacher, or an AI assistant, giving it the title of this lesson.

Share:WhatsAppLinkedIn

Going further