Testing a venue: load, replay, properties
Demonstrate twice the peak, compare the old and new engines on the same log, generate inputs against the invariants.
Lesson 910 min
By the end of this lesson
- Calculate the capacity to demonstrate from the peak of the last five years
- Check the conservation invariant between quantities entered, executed and remaining
- Explain what comparative replay requires and how to handle an intended difference
In this lesson4
Replacing an exchange's engine is like changing an aircraft's engine in flight: members, the depository and the regulator expect the same behaviour the next day. A unit test does not prove that. What you will learn here: three families of tests protect a venue: load, which says how far it holds; comparative replay, which says whether it behaves like the old one; properties, which say whether its invariants hold for every input.
What the texts require
The European delegated regulation on venues gives the most complete list found (Delegated Regulation (EU) 2017/584):
- before any deployment, testing methods ensuring the system does not behave in an unintended way and copes with a sharp rise in the number of messages (art. 8);
- capacity at least twice the highest number of messages per second recorded over the previous five years (art. 11);
- stress tests that start from that peak, cover all trading phases and simulate members' activity (art. 14);
- conformance testing of members, including cancel on disconnect and loss of the market data feed (art. 9), in an environment separate from production (art. 10).
The 2012 incident told in lesson 3 began with a deployment: new code wrongly installed on one server, waking up a function dead since 2005 (SEC, press release 2013-222). A test that does not cover the deployment itself does not cover that risk.
Load
Sizing a load testFictional
Fictional figures. Over five years, the recorded peak is 1,500 messages per second, inputs and outputs included.
- Capacity to demonstrate (European rule taken as a benchmark): 2 × 1,500 = 3,000 messages per second.
- Duration: 10 minutes at that rate, i.e. 3,000 × 600 = 1,800,000 messages.
- Criterion: no matching error, and a 99th percentile latency under a threshold set in advance, for example 2 milliseconds.
- Share splitting: if the number of orders is multiplied by 5 after a share split, the reference peak becomes 7,500 and the capacity to demonstrate 15,000 messages per second.
Writing for the highest number of messages per second recorded over the last five years, the capacity to demonstrate under the European benchmark is:
The last point is not theoretical: the BVMAC expects share splitting to cause "a massive increase in the number of orders" (May 2025 tender). The multiplier of 5 is fictional.
For the test to be replayable, orders are generated by a seeded pseudo-random generator: same seed, same orders, same expected results.
The peak over the last five years is 800 messages per second. What capacity should be demonstrated under the European benchmark, and why not 800?
Comparative replay
Comparative replay
A test that has the same log processed by the reference engine and by the candidate engine, then compares their outputs one by one; the first difference points to a defect or to a rule change that must be justified.
This is the most useful test for replacing an engine: it compares the new engine with the old one on real sessions, without having to write each expected case. It assumes a deterministic engine (lesson 4) and a complete log of inputs.
Reference
Candidate
journal · 4 entries
A production log
To replay real BVMAC sessions, you need the NSC V900 input log and its outputs. Their existence, format and access have not been checked: it is a question to put to the BVMAC at the very start of the project.
Not every difference is an error: if the new engine applies a new, intended rule (another auction tie-break, for example), the difference is expected, documented, and removed from the comparison.
Properties
A property-based test generates hundreds of inputs and checks that an invariant holds for each. With the fast-check library, which generates the inputs and shrinks a failure to the smallest input that reproduces it (fast-check), two engine invariants:
import fc from 'fast-check'
import { Carnet } from '@criee/marche'
/** Each generated integer becomes a limit order: side, quantity from 1 to 10, price on three levels. */
const ordre = (n: number, k: number) => ({
id: `o${k}`,
sens: n % 2 === 0 ? ('achat' as const) : ('vente' as const),
quantite: 1 + (n % 10),
limite: 9950 + 50 * (n % 3),
})
fc.assert(
fc.property(fc.array(fc.integer({ min: 0, max: 999 })), (tirages) => {
const c = new Carnet(10000)
let entre = 0
let execute = 0
tirages.forEach((n, k) => {
const o = ordre(n, k)
entre += o.quantite
execute += c.passer(o).reduce((s, e) => s + e.quantite, 0)
const f = c.fourchette()
// 1. The book is never crossed.
if (f.achat !== null && f.vente !== null && f.achat >= f.vente) throw new Error('crossed book')
})
const { achats, ventes } = c.ordres()
const reste = [...achats, ...ventes].reduce((s, o) => s + o.quantite, 0)
// 2. Conservation: each executed share consumes one buy and one sell.
return entre === reste + 2 * execute
}),
)
The developer track lists the other invariants and the format of a business test case (An engine's invariants, From business rule to test case).
Checking conservation by handFictional
Three orders: a buy of 10 at 10,000, a sell of 4 at 9,950, a sell of 8 at 10,000.
- Entered: 10 + 4 + 8 = 22 shares.
- The sell of 4 meets the buy: 4 executed. The sell of 8 meets the remaining 6: 6 executed. Total executed: 10.
- Left in the book: the sell, 8 − 6 = 2 shares.
- Check: 2 + 2 × 10 = 22. The invariant holds.
Writing for the quantity entered, for the quantity executed and for the quantity left in the book, each executed share consumes one buy and one sell, hence the conservation invariant:
Why does comparative replay not replace property-based tests?
Your turn
Read next: Règlement délégué (UE) 2017/584 (RTS 7), exigences organisationnelles des plateformes de négociation
A question? Ask your teacher, or an AI assistant, giving it the title of this lesson.