Publishing market data
By level or by order, numbered multicast: detecting a gap and repairing it before display.
Lesson 59 min
By the end of this lesson
- Calculate the size of a MoldUDP64 packet from the number of messages
- Spot a gap in a numbered feed and say which messages to request again
- Explain why a market data feed publishes its prices as integers
The engine has matched two orders: the market must know, at once and in full. Hundreds of screens, programs and databases rebuild the book from what they are sent. What you will learn here: a market data feed is a sequence of numbered messages that a subscriber applies in order, and the number lets it see and repair a gap.
Two ways to publish a book
Market data
The venue's publication of the orders in the book and of the executions, as numbered messages sent to every subscriber at the same moment.
- By level: for each price, the total quantity and the number of orders. That is what a trading screen shows; the BVMAC's 2008 draft instruction provided the best limit, market by order and the five best aggregated limits (instruction no. 2, 2008).
- By order: each addition, execution or cancellation of an order is a message; the subscriber rebuilds the full book itself. That is the principle of Nasdaq's ITCH feed, published in full (TotalView-ITCH 5.0).
In ITCH, an "A" add message is 36 bytes: type, security code, tracking number, timestamp in nanoseconds since midnight, order reference (unique for the day), side, quantity, symbol and price. Numbers are binary integers, and prices have four implied decimals. An "E" message reports an execution on a reference already published, "X" a partial cancel, "D" a deletion. The choice of integers is the same as in @criee/marche: a price is never stored as a floating-point number.
Why can a by-order feed rebuild a by-level feed, but not the other way round?
Send once, to everyone
Nasdaq carries ITCH over MoldUDP64, by UDP multicast: each packet is sent only once, whatever the number of receivers (MoldUDP64 1.00). The 20-byte header carries the session, the number of the first message and the message count; the following ones are numbered implicitly. A packet with no message is a heartbeat, usually once a second, announcing the next number: even when things are quiet, a receiver sees that it missed something. UDP does not guarantee delivery; a receiver that sees a gap asks a retransmission server for the missing messages, which sends them to it alone.
This choice has a fairness consequence: a single transmission goes to everyone at the same time, instead of a loop that would serve subscribers one after another.
The size of a packetFictional
A MoldUDP64 packet carries three ITCH add messages. Each message is preceded by 2 bytes of length.
- Header: 20 bytes.
- One block: 2 + 36 = 38 bytes.
- Three blocks: 3 × 38 = 114 bytes.
- Packet: 20 + 114 = 134 bytes.
If the first message is number 5,210, the next two are 5,211 and 5,212. The next packet should start at 5,213; if it starts at 5,216, then 5,213, 5,214 and 5,215 are missing: three messages to request again.
Writing for the number of messages in a packet, for the size of one message in bytes and for the number of the first message, the packet size and the number the next packet must start with are:
A gap, then a repair
The animation publishes six messages computed by @criee/marche: three adds, two executions, one deletion. Message 4 is lost.
Published by the engine
Subscriber’s book
-
50 @ 10,100
Engine’s book
-
50 @ 10,100
expected 2
Message 1
The subscriber must never apply message 5 before 4: its book would be wrong without it knowing. It waits, asks, then applies in order.
import type { Sens } from '@criee/marche'
type MessageFlux =
| { seq: number; type: 'A'; ref: string; sens: Sens; quantite: number; prix: number }
| { seq: number; type: 'E'; ref: string; quantite: number; prix: number }
| { seq: number; type: 'D'; ref: string }
interface LivreAbonne { ordres: Map<string, { sens: Sens; quantite: number; prix: number }>; attendu: number }
function appliquer(l: LivreAbonne, m: MessageFlux): { livre: LivreAbonne; trou: number | null } {
if (m.seq !== l.attendu) return { livre: l, trou: m.seq > l.attendu ? l.attendu : null }
const ordres = new Map(l.ordres)
if (m.type === 'A') ordres.set(m.ref, { sens: m.sens, quantite: m.quantite, prix: m.prix })
if (m.type === 'E') {
const o = ordres.get(m.ref)
if (o && o.quantite > m.quantite) ordres.set(m.ref, { ...o, quantite: o.quantite - m.quantite })
else ordres.delete(m.ref)
}
if (m.type === 'D') ordres.delete(m.ref)
return { livre: { ordres, attendu: l.attendu + 1 }, trou: null }
}
Arriving late
A subscriber starting at 11 am does not have the 9 am messages. Two design choices:
- Replay everything from message 1 of the session. Simple, but slow late in the day.
- A snapshot: the venue regularly publishes the full state of the book with the number of the last message it includes; the subscriber loads it, then applies the following messages. The developer track shows the classic snapshot mistake (Logging and replaying a session).
A subscriber has applied up to message 120 and receives a packet starting at 124 with 3 messages. What does it do?
Your turn
Read next: Nasdaq, TotalView-ITCH 5.0 et MoldUDP64
A question? Ask your teacher, or an AI assistant, giving it the title of this lesson.