Skip to content
Criée

Publishing market data

By level or by order, numbered multicast: detecting a gap and repairing it before display.

Lesson 59 min

By the end of this lesson

  • Calculate the size of a MoldUDP64 packet from the number of messages
  • Spot a gap in a numbered feed and say which messages to request again
  • Explain why a market data feed publishes its prices as integers
In this lesson4

The engine has matched two orders: the market must know, at once and in full. Hundreds of screens, programs and databases rebuild the book from what they are sent. What you will learn here: a market data feed is a sequence of numbered messages that a subscriber applies in order, and the number lets it see and repair a gap.

Two ways to publish a book

Market data

The venue's publication of the orders in the book and of the executions, as numbered messages sent to every subscriber at the same moment.

  • By level: for each price, the total quantity and the number of orders. That is what a trading screen shows; the BVMAC's 2008 draft instruction provided the best limit, market by order and the five best aggregated limits (instruction no. 2, 2008).
  • By order: each addition, execution or cancellation of an order is a message; the subscriber rebuilds the full book itself. That is the principle of Nasdaq's ITCH feed, published in full (TotalView-ITCH 5.0).

In ITCH, an "A" add message is 36 bytes: type, security code, tracking number, timestamp in nanoseconds since midnight, order reference (unique for the day), side, quantity, symbol and price. Numbers are binary integers, and prices have four implied decimals. An "E" message reports an execution on a reference already published, "X" a partial cancel, "D" a deletion. The choice of integers is the same as in @criee/marche: a price is never stored as a floating-point number.

Why can a by-order feed rebuild a by-level feed, but not the other way round?

Send once, to everyone

Nasdaq carries ITCH over MoldUDP64, by UDP multicast: each packet is sent only once, whatever the number of receivers (MoldUDP64 1.00). The 20-byte header carries the session, the number of the first message and the message count; the following ones are numbered implicitly. A packet with no message is a heartbeat, usually once a second, announcing the next number: even when things are quiet, a receiver sees that it missed something. UDP does not guarantee delivery; a receiver that sees a gap asks a retransmission server for the missing messages, which sends them to it alone.

This choice has a fairness consequence: a single transmission goes to everyone at the same time, instead of a loop that would serve subscribers one after another.

The size of a packetFictional

A MoldUDP64 packet carries three ITCH add messages. Each message is preceded by 2 bytes of length.

  1. Header: 20 bytes.
  2. One block: 2 + 36 = 38 bytes.
  3. Three blocks: 3 × 38 = 114 bytes.
  4. Packet: 20 + 114 = 134 bytes.

If the first message is number 5,210, the next two are 5,211 and 5,212. The next packet should start at 5,213; if it starts at 5,216, then 5,213, 5,214 and 5,215 are missing: three messages to request again.

Writing nn for the number of messages in a packet, mm for the size of one message in bytes and kk for the number of the first message, the packet size TT and the number aa the next packet must start with are:

T=20+n×(2+m)a=k+nT = 20 + n \times (2 + m) \qquad a = k + n

A gap, then a repair

The animation publishes six messages computed by @criee/marche: three adds, two executions, one deletion. Message 4 is lost.

A feed, a gap, a retransmission1/7

Published by the engine

#1A v1 S 50 @ 10,100

Subscriber’s book

-

50 @ 10,100

Engine’s book

-

50 @ 10,100

identical

expected 2

Message 1

The engine publishes “A v1”: a sell of 50 at 10,100 enters the book. Serge’s screen (fictional) adds it.

The subscriber must never apply message 5 before 4: its book would be wrong without it knowing. It waits, asks, then applies in order.

import type { Sens } from '@criee/marche'

type MessageFlux =
  | { seq: number; type: 'A'; ref: string; sens: Sens; quantite: number; prix: number }
  | { seq: number; type: 'E'; ref: string; quantite: number; prix: number }
  | { seq: number; type: 'D'; ref: string }

interface LivreAbonne { ordres: Map<string, { sens: Sens; quantite: number; prix: number }>; attendu: number }

function appliquer(l: LivreAbonne, m: MessageFlux): { livre: LivreAbonne; trou: number | null } {
  if (m.seq !== l.attendu) return { livre: l, trou: m.seq > l.attendu ? l.attendu : null }
  const ordres = new Map(l.ordres)
  if (m.type === 'A') ordres.set(m.ref, { sens: m.sens, quantite: m.quantite, prix: m.prix })
  if (m.type === 'E') {
    const o = ordres.get(m.ref)
    if (o && o.quantite > m.quantite) ordres.set(m.ref, { ...o, quantite: o.quantite - m.quantite })
    else ordres.delete(m.ref)
  }
  if (m.type === 'D') ordres.delete(m.ref)
  return { livre: { ordres, attendu: l.attendu + 1 }, trou: null }
}

Arriving late

A subscriber starting at 11 am does not have the 9 am messages. Two design choices:

  1. Replay everything from message 1 of the session. Simple, but slow late in the day.
  2. A snapshot: the venue regularly publishes the full state of the book with the number of the last message it includes; the subscriber loads it, then applies the following messages. The developer track shows the classic snapshot mistake (Logging and replaying a session).

A subscriber has applied up to message 120 and receives a packet starting at 124 with 3 messages. What does it do?

Your turn

In MoldUDP64, a packet that contains no message serves...

1In MoldUDP64, a packet that contains no message serves...

How many bytes for a packet of five ITCH add messages?

2How many bytes for a packet of five ITCH add messages?

The subscriber expects 40 and receives 42. It must...

3The subscriber expects 40 and receives 42. It must...

Why publish prices as integers rather than floats?

4Why publish prices as integers rather than floats?

Read next: Nasdaq, TotalView-ITCH 5.0 et MoldUDP64

A question? Ask your teacher, or an AI assistant, giving it the title of this lesson.

Share:WhatsAppLinkedIn

Going further