General

Introduction

The L2 Multicast Market Data Gateway will enable clients to receive market data at an improved latency.

Purpose

The purpose of this document is to provide all the information necessary for clients to develop to the L2 Multicast Market Data Gateway. This document outlines the service, its logic, and the message types involved.

Service Overview

Screenshot_2020-04-26_at_12.34.50.png

The L2 Multicast Market Data Gateway supports high-performance trading systems in two ways.

  1. By making use of the SBE Schema outlined in the Appendix at the end of this document, clients will be able to reduce CPU time spent on Market Data message encoding and decoding.

By subscribing to one or more multicast channels, multiple clients will be able to receive market data more quickly and - if they are subscribed to the same groups - simultaneously.

The multicast channels are grouped by trading pairs and market depths. They are distributed across a number of addresses within a multicast range.

Connectivity Overview

Transmission Standards

The multicast channels utilise UDP over IPv4 Ethernet standards. UDP header information is as defined in the IETF RFC 791 (IPv4) and RFC 768 (UDP) transmission standards. One UDP datagram will contain either a full or partitioned Market Data message in SBE-serialized format.

Prerequisites

Each client wishing to connect to the L2 Multicast Market Data Gateway must ensure they have already

  1. Registered with the exchange.

  2. Completed a colocation installation at the exchange’s data centre.

At this point, if the client has not already been given the network details to connect directly to the exchange, they will need to request these details. They will then be allocated a subnet and will need to configure their equipment accordingly.

Once this is done, the client will be ready to use the service by subscribing to specific multicast addresses.

IP Addresses and Ports

The exact multicast address and port details for each group are published in a separate configuration document. Please note you need to sign in into the exchange to open the page.

Message Formats

Message Type Description

Each Market Data message consists of a header followed by a message body with a payload.

Market data consists of snapshots, increments, and trades. There are two message types - snapshot and increment. Trades are sent in the same messages as increments.

A snapshot is the current state of the order book for some specific trading pair and depth. Snapshots are sent periodically for all trading pairs and for different depths.

An increment is an incremental update from the previous state of the order book, based on the last snapshot. On the sending side, increments are buffered into a batch. Increments can be sent from this batch

  1. Periodically, at an internally configured interval of time. The cut-off for this interval occurs for all trading pairs of a given depth at the same time.

  2. When the batch buffer has filled to an internally configured size limit. The limit cut-off occurs independently of trading pairs and depths.

This means that, if interval = 40ms and limit = 100, an increment would be sent to the client either after 40ms, or after the 100th update to the market (e.g. 100 new orders were placed), whichever comes first.

Message Fields

All price fields in the following messages are represented in a Decimal format, consisting of a mantissa and exponent in that order. To get the natural price, evaluate as follows:

price = sbe_price.mantissa * 10sbe_price.exponent

Snapshot

Snapshot message fields:

Name

Description

depth

Order book depth

symbolId

Numeric identifier of the trading pair

seqNum

Sequence number of the last increment from which the snapshot was generated

lastUpdateTime

Last timestamp of the order book update which corresponds to this snapshot (in nanoseconds)

levels

Group

Levels group

Name

Description

side

Order type (Bid=0 or Ask=1)

price

Order price

qty

Order volume, in lots

Increment

Increment message fields:

Name

Description

depth

Order book depth

symbolId

Numeric identifier of the trading pair

seqNum

Increment sequence number

increments

Group, sequence of increments

trades

Group, sequence of trades

Increments group

Name

Description

side

Update type (Bid=0 or Ask=1)

levelPrice

Price of the updated Market Data depth. May be negative, depending on the trading pair. If the market depth exists, this increment is considered to be its update, else the market depth is considered newly appeared. For fixed-depth order books, the appearance of a new price may result in the displacement of a price at a deeper level. The logic of preempting this displacement should be implemented by clients. There will not be any updates sent from this displaced deeper level until it returns to the required depth.

levelQty

Updated value of the offer’s total volume for a given price, in lots. A value of zero means that the price no longer exists in the order book. For fixed-depth order books, after liquidating a level, a deeper price may appear in the order book. Such updates will be generated by the exchange and published as separate increments.

updateTime

Last timestamp of the order book update which corresponds to the increment (nanoseconds)

Trades group

Name

Description

aggressorSide

Update type (Bid=0 or Ask=1)

tradePrice

Price of the trade. May be negative, depending on the trading pair.

tradeQty

Trade volume in lots

tradeId

Unique identifier of the trade

tradeTime

Transaction timestamp (nanoseconds)

Recommended Handling Procedure

The recommended procedure for handling UDP Market Data is outlined below for an arbitrary trading pair and market depth. The procedure and order of operations would be the same for all other trading pairs and market depths.

  1. Subscribe to the desired multicast groups.

  2. For each trading pair and depth, wait for the first snapshot.

  3. Taking into account that the snapshot may come with a delay, accumulate increments in a buffer while waiting for the snapshot.

  4. When the first snapshot with a sequence number of seq_num is received,

  1. If any increment is lost, wait for the next snapshot and continue the whole procedure from step 2.

To generate the appropriate decoders needed for handling, clients will need to make use of the schema provided in the Appendix of this document as well as an SBE tool. Comprehensive information about encoding and decoding SBE messages in binary format based on a given schema can be found at https://github.com/real-logic/simple-binary-encoding/wiki.

Sequence Numbers

The following points about sequence numbers should be kept in mind:

  • Each increment is numbered by an integer seq-num, which monotonically increases from 1 …​ N

  • Numbering is distributed independently by trading pairs and depths.

  • Trades are sent along with increments and so have a common sequence numbering with them.

  • Snapshots are mapped to increments, so the numbering is relative to increments for a specific trading pair / depth. So, the seq-num of a snapshot is same as the seq-num of the last increment by which the snapshot was formed.

Partitioning Algorithm

If the size of an increment or snapshot message does not exceed the internally defined MTU, then it will be encapsulated as the payload of the UDP datagram. If its size does exceed the MTU, it will be split between separate datagrams.

Recall that each datagram has a message header followed by a message body with a payload. The message header has the msgSeqNum field, which is designed to restore the integrity of a message from several received datagrams.

Sending Side

The partitioning algorithm of one increment or snapshot on the sending side is as follows:

  1. Break the message into fragments with a length not exceeding the MTU.

  2. For each datagram, add a header to which:

  1. Add a message fragment, or a whole message if its size does not exceed the MTU, to the datagram.

  2. Send the datagram to the multicast group.

  3. For each fragment, repeat steps 2 to 4 until the entire message is sent.

Receiving Side

The client should do the following:

  1. Wait for the first datagram with the first flag = 1, keep track of the seqNum flow of the message.

  2. To parse the datagram, make sure that msgSeqNum increases by 1 every time.

  3. Make sure that seqNum is equal to seqNum from step 1.

  4. Repeat reading and deserialization, until last = 1.

  5. Increment (or snapshot) is ready

Appendix A: Trading Pairs

Group Name

Trading Pairs

Group 1

BTCUSD, ETHBTC, LTCBTC, LTCUSD, ETHUSD, ETCBTC, ETCUSD, ETCETH, XRPBTC, EOSETH, EOSBTC, EOSUSD, NEOBTC, NEOETH, NEOUSD, TRXBTC, TRXETH, TRXUSD, BTGBTC, BTGETH, BTGUSD, XRPETH, XRPUSDT, ADABTC, ADAETH, ADAUSD, XLMBTC, XLMETH, XLMUSD, NEXOBTC, BTCDAI, ETHDAI, EOSDAI, USDDAI, ETHTUSD, BTCTUSD, USDTUSD, TUSDDAI, NEODAI, LTCDAI, XRPDAI, NEXOETH, NEXOUSD, BTCEURS, EOSEURS, ETHEURS, LTCEURS, NEOEURS, XRPEURS, EURSDAI, TRXEOS, BTCGUSD, ETHGUSD, USDTGUSD, EOSGUSD, BCHABCBTC, BCHABCUSD, BCHSVBTC, BCHSVUSD, BTCPAX, ETHPAX, USDPAX, BTCUSDC, ETHUSDC, USDUSDC, DAIUSDC, EOSPAX, BTCEOSDT, ETHEOSDT, EOSEOSDT, USDEOSDT, BCHABCETH, BCHABCDAI, BCHABCTUSD, BCHABCEURS, BTCSHORTUSD.

Group 2

BTCUSDB, ETHUSDB, USDUSDB, TUSDUSDB, BTCEURB, ETHEURB, EURBUSD, DAIUSDB, XRPUSDB, XRPEURB, BTCGBPB, ETHGBPB, USDGBPB, XRPGBPB, BTC3LUSD, BTC3SUSD, ETH3LUSD, ETH3SUSD, BTCUSDB_OTC, BTCEURB_OTC, USDUSDB_OTC, USDEURB_OTC, BTCUSD_BQX, XLMUSD_BQX, XLMBTC_BQX, BCHUSD_BQX, BCHBTC_BQX, EOSUSD_BQX, EOSBTC_BQX, ETHUSD_BQX, ETHBTC_BQX, LTCUSD_BQX, LTCBTC_BQX, XRPUSD_BQX, XRPBTC_BQX, PAXUSD_BQX, PAXBTC_BQX, USDCUSD_BQX, USDCBTC_BQX, TUSDUSD_BQX, TUSDBTC_BQX

Appendix B: SBE Schema

<?xml version="1.0" encoding="UTF-8"?> <sbe:messageSchema xmlns:sbe="http://fixprotocol.io/2016/sbe"                    package="md_l2_mcast_sbe_codec"                    id="1"                    version="0"                    semanticVersion="0.1"                    description="Market data L2 multicast framing protocol"                    byteOrder="littleEndian">     <types>         <type name="Depth" primitiveType="uint16"/>         <type name="SymbolId" primitiveType="uint64"/>         <type name="Price" primitiveType="int64"/>         <type name="Qty" primitiveType="int64"/>         <type name="TradeId" primitiveType="uint64"/>         <type name="SeqNum" primitiveType="uint64"/>         <type name="Timestamp" primitiveType="uint64"/>         <enum name="Side" encodingType="uint8">             <validValue name="Bid">0</validValue>             <validValue name="Ask">1</validValue>         </enum>         <enum name="MsgType" encodingType="char">             <validValue name="Snapshot">W</validValue>             <validValue name="Increment">X</validValue>         </enum>         <set name="HeaderFlags" encodingType="uint16">             <choice name="first">1</choice>             <choice name="last">2</choice>         </set>         <composite name="groupSizeEncoding" description="Repeating group dimensions">             <type name="blockLength" primitiveType="uint16"/>             <type name="numInGroup" primitiveType="uint16"/>         </composite>         <composite name="messageHeader" description="Message identifiers and length of message root">             <type name="blockLength" primitiveType="uint16"/>             <type name="templateId" primitiveType="uint16"/>             <type name="schemaId" primitiveType="uint16"/>             <type name="version" primitiveType="uint16"/>             <ref name="msgSeqNum" type="SeqNum"/>             <ref name="type" type="MsgType"/>             <ref name="flags" type="HeaderFlags"/>             <ref name="timestamp" type="Timestamp"/>         </composite>         <composite name="Decimal" >             <type name="mantissa" primitiveType="int64" />             <type name="exponent" primitiveType="int8" />         </composite>     </types>     <sbe:message name="Snapshot" id="1" description="L2 Market data snapshot">         <field name="depth" id="1" type="Depth"/>         <field name="symbolId" id="2" type="SymbolId"/>         <field name="seqNum" id="3" type="SeqNum"/>         <field name="lastUpdateTime" id="4" type="Timestamp"/>         <group name="levels" id="100" dimensionType="groupSizeEncoding">             <field name="side" id="5" type="Side"/>             <field name="price" id="6" type="Decimal"/>             <field name="qty" id="7" type="Qty"/>         </group>     </sbe:message>     <sbe:message name="Increment" id="2" description="L2 Market data increments">         <field name="depth" id="1" type="Depth"/>         <field name="symbolId" id="2" type="SymbolId"/>         <field name="seqNum" id="3" type="SeqNum"/>         <group name="increments" id="150" dimensionType="groupSizeEncoding">             <field name="side" id="5" type="Side"/>             <field name="levelPrice" id="6" type="Decimal"/>             <field name="levelQty" id="7" type="Qty"/>             <field name="updateTime" id="4" type="Timestamp"/>         </group>         <group name="trades" id="200" dimensionType="groupSizeEncoding">             <field name="aggressorSide" id="5" type="Side"/>             <field name="tradePrice" id="6" type="Decimal"/>             <field name="tradeQty" id="7" type="Qty"/>             <field name="tradeId" id="201" type="TradeId"/>             <field name="tradeTime" id="202" type="Timestamp"/>         </group>     </sbe:message> </sbe:messageSchema>

Appendix C: SymbolId to Trading Pair Mapping

SymbolId

Trade Pair

SymbolId

Trade Pair

SymbolId

Trade Pair

1

BTCUSD

1062

BTCTUSD

1304

USDUSDC

5

LTCBTC2

1068

TUSDDAI

1307

EOSPAX

6

LTCUSD

1071

EOSDAI

1335

BTCKRWB

25

ETHBTC

1074

NEODAI

1336

USDCKRWB

62

ETHUSD

1075

LTCDAI

1337

USDKRWB

96

ETCBTC

1078

XRPDAI

1383

BTCUSDB

97

ETCUSD

1092

BTCEURS

1384

ETHUSDB

117

XRPBTC2

1093

EOSEURS

1385

EOSUSDB

128

ETCETH

1094

ETHEURS

1390

USDUSDB

140

EOSETH

1095

LTCEURS

1391

TUSDUSDB

144

EOSBTC

1096

NEOEURS

1392

BTCEURB

145

EOSUSD

1101

XRPEURS

1393

ETHEURB

298

NEOBTC

1122

EURSDAI

1394

EOSEURB

299

NEOETH

1197

TRXEOS

1395

EURBEURS

300

NEOUSD

1224

BTCGUSD

1396

EURBUSD

333

TRXBTC

1225

ETHGUSD

1411

BTCEOSDT

334

TRXETH

1226

USDGUSD

1412

ETHEOSDT

335

TRXUSD

1230

EOSGUSD

1413

EOSEOSDT

391

BTGBTC

1274

BCHABCBTC

1415

USDEOSDT

392

BTGETH

1275

BCHABCUSD

1479

USDUSDT20

393

BTGUSD

1276

BCHSVBTC

1481

BCHABCETH

487

XRPUSD2

1277

BCHSVUSD

1482

BCHABCDAI

488

XRPETH2

1296

BTCPAX

1483

BCHABCTUSD

1001

BTCDAI

1297

ETHPAX

1484

BCHABCEURS

1002

ETHDAI

1298

USDPAX

1485

DAIUSDB

1003

USDDAI

1299

BTCUSDC

1488

BTCUSDT20

1032

ETHTUSD

1300

ETHUSDC

1509

BTCUSDT_BQ

1034

USDTUSD

1303

DAIUSDC

Please find the mapping table in a csv file below. We shall keep it updated with the latest information.

Did this answer your question?