General
Introduction
The L2 Multicast Market Data Gateway will enable clients to receive market data at an improved latency.
Purpose
The purpose of this document is to provide all the information necessary for clients to develop to the L2 Multicast Market Data Gateway. This document outlines the service, its logic, and the message types involved.
Service Overview
The L2 Multicast Market Data Gateway supports high-performance trading systems in two ways.
By making use of the SBE Schema outlined in the Appendix at the end of this document, clients will be able to reduce CPU time spent on Market Data message encoding and decoding.
By subscribing to one or more multicast channels, multiple clients will be able to receive market data more quickly and - if they are subscribed to the same groups - simultaneously.
The multicast channels are grouped by trading pairs and market depths. They are distributed across a number of addresses within a multicast range.
Connectivity Overview
Transmission Standards
The multicast channels utilise UDP over IPv4 Ethernet standards. UDP header information is as defined in the IETF RFC 791 (IPv4) and RFC 768 (UDP) transmission standards. One UDP datagram will contain either a full or partitioned Market Data message in SBE-serialized format.
Prerequisites
Each client wishing to connect to the L2 Multicast Market Data Gateway must ensure they have already
Registered with the exchange.
Completed a colocation installation at the exchange’s data centre.
At this point, if the client has not already been given the network details to connect directly to the exchange, they will need to request these details. They will then be allocated a subnet and will need to configure their equipment accordingly.
Once this is done, the client will be ready to use the service by subscribing to specific multicast addresses.
IP Addresses and Ports
The exact multicast address and port details for each group are published in a separate configuration document. Please note you need to sign in into the exchange to open the page.
Message Formats
Message Type Description
Each Market Data message consists of a header followed by a message body with a payload.
Market data consists of snapshots, increments, and trades. There are two message types - snapshot and increment. Trades are sent in the same messages as increments.
A snapshot is the current state of the order book for some specific trading pair and depth. Snapshots are sent periodically for all trading pairs and for different depths.
An increment is an incremental update from the previous state of the order book, based on the last snapshot. On the sending side, increments are buffered into a batch. Increments can be sent from this batch
Periodically, at an internally configured interval of time. The cut-off for this interval occurs for all trading pairs of a given depth at the same time.
When the batch buffer has filled to an internally configured size limit. The limit cut-off occurs independently of trading pairs and depths.
This means that, if interval = 40ms and limit = 100, an increment would be sent to the client either after 40ms, or after the 100th update to the market (e.g. 100 new orders were placed), whichever comes first.
Message Fields
All price fields in the following messages are represented in a Decimal format, consisting of a mantissa and exponent in that order. To get the natural price, evaluate as follows:
price = sbe_price.mantissa * 10sbe_price.exponent
Snapshot
Snapshot message fields:
Name | Description |
depth | Order book depth |
symbolId | Numeric identifier of the trading pair |
seqNum | Sequence number of the last increment from which the snapshot was generated |
lastUpdateTime | Last timestamp of the order book update which corresponds to this snapshot (in nanoseconds) |
levels | Group |
Levels group
Name | Description |
side | Order type (Bid=0 or Ask=1) |
price | Order price |
qty | Order volume, in lots |
Increment
Increment message fields:
Name | Description |
depth | Order book depth |
symbolId | Numeric identifier of the trading pair |
seqNum | Increment sequence number |
increments | Group, sequence of increments |
trades | Group, sequence of trades |
Increments group
Name | Description |
side | Update type (Bid=0 or Ask=1) |
levelPrice | Price of the updated Market Data depth. May be negative, depending on the trading pair. If the market depth exists, this increment is considered to be its update, else the market depth is considered newly appeared. For fixed-depth order books, the appearance of a new price may result in the displacement of a price at a deeper level. The logic of preempting this displacement should be implemented by clients. There will not be any updates sent from this displaced deeper level until it returns to the required depth. |
levelQty | Updated value of the offer’s total volume for a given price, in lots. A value of zero means that the price no longer exists in the order book. For fixed-depth order books, after liquidating a level, a deeper price may appear in the order book. Such updates will be generated by the exchange and published as separate increments. |
updateTime | Last timestamp of the order book update which corresponds to the increment (nanoseconds) |
Trades group
Name | Description |
aggressorSide | Update type (Bid=0 or Ask=1) |
tradePrice | Price of the trade. May be negative, depending on the trading pair. |
tradeQty | Trade volume in lots |
tradeId | Unique identifier of the trade |
tradeTime | Transaction timestamp (nanoseconds) |
Recommended Handling Procedure
The recommended procedure for handling UDP Market Data is outlined below for an arbitrary trading pair and market depth. The procedure and order of operations would be the same for all other trading pairs and market depths.
Subscribe to the desired multicast groups.
For each trading pair and depth, wait for the first snapshot.
Taking into account that the snapshot may come with a delay, accumulate increments in a buffer while waiting for the snapshot.
When the first snapshot with a sequence number of seq_num is received,
If any increment is lost, wait for the next snapshot and continue the whole procedure from step 2.
To generate the appropriate decoders needed for handling, clients will need to make use of the schema provided in the Appendix of this document as well as an SBE tool. Comprehensive information about encoding and decoding SBE messages in binary format based on a given schema can be found at https://github.com/real-logic/simple-binary-encoding/wiki.
Sequence Numbers
The following points about sequence numbers should be kept in mind:
Each increment is numbered by an integer seq-num, which monotonically increases from 1 … N
Numbering is distributed independently by trading pairs and depths.
Trades are sent along with increments and so have a common sequence numbering with them.
Snapshots are mapped to increments, so the numbering is relative to increments for a specific trading pair / depth. So, the seq-num of a snapshot is same as the seq-num of the last increment by which the snapshot was formed.
Partitioning Algorithm
If the size of an increment or snapshot message does not exceed the internally defined MTU, then it will be encapsulated as the payload of the UDP datagram. If its size does exceed the MTU, it will be split between separate datagrams.
Recall that each datagram has a message header followed by a message body with a payload. The message header has the msgSeqNum field, which is designed to restore the integrity of a message from several received datagrams.
Sending Side
The partitioning algorithm of one increment or snapshot on the sending side is as follows:
Break the message into fragments with a length not exceeding the MTU.
For each datagram, add a header to which:
Add a message fragment, or a whole message if its size does not exceed the MTU, to the datagram.
Send the datagram to the multicast group.
For each fragment, repeat steps 2 to 4 until the entire message is sent.
Receiving Side
The client should do the following:
Wait for the first datagram with the first flag = 1, keep track of the seqNum flow of the message.
To parse the datagram, make sure that msgSeqNum increases by 1 every time.
Make sure that seqNum is equal to seqNum from step 1.
Repeat reading and deserialization, until last = 1.
Increment (or snapshot) is ready
Appendix A: Trading Pairs
Group Name | Trading Pairs |
Group 1 | BTCUSD, ETHBTC, LTCBTC, LTCUSD, ETHUSD, ETCBTC, ETCUSD, ETCETH, XRPBTC, EOSETH, EOSBTC, EOSUSD, NEOBTC, NEOETH, NEOUSD, TRXBTC, TRXETH, TRXUSD, BTGBTC, BTGETH, BTGUSD, XRPETH, XRPUSDT, ADABTC, ADAETH, ADAUSD, XLMBTC, XLMETH, XLMUSD, NEXOBTC, BTCDAI, ETHDAI, EOSDAI, USDDAI, ETHTUSD, BTCTUSD, USDTUSD, TUSDDAI, NEODAI, LTCDAI, XRPDAI, NEXOETH, NEXOUSD, BTCEURS, EOSEURS, ETHEURS, LTCEURS, NEOEURS, XRPEURS, EURSDAI, TRXEOS, BTCGUSD, ETHGUSD, USDTGUSD, EOSGUSD, BCHABCBTC, BCHABCUSD, BCHSVBTC, BCHSVUSD, BTCPAX, ETHPAX, USDPAX, BTCUSDC, ETHUSDC, USDUSDC, DAIUSDC, EOSPAX, BTCEOSDT, ETHEOSDT, EOSEOSDT, USDEOSDT, BCHABCETH, BCHABCDAI, BCHABCTUSD, BCHABCEURS, BTCSHORTUSD. |
Group 2 | BTCUSDB, ETHUSDB, USDUSDB, TUSDUSDB, BTCEURB, ETHEURB, EURBUSD, DAIUSDB, XRPUSDB, XRPEURB, BTCGBPB, ETHGBPB, USDGBPB, XRPGBPB, BTC3LUSD, BTC3SUSD, ETH3LUSD, ETH3SUSD, BTCUSDB_OTC, BTCEURB_OTC, USDUSDB_OTC, USDEURB_OTC, BTCUSD_BQX, XLMUSD_BQX, XLMBTC_BQX, BCHUSD_BQX, BCHBTC_BQX, EOSUSD_BQX, EOSBTC_BQX, ETHUSD_BQX, ETHBTC_BQX, LTCUSD_BQX, LTCBTC_BQX, XRPUSD_BQX, XRPBTC_BQX, PAXUSD_BQX, PAXBTC_BQX, USDCUSD_BQX, USDCBTC_BQX, TUSDUSD_BQX, TUSDBTC_BQX |
Appendix B: SBE Schema
<?xml version="1.0" encoding="UTF-8"?> <sbe:messageSchema xmlns:sbe="http://fixprotocol.io/2016/sbe" package="md_l2_mcast_sbe_codec" id="1" version="0" semanticVersion="0.1" description="Market data L2 multicast framing protocol" byteOrder="littleEndian"> <types> <type name="Depth" primitiveType="uint16"/> <type name="SymbolId" primitiveType="uint64"/> <type name="Price" primitiveType="int64"/> <type name="Qty" primitiveType="int64"/> <type name="TradeId" primitiveType="uint64"/> <type name="SeqNum" primitiveType="uint64"/> <type name="Timestamp" primitiveType="uint64"/> <enum name="Side" encodingType="uint8"> <validValue name="Bid">0</validValue> <validValue name="Ask">1</validValue> </enum> <enum name="MsgType" encodingType="char"> <validValue name="Snapshot">W</validValue> <validValue name="Increment">X</validValue> </enum> <set name="HeaderFlags" encodingType="uint16"> <choice name="first">1</choice> <choice name="last">2</choice> </set> <composite name="groupSizeEncoding" description="Repeating group dimensions"> <type name="blockLength" primitiveType="uint16"/> <type name="numInGroup" primitiveType="uint16"/> </composite> <composite name="messageHeader" description="Message identifiers and length of message root"> <type name="blockLength" primitiveType="uint16"/> <type name="templateId" primitiveType="uint16"/> <type name="schemaId" primitiveType="uint16"/> <type name="version" primitiveType="uint16"/> <ref name="msgSeqNum" type="SeqNum"/> <ref name="type" type="MsgType"/> <ref name="flags" type="HeaderFlags"/> <ref name="timestamp" type="Timestamp"/> </composite> <composite name="Decimal" > <type name="mantissa" primitiveType="int64" /> <type name="exponent" primitiveType="int8" /> </composite> </types> <sbe:message name="Snapshot" id="1" description="L2 Market data snapshot"> <field name="depth" id="1" type="Depth"/> <field name="symbolId" id="2" type="SymbolId"/> <field name="seqNum" id="3" type="SeqNum"/> <field name="lastUpdateTime" id="4" type="Timestamp"/> <group name="levels" id="100" dimensionType="groupSizeEncoding"> <field name="side" id="5" type="Side"/> <field name="price" id="6" type="Decimal"/> <field name="qty" id="7" type="Qty"/> </group> </sbe:message> <sbe:message name="Increment" id="2" description="L2 Market data increments"> <field name="depth" id="1" type="Depth"/> <field name="symbolId" id="2" type="SymbolId"/> <field name="seqNum" id="3" type="SeqNum"/> <group name="increments" id="150" dimensionType="groupSizeEncoding"> <field name="side" id="5" type="Side"/> <field name="levelPrice" id="6" type="Decimal"/> <field name="levelQty" id="7" type="Qty"/> <field name="updateTime" id="4" type="Timestamp"/> </group> <group name="trades" id="200" dimensionType="groupSizeEncoding"> <field name="aggressorSide" id="5" type="Side"/> <field name="tradePrice" id="6" type="Decimal"/> <field name="tradeQty" id="7" type="Qty"/> <field name="tradeId" id="201" type="TradeId"/> <field name="tradeTime" id="202" type="Timestamp"/> </group> </sbe:message> </sbe:messageSchema>
Appendix C: SymbolId to Trading Pair Mapping
SymbolId | Trade Pair | SymbolId | Trade Pair | SymbolId | Trade Pair |
1 | BTCUSD | 1062 | BTCTUSD | 1304 | USDUSDC |
5 | LTCBTC2 | 1068 | TUSDDAI | 1307 | EOSPAX |
6 | LTCUSD | 1071 | EOSDAI | 1335 | BTCKRWB |
25 | ETHBTC | 1074 | NEODAI | 1336 | USDCKRWB |
62 | ETHUSD | 1075 | LTCDAI | 1337 | USDKRWB |
96 | ETCBTC | 1078 | XRPDAI | 1383 | BTCUSDB |
97 | ETCUSD | 1092 | BTCEURS | 1384 | ETHUSDB |
117 | XRPBTC2 | 1093 | EOSEURS | 1385 | EOSUSDB |
128 | ETCETH | 1094 | ETHEURS | 1390 | USDUSDB |
140 | EOSETH | 1095 | LTCEURS | 1391 | TUSDUSDB |
144 | EOSBTC | 1096 | NEOEURS | 1392 | BTCEURB |
145 | EOSUSD | 1101 | XRPEURS | 1393 | ETHEURB |
298 | NEOBTC | 1122 | EURSDAI | 1394 | EOSEURB |
299 | NEOETH | 1197 | TRXEOS | 1395 | EURBEURS |
300 | NEOUSD | 1224 | BTCGUSD | 1396 | EURBUSD |
333 | TRXBTC | 1225 | ETHGUSD | 1411 | BTCEOSDT |
334 | TRXETH | 1226 | USDGUSD | 1412 | ETHEOSDT |
335 | TRXUSD | 1230 | EOSGUSD | 1413 | EOSEOSDT |
391 | BTGBTC | 1274 | BCHABCBTC | 1415 | USDEOSDT |
392 | BTGETH | 1275 | BCHABCUSD | 1479 | USDUSDT20 |
393 | BTGUSD | 1276 | BCHSVBTC | 1481 | BCHABCETH |
487 | XRPUSD2 | 1277 | BCHSVUSD | 1482 | BCHABCDAI |
488 | XRPETH2 | 1296 | BTCPAX | 1483 | BCHABCTUSD |
1001 | BTCDAI | 1297 | ETHPAX | 1484 | BCHABCEURS |
1002 | ETHDAI | 1298 | USDPAX | 1485 | DAIUSDB |
1003 | USDDAI | 1299 | BTCUSDC | 1488 | BTCUSDT20 |
1032 | ETHTUSD | 1300 | ETHUSDC | 1509 | BTCUSDT_BQ |
1034 | USDTUSD | 1303 | DAIUSDC |
Please find the mapping table in a csv file below. We shall keep it updated with the latest information.