Data
We strive to provide complete raw market data capture as published by exchange' real-time WebSocket feeds. This means that data types we support vary for every exchange, for example for BitMEX we store liquidations and chat messages data in addition to tick by tick trades and order book L2 messages, but not for FTX which doesn't provide those non standard data types. See historical data details to find out captured real-time channels for each exchange.
We also provide following normalized data types via our client libs:
tick-by-tick trades
order book L2 updates
order book snapshots (tick-by-tick, 10ms, 100ms, 1s, 10s etc)
quotes
derivative tick info (open interest, funding rate, mark price, index price)
OHLCV
volume/tick based trade bars
and downloadable CSV data files:
​tick-by-tick trades​
​quotes​
​derivative tick info (open interest, funding rate, mark price, index price)
​
​
We always collect and provide data with the most granularity that exchange can offer via it's real-time feeds. High frequency means different things for each exchange so for example for Coinbase Pro it can mean L3 order book data (market-by-order), for Binance Futures all order book L2 real-time updates.
​
​
Order Book Level 2 aggregated by price data can be used to analyse among other things:
order book imbalance
average execution cost
average liquidity away from midpoint
average spread
hidden interest (i.e., iceberg orders)
We store and provide L2 data with highest resolution possible.
​
​
Order Book Level 3 data with all individual orders can be used to analyse among other things:
order resting time
order fill probability
order queue dynamics
Historical L3 data is currently available by Coinbase Pro, CoinFLEX and Bitstamp.
​
​
Yes, we do provide historical options data for Deribit and OKEx Options.
​
​
We specialize in derivatives exchanges market data and cover all top venues that trade futures contracts: BitMEX, Deribit, Binance Futures, FTX, OKEx, Huobi Futures, Bitfinex Derivatives and more.
​
​
Yes, we're always open to support new promising exchanges. Contact us and we'll get back to you as soon as possible to discuss details.
​
​
Normalized market data (unified data format for every exchange) is available via our official libraries and downloadable CSV files. Our HTTP API provides data only in exchange-native format.
​
​
Cryptocurrency markets are very fragmented and every exchange provides data in it's own bespoke data format which we call exchange-native data format. Our HTTP API provides market data in this format.
For example BitMEX trade message looks like this:
{"table":"trade","action":"insert","data":[{"timestamp":"2019-06-01T00:03:11.589Z","symbol":"ETHUSD","side":"Sell","size":10,"price":268.7,"tickDirection":"ZeroMinusTick","trdMatchID":"ebc230d9-0b6e-2d5d-f99a-f90109a2b113","grossValue":268700,"homeNotional":0.08555051758063137,"foreignNotional":22.987424073915648}]}
and Deribit trade message:
{"jsonrpc":"2.0","method":"subscription","params":{"channel":"trades.ETH-26JUN20.raw","data":[{"trade_seq":18052,"trade_id":"ETH-10813935","timestamp":1577836825724,"tick_direction":0,"price":132.65,"instrument_name":"ETH-26JUN20","index_price":128.6,"direction":"buy","amount":1.0}]}}
​
In contrast, normalized data format means the same, unified format across multiple exchanges. We provide normalized data via our client libs (data normalization is performed client-side) as well as via downloadable CSV files.
Sample normalized trade message:
{"type": "trade","symbol": "XBTUSD","exchange": "bitmex","id": "282a0445-0e3a-abeb-f403-11003204ea1b","price": 7996,"amount": 50,"side": "sell","timestamp": "2019-10-23T10:32:49.669Z","localTimestamp": "2019-10-23T10:32:49.740Z"}
​
​
UTC.
​
​
Raw market data is sourced from WebSocket real-time APIs provided by the exchanges. See details.
​
​
We're doing our best to provide the most complete and reliable historical raw data API on the market. To do so amongst many other things, we utilize highly available Kubernetes clusters on Google Cloud Platform that offer best in the class availability, networking and monitoring. However due to exchanges' APIs downtimes (maintenance, deployments, connection drops etc.) we can experience market data gaps and cannot guarantee 100% data completeness. In rare circumstances, when exchange's API changes without any notice or we hit new unexpected rate limits we also may fail to record data during such period, it happens very rarely and is very specific for each exchange. Use /exchanges/:exchange API endpoint and check for incidentReports field in order to get most detailed and up to date information on that subject.
​
​
Although is should never happen in theory, in practice due to various crypto exchanges bugs and peculiarities it can happen (very occasionally), see some posts from users reporting those issues:
We do track sequence numbers of WebSocket L2 order book messages when collecting the data and restart connection when sequence gap is detected for exchanges that do provide those numbers. Even in that scenario when sequence numbers are in check, bid/ask overlap can occur. When such scenario occurs, exchanges 'forget' to publish delete messages for the opposite side of the book when publishing new level for given side - we validated that hypothesis by comparing reconstructed order book snapshots that had crossed order book (bid/ask overlap) for which we removed order book levels for the opposite side manually (as exchange didn't publish that 'delete'), with quote/ticker feeds if best bid/ask matches (for exchanges that provide those) - see sample code that implements that manual level removal logic.
​
​
That shouldn't happen in theory, but we've detected that for some exchanges when new connection is established sometimes first message for given channel & symbol has newer timestamp than subsequent message, e.g., order book snapshot has newer timestamp than first order book update. This is why we provide data via API and CSV downloads for given data ranges based on local timestamps (timestamp of message arrival) which are always monotonically increasing.
​
​
Some exchanges are occasionally publishing duplicated trades (trades with the same ids). Since we collect real-time data we also collect and provide duplicate trades via API if those were published by real-time WebSocket feeds of exchanges. Some of our client libraries have functionality that when working with normalized data can deduplicate such trades, similarly for downloadable CSV files we deduplicate tick-by-tick trades data.
​
​
We use WebSocket protocol for real-time data collection and occasionally HTTP REST APIs for fetching initial full order book snapshots and other info for exchanges that do not provide it via WebSocket.
​
​
Collecting real-time streaming WebSocket data allows us preserving and providing the most granular data that exchanges offer in contrary to point in time snapshots which are the only available option when collecting data via periodically pooling REST API. Historical data sourced from WebSocket real-time exchanges' feeds adheres to what you'll see when trading live even if it means occasional connection drops and missing data, real-time data publishing delays by exchanges, duplicated trades or crossed books for some exchanges.
​
​
Historical market data available via HTTP API provides order book snapshots at the beginning of each day (00:00 UTC) - see details.
We also provide custom order book snapshots with customizable time intervals from tick-by-tick, milliseconds to minutes or hours via client libs in which case custom snapshots are computed client side from raw data provided via HTTP API.
​
​
Order books are collected in streaming mode - snapshot at the beginning of each day and then incremental delta updates. See details.
We do also provide custom order book snapshots with customizable time intervals from tick-by-tick, milliseconds to minutes or hours via client libs in which case custom snapshots are computed client side from raw data provided via HTTP API.
​
​
Each message received via WebSocket connection is timestamped with 100ns precision using synchronized clock at arrival time (before any message processing) and stored in ISO 8601 format.
​
​
For API access it's 4 minutes (T - 4min), downloadable CSV files for given day are available on the next day around 03:00 UTC.
​
​
​
