# Data

## What data types do you support?

We provide the most comprehensive and granular market data on the market [sourced from real-time WebSocket APIs](#why-data-source-matters-and-why-we-use-real-time-websocket-feeds-as-data-source-vs-periodically-calling-rest-endpoints) with complete [control and transparency](../../historical-data-details#market-data-collection-overview) how the data is being recorded.

Via [downloadable CSV data files](../downloadable-csv-files) following normalized **tick-level** data types are available:

* [trades](../../downloadable-csv-files#trades)
* [incremental order book L2 updates](../../downloadable-csv-files#incremental_book_l2)
* [order book snapshots](../../downloadable-csv-files#book_snapshot_25) (top 25 and top 5 levels)
* [options\_chain](../../downloadable-csv-files#options_chain)
* [quotes](../../downloadable-csv-files#quotes)
* [derivative tick info](../../downloadable-csv-files#derivative_ticker) (open interest, funding rate, mark price, index price)
* [liquidations](../../downloadable-csv-files#liquidations)

\
\
[Raw data API](../api/getting-started) that is **available for** [**pro and business** ](../billing-and-subscriptions#what-are-the-differences-between-subscriptions-types)**subscriptions** provides data in [exchange-native data format](#what-is-a-difference-between-exchange-native-and-normalized-data-format). See [historical data details](../historical-data-details) to learn about [real-time channels](#what-is-the-channel-field-used-in-the-http-api-and-client-libs-replay-functions) captured for each exchange. Each captured channel can be considered a different exchange specific data type (for example [Binance bookTicker channel](../../historical-data-details/binance#captured-real-time-channels), or [BitMEX liquidation channel](../../historical-data-details/bitmex#captured-real-time-channels)).

We also provide following [normalized data types](#what-is-a-difference-between-exchange-native-and-normalized-data-format) via our [client libs](../api/getting-started) (normalization is done client-side, using [raw data API](../api/getting-started) as a data source):

* trades
* order book L2 updates
* order book snapshots (tick-by-tick, 10ms, 100ms, 1s, 10s etc)
* quotes
* derivative tick info (open interest, funding rate, mark price, index price)
* liquidations
* options summary
* OHLCV
* volume/tick based trade bars









## What does high frequency historical data mean?

We always collect and provide data with the most granularity that exchange can offer via it's [real-time WS feeds](#why-data-source-matters-and-why-we-use-real-time-websocket-feeds-as-data-source-vs-periodically-calling-rest-endpoints). High frequency can mean different things for different exchanges due to exchanges APIs limitations. For example for [Coinbase Pro](../historical-data-details/coinbase) it can mean [L3 order book data](#what-l3-order-book-data-can-be-used-for) (market-by-order), for [Binance Futures](../historical-data-details/binance-futures) all order book [L2](#what-l2-order-book-data-can-be-used-for) real-time updates and for [Binance](../historical-data-details/binance) Spot it means order book updates aggregated in 100ms intervals.





## How historical raw market data is being sourced?

Raw market data is sourced from exchanges real-time WebSocket APIs. \
For cases where exchange lacks WebSocket API for particular data type we fallback to pooling REST API periodically, e.g.,  Binance Futures open interest data.

{% hint style="info" %}
See [market data collection overview](../../historical-data-details#market-data-collection-overview) for more details and why [data source matters](#why-data-source-matters-and-why-we-use-real-time-websocket-feeds-as-data-source-vs-periodically-calling-rest-endpoints).
{% endhint %}





## Why data source matters and why we use real-time WebSocket feeds as data source vs periodically calling REST endpoints?

Recording exchanges real-time WebSocket feeds allows us preserving and providing[ the most granular data](#what-does-high-frequency-historical-data-mean) that exchanges APIs can offer including data that is simply not available via their REST APIs like tick level order book updates.\
\
Historical data sourced from WebSocket real-time feeds adheres to what you'll see when trading live and can be used to exactly replicate live conditions even if it means some occasional [connection drops](#is-provided-raw-market-data-complete) causing [small data gaps](#is-provided-raw-market-data-complete), real-time data publishing delays especially during larger market moves, [duplicated trades](#are-exchanges-publishing-duplicated-trades-data-messages) or [crossed books](#can-historical-order-books-reconstructed-from-l2-updates-be-crossed-bid-ask-overlap-occasionally) in some edge cases. We find that trade-off acceptable and even if data isn't as clean and corrected as sourced from REST APIs, it allows for more insight into market microstructure and various unusual exchanges behaviors that simply can't be captured otherwise.\
\
Simple example would be latency spikes for many exchanges during increased volatility periods where exchange publish trade/order book/quote WebSocket messages with larger than usual latency or simply skip some of the the updates and then return those in one batch. Querying the REST API would result in nice, clean trade history, but such data wouldn't fully reflect real actionable market behavior and would result in unrealistic backtesting results, breaking  in the real-time scenarios.

{% hint style="info" %}
See [market data collection overview](../../historical-data-details#market-data-collection-overview) for more details.
{% endhint %}





## What L2 order book data can be used for?

L2 data (market-by-price) includes bids and asks orders aggregated by price level and can be used to analyze among other things:

* order book imbalance
* average execution cost
* average liquidity away from midpoint
* average spread
* hidden interest (i.e., iceberg orders)

We do provide L2 data both in [CSV format as incremental order book L2 updates](../../downloadable-csv-files#incremental_book_l2), [tick level order book snapshots](../../downloadable-csv-files#book_snapshot_25) (top 25 and top 5 levels) as well as in [exchange-native](#what-is-a-difference-between-exchange-native-and-normalized-data-format) format via [API and client libraries that can perform full order book reconstruction](../api/getting-started) client-side.





## What L3 order book data can be used for?

L3 data (market-by-order) includes every order book order addition, update, cancellation and match and can be used to analyze among other things:

* order resting time
* order fill probability
* order queue dynamics

Historical L3 data is currently available via API for [Bitfinex](../historical-data-details/bitfinex), [Coinbase Pro](../historical-data-details/coinbase) and [Bitstamp](../historical-data-details/bitstamp) - remaining supported exchanges provide [L2 data](#what-l2-order-book-data-can-be-used-for) only.





## What is the maximum order book depth available for each supported exchange?

We always collect full depth order book data as long as exchange's WebSocket API supports it. Table below shows current state of affairs for each supported exchange.

<table data-header-hidden><thead><tr><th>exchange</th><th width="284.3333333333333">order book depth</th><th>order book updates frequency</th></tr></thead><tbody><tr><td><strong>exchange</strong></td><td><strong>order book depth</strong></td><td><strong>order book updates frequency</strong></td></tr><tr><td><a href="../historical-data-details/bitmex">BitMEX</a></td><td>full order book depth snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/deribit">Deribit</a></td><td>full order book depth snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/binance-futures">Binance USDT Futures</a></td><td>top 1000 levels initial order book snapshot, full depth incremental order book updates</td><td>real-time, dynamically adjusted</td></tr><tr><td><a href="../historical-data-details/binance-delivery">Binance COIN Futures</a></td><td>top 1000 levels initial order book snapshot, full depth incremental order book updates</td><td>real-time, dynamically adjusted</td></tr><tr><td><a href="../historical-data-details/binance">Binance Spot</a></td><td>top 1000 levels initial order book snapshot, full depth incremental order book updates</td><td>100ms</td></tr><tr><td><a href="../historical-data-details/ftx">FTX</a></td><td>top 100 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/okex-futures">OKX Futures</a></td><td>top 400 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/okex-swap">OKX Swap</a></td><td>top 400 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/okex-options">OKX Options</a></td><td>top 400 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/okex">OKX Spot</a></td><td>top 400 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/huobi-dm">Huobi Futures</a></td><td>top 150 levels initial order book snapshot and updates</td><td>30ms</td></tr><tr><td><a href="../historical-data-details/huobi-dm-swap">Huobi COIN Swaps</a></td><td>top 150 levels initial order book snapshot and updates</td><td>30ms</td></tr><tr><td><a href="../historical-data-details/huobi-dm-linear-swap">Huobi USDT Swaps</a></td><td>top 150 levels initial order book snapshot and updates</td><td>30ms</td></tr><tr><td><a href="../historical-data-details/huobi">Huobi Global</a></td><td>top 150 levels initial order book snapshot and updates</td><td>100ms</td></tr><tr><td><a href="../historical-data-details/bitfinex-derivatives">Bitfinex Derivatives</a></td><td>top 100 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/bitfinex">Bitfinex</a></td><td>top 100 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/coinbase">Coinbase Pro</a></td><td>full order book depth snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/cryptofacilities">Kraken Futures</a></td><td>full order book depth snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/kraken">Kraken</a></td><td>top 1000 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/bitstamp">Bitstamp</a></td><td>full order book depth snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/gemini">Gemini</a></td><td>full order book depth snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/poloniex">Poloniex</a></td><td>full order book depth snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/bybit">Bybit</a></td><td>top 25 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/dydx">dYdX</a></td><td>full order book depth snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/upbit">Upbit</a></td><td>top 15 levels snapshots</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/phemex">Phemex</a></td><td>top 30 levels initial order book snapshot and updates</td><td>20ms</td></tr><tr><td><a href="../historical-data-details/ftx-us">FTX US</a></td><td>top 100 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/binance-us">Binance US</a></td><td>top 1000 levels initial order book snapshot, full depth incremental order book updates</td><td>100ms</td></tr><tr><td><a href="../historical-data-details/gate-io-futures">Gate.io Futures</a></td><td>top 20 levels order book snapshots</td><td>unknown</td></tr><tr><td><a href="../historical-data-details/gate-io">Gate.io</a></td><td>top 30 levels order book snapshots</td><td>unknown</td></tr><tr><td><a href="../historical-data-details/okcoin">OKCoin</a></td><td>top 400 levels initial order book snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/bitflyer">bitFlyer</a></td><td>full order book depth snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/hitbtc">HitBTC</a></td><td>full order book depth snapshot and updates</td><td>real-time</td></tr><tr><td><a href="../historical-data-details/binance-dex">Binance DEX</a></td><td>top 1000 levels initial order book snapshot, full depth incremental order book updates</td><td>100ms</td></tr></tbody></table>





## Which exchanges support liquidations data type?

[Liquidations](https://medium.com/tardis-dev/live-liquidations-monitor-for-top-cryptocurrency-exchanges-ac8e429e9556) data is sourced from exchanges WebSocket APIs when supported with fallback to pooling REST APIs when WebSockets APIs do not support that data type and can be accessed via [raw data APIs (replaying relevant channel](../api/getting-started)) or as [normalized data type via CSV downloads](../../downloadable-csv-files#liquidations).

<table data-header-hidden><thead><tr><th width="228.56469150538288">exchange</th><th width="163.5284279930997">available since</th><th>notes</th></tr></thead><tbody><tr><td><strong>exchange</strong></td><td><strong>available since</strong></td><td><strong>notes</strong></td></tr><tr><td><a href="../historical-data-details/bitmex">BitMEX</a></td><td>2019-03-30</td><td>collected from <a href="https://github.com/tardis-dev/tardis-node/blob/master/src/mappers/bitmex.ts#L191">WS liquidation</a> channel</td></tr><tr><td><a href="../historical-data-details/deribit">Deribit</a></td><td>2019-03-30</td><td>collected from <a href="https://github.com/tardis-dev/tardis-node/blob/master/src/mappers/deribit.ts#L217">WS trade</a> channel (trades with <code>liquidation</code> flag)</td></tr><tr><td><a href="../historical-data-details/binance-futures">Binance USDT Futures</a></td><td>2019-11-17</td><td>collected from <a href="https://github.com/tardis-dev/tardis-node/blob/master/src/mappers/binance.ts#L386">WS forceOrder</a> stream, since 2021-04-27 liquidation orders streams do not push realtime order data anymore, instead, they push snapshot order data at a maximum frequency of 1 order push per second</td></tr><tr><td><a href="../historical-data-details/binance-delivery">Binance COIN Futures</a></td><td>2020-06-16</td><td>collected from <a href="https://github.com/tardis-dev/tardis-node/blob/master/src/mappers/binance.ts#L386">WS forceOrder</a> stream, since 2021-04-27 liquidation orders streams do not push realtime order data anymore, instead, they push snapshot order data at a maximum frequency of 1 order push per second</td></tr><tr><td><a href="../historical-data-details/ftx">FTX</a></td><td>2019-08-01</td><td>collected from <a href="https://github.com/tardis-dev/tardis-node/blob/master/src/mappers/ftx.ts#L159">WS trades</a> channel (trades with <code>liquidation</code> flag) </td></tr><tr><td><a href="../historical-data-details/okex-futures">OKX Futures</a></td><td>2020-12-18</td><td>collected by pooling OKEx REST APIs since liquidations aren't available via WS feeds</td></tr><tr><td><a href="../historical-data-details/okex-swap">OKX Swap</a></td><td>2020-12-18</td><td>collected by pooling OKEx REST APIs since liquidations aren't available via WS feeds</td></tr><tr><td><a href="../historical-data-details/huobi-dm">Huobi Futures</a></td><td>2020-06-24</td><td>collected from <a href="https://github.com/tardis-dev/tardis-node/blob/master/src/mappers/huobi.ts#L377">WS liquidation_orders</a> channel</td></tr><tr><td><a href="../historical-data-details/huobi-dm-swap">Huobi Swap</a></td><td>2020-06-24</td><td>collected from <a href="https://github.com/tardis-dev/tardis-node/blob/master/src/mappers/huobi.ts#L377">WS liquidation_orders</a> channel</td></tr><tr><td><a href="../historical-data-details/bitfinex-derivatives">Bitfinex Derivatives</a></td><td>2019-09-14</td><td>collected from <a href="https://github.com/tardis-dev/tardis-node/blob/master/src/mappers/bitfinex.ts#L289">WS liquidations </a>channel</td></tr><tr><td><a href="../historical-data-details/cryptofacilities">Kraken Futures</a></td><td>2019-03-30</td><td>collected from <a href="https://github.com/tardis-dev/tardis-node/blob/master/src/mappers/cryptofacilities.ts#L141">WS trade</a> channel (trades with <code>liquidation</code> type) </td></tr><tr><td><a href="../historical-data-details/bybit">Bybit</a></td><td>2020-12-18</td><td>up until 2021-09-20 collected by pooling Bybit REST APIs since liquidations weren't available via WS feeds, starting from 2021-09-20 collected from <a href="https://github.com/tardis-dev/tardis-node/blob/master/src/mappers/bybit.ts#L201">WS liquidation</a> channel</td></tr></tbody></table>





## Do you provide historical options data?&#x20;

Yes, we do provide historical options data for [Deribit](../historical-data-details/deribit) and [OKEx Options](../historical-data-details/okex-options) - see [options chain](../../downloadable-csv-files#options_chain) CSV data type and [Deribit](../historical-data-details/deribit) and [OKEx Options](../historical-data-details/okex-options)  exchange details pages.





## Do you provide historical futures data?

We cover all leading derivatives exchanges such as [BitMEX](../historical-data-details/bitmex), [Deribit](../historical-data-details/deribit), [Binance USDT Futures](../historical-data-details/binance-futures), [Binance COIN Futures](../historical-data-details/binance-delivery), [FTX](../historical-data-details/ftx), [OKEx](../historical-data-details/okex-futures), [Huobi Futures](../historical-data-details/huobi-dm), [Huobi Swap](../historical-data-details/huobi-dm-swap), [Bitfinex Derivatives](../historical-data-details/bitfinex-derivatives), [Bybit](../historical-data-details/bybit) and [many more.](../historical-data-details)





## What is the difference between futures and perpetual swaps contracts?

Futures contract is a contract that has expiry date (for example quarter ahead for quarterly futures). Futures contract price converges to spot price as the contract approaches expiration/settlement date.\
After futures contract expires, exchange settles it and replaces with a new contract for the next period (next quarter for our previous example).&#x20;

Perpetual swap contract also commonly called "perp", "swap", "perpetual" or "perpetual future" in crypto exchanges nomenclature is very similar to futures contract, but does not have expiry date (hence perpetual). In order to ensure that the perpetual swap contract price stays near the spot price exchanges employ mechanism called funding rate. When the funding rate is positive, Longs pay Shorts. When the funding rate is negative, Shorts pay Longs. This mechanism can be quite nuanced and vary between exchanges, so it's best to study each contract specification to learn all the details (funding periods, mark price mechanisms etc.).

{% hint style="info" %}
See CSV [grouped symbols section](../../downloadable-csv-files#grouped-symbols) if you'd like to download data for all futures or perpetual swaps as a single file for given exchange instead one by one for each individual instrument.
{% endhint %}





## Do you provide time based aggregated data as well?

We are focusing on providing the best possible tick-level historical data for cryptocurrency exchanges and as of now our APIs (both [HTTP](../../api/http#data-feeds-exchange) and [CSV datasets](../../downloadable-csv-files#datasets-api-details)) do offer access to tick-level data only and do not offer support for time based aggregated data.

If you're interested in time based aggregated data (OHLC, interval based order book snapshots) see our [client libs](../api/getting-started) that provide such capabilities, but with the caveat that data aggregation is performed client-side from tick-level data sourced from the API, meaning it can be relatively slow process in contrast to ready to download aggregated data.





## Can you record market data for exchange that's not currently supported?

Yes, we're always open to support new promising exchanges. [Contact us](https://tardis.dev/#contact) and we'll get back to you to discuss the details.





## Do you provide market data in normalized format?

[Normalized market data](#what-is-a-difference-between-exchange-native-and-normalized-data-format) (unified data format for every exchange) is available via our [official libraries](../api/getting-started) and [downloadable CSV files](../downloadable-csv-files). Our [HTTP API](../../api/http#data-feeds-exchange) provides data only in [exchange-native format](../general#what-is-a-difference-between-exchange-native-and-normalized-data-formats).





## Do you provide normalized contract amounts for derivatives exchanges in your historical data feeds?

Data we provide has contract amounts exactly as provided by exchanges APIs, meaning in some cases it can be tricky to compare across exchanges due to different contract multipliers (like for example OKEx where each contract has $100 value) or different contract types (linear or inverse).\
\
We'll keep it this way, but we also provide [instrument metadata API ](../api/instruments-metadata-api)that returns contract multipliers, tick sizes and more for each instrument in uniform way, allowing easily normalize the contract amounts client-side without having to go through all kinds of documentation on various exchange to find this information.

{% hint style="info" %}
[See instrument metadata API docs](../api/instruments-metadata-api).
{% endhint %}





## What is a difference between exchange-native and normalized data format?&#x20;

Cryptocurrency markets are very fragmented and every exchange provides data in it's own bespoke data format which we call **exchange-native** data forma&#x74;**.** \
\
Our[ HTTP API](../../api/http#data-feeds-exchange) and [client libs](../api/getting-started) can provide market data in this format, meaning data you receive is exactly the same as the live data you would have received from exchanges ("as-is").&#x20;

{% hint style="info" %}
See [how we collect data](../../historical-data-details#market-data-collection-overview) in exchange-native format and [why it's important](#why-data-source-matters-and-why-we-use-real-time-websocket-feeds-as-data-source-vs-periodically-calling-rest-endpoints).
{% endhint %}

For example BitMEX trade message looks like this:

```javascript
{
  "table": "trade",
  "action": "insert",
  "data": [
    {
      "timestamp": "2019-06-01T00:03:11.589Z",
      "symbol": "ETHUSD",
      "side": "Sell",
      "size": 10,
      "price": 268.7,
      "tickDirection": "ZeroMinusTick",
      "trdMatchID": "ebc230d9-0b6e-2d5d-f99a-f90109a2b113",
      "grossValue": 268700,
      "homeNotional": 0.08555051758063137,
      "foreignNotional": 22.987424073915648
    }
  ]
}
```

and this is Deribit trade message:

```javascript
{
  "jsonrpc": "2.0",
  "method": "subscription",
  "params": {
    "channel": "trades.ETH-26JUN20.raw",
    "data": [
      {
        "trade_seq": 18052,
        "trade_id": "ETH-10813935",
        "timestamp": 1577836825724,
        "tick_direction": 0,
        "price": 132.65,
        "instrument_name": "ETH-26JUN20",
        "index_price": 128.6,
        "direction": "buy",
        "amount": 1.0
      }
    ]
  }
}

```



In contrast, **normalized data format** means the same, unified format across multiple exchanges. We provide normalized data via our [client libs](../api/getting-started) (data normalization is performed client-side) as well as via [downloadable CSV files](../downloadable-csv-files).

{% hint style="info" %}
In the process of data normalization we map the data we [collected from real-time WebSocket APIs](#why-data-source-matters-and-why-we-use-real-time-websocket-feeds-as-data-source-vs-periodically-calling-rest-endpoints) (exchange-native format) to normalized/unified format across exchanges that is easier to deal with (one data format across multiple exchanges).\
\
[We've open sourced all the data mappings](https://github.com/tardis-dev/tardis-node/tree/master/src/mappers) from exchange-native to normalized format to make the whole process as transparent as possible.
{% endhint %}

Sample **normalized** trade message:

```javascript
{
  "type": "trade",
  "symbol": "XBTUSD",
  "exchange": "bitmex",
  "id": "282a0445-0e3a-abeb-f403-11003204ea1b",
  "price": 7996,
  "amount": 50,
  "side": "sell",
  "timestamp": "2019-10-23T10:32:49.669Z",
  "localTimestamp": "2019-10-23T10:32:49.740Z"
}
```

We support following normalized data types via our [client libs](../api/getting-started):

* tick-by-tick trades
* order book L2 updates
* order book snapshots (tick-by-tick, 10ms, 100ms, 1s, 10s etc)
* quotes
* derivative tick info (open interest, funding rate, mark price, index price)
* liquidations
* OHLCV
* volume/tick based trade bars

and [downloadable CSV data files](../downloadable-csv-files):

* [tick-by-tick trades](../../downloadable-csv-files#trades)
* [incremental order book L2 updates](../../downloadable-csv-files#incremental_book_l2)
* [tick level order book snapshots](../../downloadable-csv-files#book_snapshot_25) (top 25 and top 5 levels)
* [options\_chain](../../downloadable-csv-files#options_chain)
* [quotes](../../downloadable-csv-files#quotes)
* [derivative tick info](../../downloadable-csv-files#derivative_ticker) (open interest, funding rate, mark price, index price)
* [liquidations](../../downloadable-csv-files#liquidations)





## What is the `channel` field used in the HTTP API and client libs `replay` functions?

Exchanges when publishing real-time data messages, always publish those for subscription topics clients have subscribed to. Those subscriptions topics are also very often called "channels" or "streams" in exchanges documentations pages and describe data type given message belongs to - for example [BitMEX](../historical-data-details/bitmex) publishes it's trades data via [trade channel](../../historical-data-details/bitmex#captured-real-time-channels) and order book L2 updates data via [orderBookL2](../../historical-data-details/bitmex#captured-real-time-channels).

Since we collect the data for all the channels described in exchanges' details page ([Captured real-time market data channels section](../../historical-data-details/bitmex#captured-real-time-channels)) our [HTTP API](../../api/http#data-feeds-exchange) and [client libs](../api/getting-started) offer filtering capability by those channels names, so for example to get historical trades for [BitMEX](../historical-data-details/bitmex), channel [trade](../../historical-data-details/bitmex#captured-real-time-channels) needs to be provided alongside requested instruments symbols (via HTTP API or client lib `replay` function args).





## What time zone is used in the data?

UTC, always.





## Is provided raw market data complete?

We're doing our best to provide the most complete and reliable historical raw data API on the market. To do so amongst [many other things](../../historical-data-details#market-data-collection-overview),  we utilize [highly available Kubernetes clusters](../general#what-is-your-infrastructure-setup) on Google Cloud Platform that offer best in the class availability, networking and monitoring. However due to exchanges' APIs downtimes (maintenance, deployments, [connection drops](#how-frequently-exchanges-drop-websocket-connections) etc.) we can experience data gaps and cannot guarantee 100% data completeness, but 99.9% (99.99% on most days) which should be more than enough for most of the use cases that tick level data is useful for.\
\
In rare circumstances, when exchange's API changes without any notice or we hit new unexpected rate limits we also may fail to record data during such period, it happens very rarely and is very specific for each exchange. Use [`/exchanges/:exchange`](https://docs.tardis.dev/api#exchanges-exchange) API endpoint and check for `incidentReports` field in order to get most detailed and up to date information on that subject.





## How frequently exchanges drop WebSocket connections?

As long as exchange WebSocket API is not 'hidden' behind Cloudflare proxy (causing relatively frequent "CloudFlare WebSocket proxy restarting, Connection reset by peer" errors) connections are stable for majority of supported exchanges and there are almost no connection drops during the day. In cases when there is more volatility in the market some exchanges tend to drop connections more frequently or have larger latency spikes. Overall it's a nuanced matter that changes over time, if you'd have any questions regarding particular exchange, please do not hesitate to [contact us](https://tardis.dev/#contact).





## Can historical order books reconstructed from L2 updates be crossed (bid/ask overlap) occasionally?

Although is should never happen in theory, in practice due to various crypto exchanges bugs and peculiarities it can happen (very occasionally), see some posts from users reporting those issues:

* [https://www.reddit.com/r/BitMEX/comments/8lbj9e/bidask\_ledger\_weirdness/](https://www.reddit.com/r/BitMEX/comments/8lbj9e/bidask_ledger_weirdness/)
* [https://www.reddit.com/r/KrakenSupport/comments/emu7xc/websocket\_bid\_sometimes\_not\_being\_deletedupdated/](https://www.reddit.com/r/KrakenSupport/comments/emu7xc/websocket_bid_sometimes_not_being_deletedupdated/)
* [https://www.reddit.com/r/KrakenSupport/comments/d1a4nx/websocket\_orderbook\_receiving\_wrong\_bid\_price\_for/](https://www.reddit.com/r/KrakenSupport/comments/d1a4nx/websocket_orderbook_receiving_wrong_bid_price_for/)
* [https://twitter.com/coinarb/status/931260529993170944](https://twitter.com/coinarb/status/931260529993170944)

We do track sequence numbers of WebSocket L2 order book messages when collecting the data and restart connection when sequence gap is detected for exchanges that do provide those numbers.\
We observe that even in scenario when sequence numbers are in check, bid/ask overlap can occur. \
\
When such scenario occurs, exchanges tend to 'forget' to publish  delete messages for the opposite side of the book when publishing new level for given side - we validated that hypothesis by comparing reconstructed order book snapshots that had crossed order book (bid/ask overlap) for which we removed order book levels for the opposite side manually (as exchange didn't publish that 'delete'), with quote/ticker feeds if best bid/ask matches (for exchanges that provide those) - see [sample code that implements that manual level removal logic](https://github.com/tardis-dev/tardis-node/blob/master/src/computable/booksnapshot.ts#L118).





## Can exchange publish data with non monotonically increasing timestamps for single data channel?

That shouldn't happen in theory, but we've detected that for some exchanges when new connection is established sometimes first message for given channel & symbol has newer timestamp than subsequent message, e.g., order book snapshot has newer timestamp than first order book update. This is why we provide data via [API](../api/getting-started) and [CSV downloads](../downloadable-csv-files) for given data ranges based on [local timestamps](#how-market-data-messages-are-being-timestamped) (timestamp of message arrival) which are always monotonically increasing.





## Are exchanges publishing duplicated trades data messages?

Some exchanges are occasionally publishing duplicated trades (trades with the same ids). Since we collect real-time data we also collect and provide duplicate trades via [API](../api/getting-started) if those were published by real-time WebSocket feeds of exchanges. Our [client libraries](../api/getting-started) have functionality that when working with [normalized data](#what-is-a-difference-between-exchange-native-and-normalized-data-format) can deduplicate such trades, similarly for [downloadable CSV files](../downloadable-csv-files) we deduplicate [tick-by-tick trades](../../downloadable-csv-files#trades) data.





## How order book data snapshots are provided?

Historical market data available via [HTTP API](../../api/http#data-feeds-exchange) provides order book snapshots at the beginning of each day (`00:00 UTC`) - [see details](../../historical-data-details#collected-order-book-data-details). \


We also provide custom order book snapshots with customizable time intervals from tick-by-tick, milliseconds to minutes or hours via [client libs](../api/getting-started) in which case custom snapshots are computed client side from raw data provided via HTTP API as well as via [downloadable CSV files](../downloadable-csv-files) - [book\_snapshot\_25](../../downloadable-csv-files#book_snapshot_25) and [book\_snapshot\_5](../../downloadable-csv-files#book_snapshot_5) .





## Do you collect order books as snapshots or in streaming mode?

Order books are collected in streaming mode - snapshot at the beginning of each day and then incremental updates. [See details](../../historical-data-details#collected-order-book-data-details).\


We also provide custom order book snapshots with customizable time intervals from tick-by-tick, milliseconds to minutes or hours via [client libs](../api/getting-started) in which case custom snapshots are computed client side from raw data provided via HTTP API as well as via [downloadable CSV files](../downloadable-csv-files) - [book\_snapshot\_25](../../downloadable-csv-files#book_snapshot_25) and [book\_snapshot\_5](../../downloadable-csv-files#book_snapshot_5) .





## How [`incremental_book_l2`](../../downloadable-csv-files#incremental_book_l2) CSV dataset is built from real-time data?

Cryptocurrency exchanges real-time APIs vary a lot, but for [L2 order book data ](#what-l2-order-book-data-can-be-used-for)they all tend to follow similar flow, first when WS connection is established and subscription is confirmed, exchanges send initial order book snapshot (all existing price levels or top 'x' levels depending on exchange) and then start streaming 'book update' messages (called frequently deltas as well). Those updates when applied to initial snapshot, result in up to data order book state at given time.

{% hint style="info" %}
We do provide initial L2 snapshots in `incremental_book_L2` dataset at the beginning of each day (00:00 UTC, [more details](../../historical-data-details#collected-order-book-data-details)), but also anytime exchange closes it's real-time WebSocket connection, [see details](#how-frequently-exchanges-drop-websocket-connections).
{% endhint %}

\
Let's take FTX as an example and start with it's snapshot orderbook message (that is frequently called 'partial' in exchanges API docs as well).\
\
Remaining bids and asks levels were removed from this sample message for the sake of clarity.

```javascript
{
  "channel": "orderbook",
  "market": "ETH/USD",
  "type": "partial",
  "data": {
    "time": 1601510401.2166328,
    "checksum": 204980439,
    "bids": [
      [
        359.72,
        121.259
      ]
    ],
    "asks": [
      [
        359.8,
        8.101
      ]
    ],
    "action": "partial"
  }
}
```

Such snapshot message maps to the following rows in CSV file:

| exchange | symbol  | timestamp        | local\_timestamp | is\_snapshot | side | price  | amount  |
| -------- | ------- | ---------------- | ---------------- | ------------ | ---- | ------ | ------- |
| ftx      | ETH/USD | 1601510401216632 | 1601510401316432 | true         | ask  | 359.8  | 8.101   |
| ftx      | ETH/USD | 1601510401216632 | 1601510401316432 | true         | bid  | 359.72 | 121.259 |



... and here's a sample FTX orderbook update message.

```javascript
{
  "channel": "orderbook",
  "market": "ETH/USD",
  "type": "update",
  "data": {
    "time": 1601510427.1840546,
    "checksum": 1377242400,
    "bids": [],
    "asks": [
      [
        360.24,
        4.962
      ],
      [
        361.02,
        0
      ]
    ],
    "action": "update"
  }
}
```

Let's see how it maps to CSV format.

| exchange | symbol  | timestamp        | local\_timestamp | is\_snapshot | side | price  | amount |
| -------- | ------- | ---------------- | ---------------- | ------------ | ---- | ------ | ------ |
| ftx      | ETH/USD | 1601510427184054 | 1601510427204046 | false        | ask  | 360.24 | 4.962  |
| ftx      | ETH/USD | 1601510427184054 | 1601510427204036 | false        | ask  | 361.02 | 0      |

See [this answer](#how-can-i-reconstruct-full-order-book-state-from-incremental_book_l2-csv-dataset) if you have doubts how to reconstruct order book state based on data provided in `incremental_book_L2` dataset.





## How can I reconstruct full order book state from  [`incremental_book_L2`](../../downloadable-csv-files#incremental_book_l2) CSV dataset?

{% hint style="info" %}
See also [how incremental\_book\_l2 CSV dataset is built from real-time data](#how-incremental_book_l2-csv-dataset-is-built-from-real-time-data).
{% endhint %}

In order to reconstruct full order book state correctly from [`incremental_book_L2`](../../downloadable-csv-files#incremental_book_l2) data:

*   For each row in the CSV file (iterate in the same order as provided in file):\


    * only if local timestamp of current row is larger than previous row local timestamp(`local_timestamp` column value) it means you can read your local order book state as it's consistent, why? CSV format is flat where each row represents single price level update, but most exchanges real-time feeds publish multiple order book levels updates via single WebSocket message that need to be processed together before reading locally maintained order book state. We use local timestamp value here to detect all price level updates belonging to single 'update' message.



    * if current row is a part of the snapshot (`is_snapshot` column value set to `true`) and previous one was not, reset your local order book state object that tracks price levels for each order book side as it means that there was a connection restart and exchange provided full order book snapshot or it was a start of a new day (each incremental\_book\_L2 file starts with the snapshot)\

    * if current row amount **is** set to zero (`amount` column value set to `0`) remove such price level (row's `price` column) from your local order book state as such price level does not exist anymore\

    * if current row amount **is not** set to zero update your local order book state price level with new value or add new price level if not exist yet in your local order book state - maintain separately bids and asks order book sides (`side` column value)



Alternatively we do also provide [top 25 and top 5 levels](../../downloadable-csv-files#book_snapshot_25) order book snapshots CSV datasets ready to download.





## How CSV datasets are split into the files?

CSV datasets are available in daily intervals split by exchange, data type and symbol. In addition to standard currency pairs/instrument symbols, each exchange also has special ['grouped' symbols available ](../../downloadable-csv-files#grouped-symbols)depending if it supports given market type: SPOT, FUTURES, OPTIONS and PERPETUALS. That feature is useful if someone is interested in for examples all Deribit's options instruments' trades or quotes data without a need to request data for each symbol separately one by one.





## How market data messages are being timestamped?

Each message received via WebSocket connection is timestamped with 100ns precision using [synchronized clock](https://developers.google.com/time) at arrival time (before any message processing) and stored in ISO 8601 format.





## What is the new historical market data delay in relation to real-time?

For [API access](../api/getting-started) it's 15 minutes (T - 15min), [downloadable CSV files](../downloadable-csv-files) for given day are available on the next day around 06:00 UTC.





