Quick start

Downloadable CSV files are available via dedicated datasets API that provides order book L2 delta updates, tick-by-tick trades, quotes and derivative tickers datasets in daily intervals split by exchange, data type and symbol.

Both Node.js and Python clients have built-in functions to efficiently download whole date range of data.

# pip install tardis-dev
# requires Python >=3.6
from tardis_dev import datasets
​
datasets.download(
    exchange="deribit",
    data_types=[
        "incremental_book_L2",
        "trades",
        "quotes",
        "derivative_ticker"
    ],
    from_date="2019-11-01",
    to_date="2019-11-02",
    symbols=["BTC-PERPETUAL", "ETH-PERPETUAL"],
    api_key="YOUR API KEY (optionally)",
)

// npm install [email protected]
// requires node version >=12
const { downloadDatasets } = require('tardis-dev')
​
;(async () => {
  await downloadDatasets({
    exchange: 'deribit',
    dataTypes: [
      'incremental_book_L2',
      'trades',
      'quotes',
      'derivative_ticker'
    ],
    from: '2019-11-01',
    to: '2019-11-02',
    symbols: ['BTC-PERPETUAL', 'ETH-PERPETUAL'],
    apiKey: 'YOUR API KEY (optionally)'
  })
})()

curl -o deribit_trades_2019-11-01_BTC-PERPETUAL.csv.gz https://datasets.tardis.dev/v1/deribit/trades/2019/11/01/BTC-PERPETUAL.csv.gz
​

​

​

CSV format details

• columns delimiter: , (comma)

• new line marker: \n (LF)

• decimal mark: . (dot)

• date time format: microseconds since epoch (https://www.epochconverter.com/)

• date time timezone: UTC

​

​

Data types

● incremental_book_L2

Order book L2 incremental delta updates collected from exchanges' real-time WebSocket order book L2 data feeds - data as deep and granular as underlying real-time data source. Each CSV file includes an initial snapshot (rows with is_snapshot=true) and then incremental updates to the order book on each row. Additionally, snapshot are also provided for events when connection that was recording the historical data got disconnected.

As exchanges real-time feeds usually publish multiple order book levels updates via single message you can recognize that by grouping rows by timestamp field if needed.

​

● trades

Tick-by-tick trades data collected from exchanges' real-time WebSocket trades data feeds.

​

● quotes

Top of the book (best bid/ask) data sourced from reconstructed L2 order book state - best bid/ask recorded every time top of the book has changed. We on purpose choose this solution over native exchanges real-time quotes feeds as those vary a lot between exchanges, can be throttled, some are absent at all, often are delayed and published in batches in comparison to more granular L2 updates which are the basis for our quotes dataset.

​

● derivative_ticker

Derivative instrument ticker info (open interest, funding, mark price, index price) collected from exchanges' real-time WebSocket instruments & tickers data feeds. Anytime any of the tracked values has changed data was added to final dataset.

​

​

Datasets API details

• all downloadable datasets are gzip compressed

• historical market data is available in daily intervals split by exchange, data type and symbol

  ​

• data for a given day is available on the next day around 3h after 00:00 UTC - exact date until when data is available can be requested via /exchanges/:exchange API call (datasets.exportedUntil), e.g., https://api.tardis.dev/v1/exchanges/ftx

• datasets are ordered and split into separate daily files by local_timestamp (timestamp of message arrival time)

• empty gzip compressed file is being returned in case of no data available for a given day, symbol and data type, e.g., exchange downtime, very low volume currency pairs etc.

• iftimestamp equals to local_timestamp it means that exchange didn't provide timestamp for message, e.g., BitMEX order book updates

• cell in CSV file is empty if there's no value for it, e.g., no trade id if a given exchange doesn't provide one

• in addition to standard currency pairs/instrument symbols, each exchange also has special 'grouped' symbols available depending if it supports given market type: SPOT, FUTURES, OPTIONS and PERPETUALS. That feature is useful if someone is interested in for examples all Deribit's options instruments' trades data without a need to specify each symbol separately one by one.

  For example Deribit supports futures, options and perpetuals markets so following requests are allowed:

  • ​https://datasets.tardis.dev/v1/deribit/trades/2019/11/01/FUTURES.csv.gz​

  • ​https://datasets.tardis.dev/v1/deribit/trades/2019/11/01/PERPETUALS.csv.gz​

  • ​https://datasets.tardis.dev/v1/deribit/trades/2019/11/01/OPTIONS.csv.gz

  those special symbols are also listed in response to /exchanges/:exchange API call

• datasets are sourced from Tardis.dev HTTP API, which in turn provides the the data sourced from exchanges real-time WebSocket market data feeds (in contrast to REST API endpoints)

• See "Data FAQ" regarding potential order book overlaps issues, non monotonically increasing exchanges timestamps, duplicated trade data and more

​

​

Download via client libraries

# pip install tardis-dev
# requires Python >=3.6
from tardis_dev import datasets, get_exchange_details
import logging
​
# comment out to disable debug logs
logging.basicConfig(level=logging.DEBUG)
​
# function used by default if not provided via options
def default_file_name(exchange, data_type, date, symbol, format):
    return f"{exchange}_{data_type}_{date.strftime('%Y-%m-%d')}_{symbol}.{format}.gz"
​
​
# customized get filename function - saves data in nested directory structure
def file_name_nested(exchange, data_type, date, symbol, format):
    return f"{exchange}/{data_type}/{date.strftime('%Y-%m-%d')}_{symbol}.{format}.gz"
​
​
# returns data available at https://api.tardis.dev/v1/exchanges/deribit
deribit_details = get_exchange_details("deribit")
# print(deribit_details)
​
datasets.download(
    # one of https://api.tardis.dev/v1/exchanges with supportsDatasets:true - use 'id' value
    exchange="deribit",
    # accepted data types - 'datasets.dataTypes' field in https://api.tardis.dev/v1/exchanges/deribit,
    # or get those values from 'deribit_details["datasets"]["dataTypes"] dict above
    data_types=["incremental_book_L2", "trades", "quotes", "derivative_ticker"],
    # change date ranges as needed to fetch full month or year for example
    from_date="2019-11-01",
    # to date is non inclusive
    to_date="2019-11-02",
    # accepted values: 'datasets.symbols[].id' field in https://api.tardis.dev/v1/exchanges/deribit
    symbols=["BTC-PERPETUAL", "ETH-PERPETUAL",],
    # (optional) your API key to get access to non sample data as well
    # api_key="YOUR API KEY",
    # (optional) path where data will be downloaded into, default dir is './datasets'
    # download_dir="./datasets",
    # (optional) - one can customize downloaded file name/path (flat dir strucure, or nested etc) - by default function 'default_file_name' is used
    # get_filename=default_file_name,
    # (optional) file_name_nested will download data to nested directory structure (split by exchange and data type)
    # get_filename=file_name_nested,
)

// npm install [email protected]
// requires node version >=12
​
// remove it to disable debug logs
process.env.DEBUG = 'tardis-dev*'
​
const { downloadDatasets, getExchangeDetails } = require('tardis-dev')
​
;(async () => {
  // returns data available at https://api.tardis.dev/v1/exchanges/deribit
  const deribitDetails = await getExchangeDetails('deribit')
​
  // console.log(deribitDetails.datasets)
​
  await downloadDatasets({
    exchange: 'deribit', // one of https://api.tardis.dev/v1/exchanges with supportsDatasets:true - use 'id' value
    dataTypes: ['incremental_book_L2', 'trades', 'quotes', 'derivative_ticker'], // accepted data types - 'datasets.dataTypes' field in https://api.tardis.dev/v1/exchanges/deribit, or get those values from 'deribitDetails.datasets.dataTypes' object above
    from: '2019-11-01', // change date ranges as needed to fetch full month or year for example
    to: '2019-11-02', // to date is non inclusive
    symbols: ['BTC-PERPETUAL', 'ETH-PERPETUAL'] // accepted values: 'datasets.symbols[].id' field in https://api.tardis.dev/v1/exchanges/deribit, or `deribitDetails.datasets.symbols[].id` from object above
​
    // apiKey: 'YOUR_API_KEY', // (optional) your API key to get access to non sample data as well
    // downloadDir:'./datasets', // (optional) path where data will be downloaded into, default dir is './datasets'
​
    // getFilename: getFilenameDefault, // (optional) - one can customize downloaded file name/path (flat dir strucure, or nested etc) - by default function 'getFilenameDefault' is used
    // getFilename: getFilenameCustom // (optional) getFilenameCustom will download data to nested directory structure (split by exchange and data type)
  })
})().catch((e) => {
  console.log('download error', e)
})
​
// function used by default if not provided via options
function getFilenameDefault({ exchange, dataType, format, date, symbol }) {
  return `${exchange}_${dataType}_${date.toISOString().split('T')[0]}_${symbol}.${format}.gz`
}
​
// customized get filename function - saves data in nested directory structure
function getFilenameCustom({ exchange, dataType, format, date, symbol }) {
  return `${exchange}/${dataType}/${date.toISOString().split('T')[0]}_${symbol}.${format}.gz`
}

​

​

​

​

• symbols param provided to datasets API in comparison to HTTP API needs to be both always uppercase and have '/' and ':' characters replaced with '-' so symbol is url safe.

• list of allowed symbols for each exchange can be requested via /exchanges/:exchange API call, e.g., https://api.tardis.dev/v1/exchanges/deribit - datasets.symbols[].id field

• in addition to standard currency pairs/instrument symbols, each exchange also has special 'grouped' symbols available depending if it supports given market type: SPOT, FUTURES, OPTIONS and PERPETUALS. That feature is useful if someone is interested in for examples all Deribit's options instruments' trades data without a need to specify each symbol separately one by one.

  For example Deribit supports futures, options and perpetuals markets so following requests are allowed:

  • ​https://datasets.tardis.dev/v1/deribit/trades/2019/11/01/FUTURES.csv.gz​

  • ​https://datasets.tardis.dev/v1/deribit/trades/2019/11/01/PERPETUALS.csv.gz​

  • ​https://datasets.tardis.dev/v1/deribit/trades/2019/11/01/OPTIONS.csv.gz

  those special symbols are also listed in response to /exchanges/:exchange API call

  ​

Sample requests

​

​