Downloadable CSV files

Historical datasets in daily intervals split by exchange, data type and symbol available via Datasets API

Quick start

Downloadable CSV files are available via dedicated datasets API that provides order book L2 delta updates, tick-by-tick trades, quotes and derivative tickers datasets in daily intervals split by exchange, data type and symbol.

Both Node.js and Python clients have built-in functions to efficiently download whole date range of data.

Historical datasets for the first day of each month are available to download without API key.

Python
Node.js
cURL
Python
# pip install tardis-dev
# requires Python >=3.6
from tardis_dev import datasets
datasets.download(
exchange="deribit",
data_types=[
"incremental_book_L2",
"trades",
"quotes",
"derivative_ticker"
],
from_date="2019-11-01",
to_date="2019-11-02",
symbols=["BTC-PERPETUAL", "ETH-PERPETUAL"],
api_key="YOUR API KEY (optionally)",
)

See full example that shows all available download options (download path customization, filenames conventions and more).

Node.js
// npm install [email protected]
// requires node version >=12
const { downloadDatasets } = require('tardis-dev')
;(async () => {
await downloadDatasets({
exchange: 'deribit',
dataTypes: [
'incremental_book_L2',
'trades',
'quotes',
'derivative_ticker'
],
from: '2019-11-01',
to: '2019-11-02',
symbols: ['BTC-PERPETUAL', 'ETH-PERPETUAL'],
apiKey: 'YOUR API KEY (optionally)'
})
})()

See full example that shows all available download options (download path customization, filenames conventions and more).

cURL
curl -o deribit_trades_2019-11-01_BTC-PERPETUAL.csv.gz https://datasets.tardis.dev/v1/deribit/trades/2019/11/01/BTC-PERPETUAL.csv.gz

See datasets API reference.

CSV format details

  • columns delimiter: , (comma)

  • new line marker: \n (LF)

  • decimal mark: . (dot)

  • date time format: microseconds since epoch (https://www.epochconverter.com/)

  • date time timezone: UTC

Data types

incremental_book_L2

If you'd like to access L3 data (market by order) that some exchanges provide, contact us.

Order book L2 incremental delta updates collected from exchanges' real-time WebSocket order book L2 data feeds - data as deep and granular as underlying real-time data source. Each CSV file includes an initial snapshot (rows with is_snapshot=true) and then incremental updates to the order book on each row. Additionally, snapshot are also provided for events when connection that was recording the historical data got disconnected.

As exchanges real-time feeds usually publish multiple order book levels updates via single message you can recognize that by grouping rows by timestamp field if needed.

CSV incremental_book_L2 schema
dataset preview
CSV incremental_book_L2 schema

column name

description

exchange

exchange id, one of https://api.tardis.dev/v1/exchanges ([].id field)

symbol

instrument symbol as provided by exchange (always uppercase)

timestamp

timestamp provided by exchange in microseconds since epoch - if exchange does not provide one local_timestamp value is used as a fallback

local_timestamp

message arrival timestamp in microseconds since epoch

is_snapshot

possible values:

  • true - if update was a part of initial order book snapshot

  • false - if update was not a part of initial order book snapshot

If last update was not a snapshot and current one is, then existing order book state must be discarded (all existing levels removed)

side

determines to which side of the order book update belongs to:

  • bid - bid side of the book, buy orders

  • ask - ask side of the book, sell orders

price

price identifying book level being updated

amount

updated price level amount as provided by exchange, not a delta - an amount of 0 indicates that the price level can be removed

dataset preview

exchange

symbol

timestamp

local_timestamp

is_snapshot

side

price

amount

deribit

BTC-PERPETUAL

1585699209920000

1585699209934201

false

ask

6443.5

38640

deribit

BTC-PERPETUAL

1585699209947000

1585699209957629

false

bid

6311.5

0

deribit

BTC-PERPETUAL

1585699209950000

1585699209963464

false

ask

6428

13210

deribit

BTC-PERPETUAL

1585699209967000

1585699209979152

false

bid

6311.5

750

deribit

BTC-PERPETUAL

1585699209970000

1585699209983585

false

bid

6327

16010

deribit

BTC-PERPETUAL

1585699209970000

1585699209983585

false

bid

6325

210530

deribit

BTC-PERPETUAL

1585699209972000

1585699209983691

false

bid

6351

810

deribit

BTC-PERPETUAL

1585699209972000

1585699209983691

false

bid

6352.5

18830

deribit

BTC-PERPETUAL

1585699209974000

1585699209983703

false

ask

6492

100

● trades

Tick-by-tick trades data collected from exchanges' real-time WebSocket trades data feeds.

CSV trades schema
dataset preview
CSV trades schema

column name

description

exchange

exchange id, one of https://api.tardis.dev/v1/exchanges ([].id field)

symbol

instrument symbol as provided by exchange (always uppercase)

timestamp

timestamp provided by exchange in microseconds since epoch - if exchange does not provide one local_timestamp value is used as a fallback

local_timestamp

message arrival timestamp in microseconds since epoch

id

trade id as provided by exchange, empty if exchange does not provide one - different exchanges provide id's as numeric values, GUID's or other strings, and some do not provide that information at all

side

liquidity taker side (aggressor), possible values:

  • buy - liquidity taker was buying

  • sell - liquidity taker was selling

  • unknown - exchange did not provide that information

price

trade price as provided by exchange

amount

trade amount as provided by exchange

dataset preview

exchange

symbol

timestamp

local_timestamp

id

side

price

amount

bitmex

XBTUSD

1585699202957000

1585699203089980

d20...

buy

6425.5

12

bitmex

XBTUSD

1585699202980000

1585699203095276

619...

sell

6425

150

bitmex

XBTUSD

1585699203002000

1585699203099299

751...

sell

6425

25

bitmex

XBTUSD

1585699203092000

1585699203122233

3c1...

buy

6425.5

1

bitmex

XBTUSD

1585699203092000

1585699203122233

b9b...

buy

6425.5

1

bitmex

XBTUSD

1585699203092000

1585699203122233

433...

buy

6425.5

1

bitmex

XBTUSD

1585699203092000

1585699203122233

d16...

buy

6425.5

1

bitmex

XBTUSD

1585699203092000

1585699203122233

402...

buy

6425.5

1

bitmex

XBTUSD

1585699203092000

1585699203122233

2f8...

buy

6425.5

1

● quotes

Top of the book (best bid/ask) data sourced from reconstructed L2 order book state - best bid/ask recorded every time top of the book has changed. We on purpose choose this solution over native exchanges real-time quotes feeds as those vary a lot between exchanges, can be throttled, some are absent at all, often are delayed and published in batches in comparison to more granular L2 updates which are the basis for our quotes dataset.

CSV quotes schema
dataset preview
CSV quotes schema

column name

description

exchange

exchange id, one of https://api.tardis.dev/v1/exchanges ([].id field)

symbol

instrument symbol as provided by exchange (always uppercase)

timestamp

timestamp provided by exchange in microseconds since epoch - if exchange does not provide one local_timestamp value is used as a fallback

local_timestamp

message arrival timestamp in microseconds since epoch

ask_amount

best ask amount as provided by exchange

ask_price

best ask price as provided by exchange

bid_price

best bid price as provided by exchange

bid_amount

best bid amount as provided by exchange

dataset preview

exchange

symbol

timestamp

local_timestamp

ask_amount

ask_price

bid_price

bid_amount

huobi-dm-swap

BTC-USD

1585699201147000

1585699201270777

86

6423

6422.9

112

huobi-dm-swap

BTC-USD

1585699201175000

1585699201292111

86

6423

6422.9

114

huobi-dm-swap

BTC-USD

1585699201257000

1585699201373479

84

6423

6422.9

219

huobi-dm-swap

BTC-USD

1585699201279000

1585699201495667

64

6423

6422.9

219

huobi-dm-swap

BTC-USD

1585699201295000

1585699201495715

64

6423

6422.9

229

huobi-dm-swap

BTC-USD

1585699201447000

1585699201564788

2

6423

6422.9

229

huobi-dm-swap

BTC-USD

1585699201556000

1585699201677770

64

6423

6422.9

229

huobi-dm-swap

BTC-USD

1585699201668000

1585699201784213

64

6423

6422.9

235

huobi-dm-swap

BTC-USD

1585699201747000

1585699201865051

2

6423

6422.9

235

● derivative_ticker

Derivative instrument ticker info (open interest, funding, mark price, index price) collected from exchanges' real-time WebSocket instruments & tickers data feeds. Anytime any of the tracked values has changed data was added to final dataset.

CSV derivative_ticker schema
dataset preview
CSV derivative_ticker schema

column name

description

exchange

exchange id, one of https://api.tardis.dev/v1/exchanges ([].id field)

symbol

instrument symbol as provided by exchange (always uppercase)

timestamp

timestamp provided by exchange in microseconds since epoch - if exchange does not provide one local_timestamp value is used as a fallback

local_timestamp

message arrival timestamp in microseconds since epoch

funding_timestamp

timestamp of the next funding event in microseconds since epoch, empty if exchange does not provide one

funding_rate

current funding rate, empty if exchange does not provide one

predicted_funding_rate

estimated next funding rate, empty if exchange does not provide one

open_interest

current open interest, empty if exchange does not provide one

last_price

last instrument price, empty if exchange does not provide one

index_price

index price of the instrument, empty if exchange does not provide one

mark_price

mark price of the instrument, empty if exchange does not provide one

dataset preview

1

exchange

symbol

timestamp

local_timestamp

funding_timestamp

funding_rate

predicted_funding_rate

open_interest

last_price

index_price

mark_price

2

bitmex

ETHUSD

1585699199651000

1585699202577291

1585713600000000

0.0001

0.001654

45921455

133.25

133.14

133.15

3

bitmex

ETHUSD

1585699200000000

1585699204834359

1585713600000000

0.0001

0.001654

45921455

133.25

133.12

133.13

4

bitmex

ETHUSD

1585699202925000

1585699205076090

1585713600000000

0.0001

0.001654

45921455

133.3

133.12

133.13

5

bitmex

ETHUSD

1585699202925000

1585699205090339

1585713600000000

0.0001

0.001654

45883853

133.3

133.12

133.13

6

bitmex

ETHUSD

1585699203465000

1585699205274555

1585713600000000

0.0001

0.001654

45883853

133.25

133.12

133.13

7

bitmex

ETHUSD

1585699204439000

1585699205951209

1585713600000000

0.0001

0.001654

45883853

133.15

133.12

133.13

8

bitmex

ETHUSD

1585699205000000

1585699206389317

1585713600000000

0.0001

0.001654

45883853

133.15

133.09

133.1

9

bitmex

ETHUSD

1585699207279000

1585699207490211

1585713600000000

0.0001

0.001654

45883853

133.2

133.09

133.1

10

bitmex

ETHUSD

1585699207279000

1585699208084951

1585713600000000

0.0001

0.001654

45867677

133.2

133.09

133.1

Datasets API details

Download via client libraries

Historical datasets for the first day of each month are available to download without API key.

Python
Node.js
Python
# pip install tardis-dev
# requires Python >=3.6
from tardis_dev import datasets, get_exchange_details
import logging
# comment out to disable debug logs
logging.basicConfig(level=logging.DEBUG)
# function used by default if not provided via options
def default_file_name(exchange, data_type, date, symbol, format):
return f"{exchange}_{data_type}_{date.strftime('%Y-%m-%d')}_{symbol}.{format}.gz"
# customized get filename function - saves data in nested directory structure
def file_name_nested(exchange, data_type, date, symbol, format):
return f"{exchange}/{data_type}/{date.strftime('%Y-%m-%d')}_{symbol}.{format}.gz"
# returns data available at https://api.tardis.dev/v1/exchanges/deribit
deribit_details = get_exchange_details("deribit")
# print(deribit_details)
datasets.download(
# one of https://api.tardis.dev/v1/exchanges with supportsDatasets:true - use 'id' value
exchange="deribit",
# accepted data types - 'datasets.dataTypes' field in https://api.tardis.dev/v1/exchanges/deribit,
# or get those values from 'deribit_details["datasets"]["dataTypes"] dict above
data_types=["incremental_book_L2", "trades", "quotes", "derivative_ticker"],
# change date ranges as needed to fetch full month or year for example
from_date="2019-11-01",
# to date is non inclusive
to_date="2019-11-02",
# accepted values: 'datasets.symbols[].id' field in https://api.tardis.dev/v1/exchanges/deribit
symbols=["BTC-PERPETUAL", "ETH-PERPETUAL",],
# (optional) your API key to get access to non sample data as well
# api_key="YOUR API KEY",
# (optional) path where data will be downloaded into, default dir is './datasets'
# download_dir="./datasets",
# (optional) - one can customize downloaded file name/path (flat dir strucure, or nested etc) - by default function 'default_file_name' is used
# get_filename=default_file_name,
# (optional) file_name_nested will download data to nested directory structure (split by exchange and data type)
# get_filename=file_name_nested,
)
Node.js
// npm install [email protected]
// requires node version >=12
// remove it to disable debug logs
process.env.DEBUG = 'tardis-dev*'
const { downloadDatasets, getExchangeDetails } = require('tardis-dev')
;(async () => {
// returns data available at https://api.tardis.dev/v1/exchanges/deribit
const deribitDetails = await getExchangeDetails('deribit')
// console.log(deribitDetails.datasets)
await downloadDatasets({
exchange: 'deribit', // one of https://api.tardis.dev/v1/exchanges with supportsDatasets:true - use 'id' value
dataTypes: ['incremental_book_L2', 'trades', 'quotes', 'derivative_ticker'], // accepted data types - 'datasets.dataTypes' field in https://api.tardis.dev/v1/exchanges/deribit, or get those values from 'deribitDetails.datasets.dataTypes' object above
from: '2019-11-01', // change date ranges as needed to fetch full month or year for example
to: '2019-11-02', // to date is non inclusive
symbols: ['BTC-PERPETUAL', 'ETH-PERPETUAL'] // accepted values: 'datasets.symbols[].id' field in https://api.tardis.dev/v1/exchanges/deribit, or `deribitDetails.datasets.symbols[].id` from object above
// apiKey: 'YOUR_API_KEY', // (optional) your API key to get access to non sample data as well
// downloadDir:'./datasets', // (optional) path where data will be downloaded into, default dir is './datasets'
// getFilename: getFilenameDefault, // (optional) - one can customize downloaded file name/path (flat dir strucure, or nested etc) - by default function 'getFilenameDefault' is used
// getFilename: getFilenameCustom // (optional) getFilenameCustom will download data to nested directory structure (split by exchange and data type)
})
})().catch((e) => {
console.log('download error', e)
})
// function used by default if not provided via options
function getFilenameDefault({ exchange, dataType, format, date, symbol }) {
return `${exchange}_${dataType}_${date.toISOString().split('T')[0]}_${symbol}.${format}.gz`
}
// customized get filename function - saves data in nested directory structure
function getFilenameCustom({ exchange, dataType, format, date, symbol }) {
return `${exchange}/${dataType}/${date.toISOString().split('T')[0]}_${symbol}.${format}.gz`
}

get
Datasets API reference

https://datasets.tardis.dev/v1/:exchange/:dataType/:year/:month/:day/:symbol.csv.gz
Returns gzip compressed CSV dataset for given exchange, data type, date (year, month, day) and symbol.
Request
Response
Request
Path Parameters
exchange
required
string
one of https://api.tardis.dev/v1/exchanges (field id, only exchanges with "supportsDatasets":true)
dataType
required
string
one of datasets.dataTypes values from https://api.tardis.dev/v1/exchanges/:exchange API response
year
required
string
year in format YYYY (four-digit year)
month
required
string
month in format MM (two-digit month of the year)
day
optional
string
day in format DD (two-digit day of the month)
symbol
required
string
one of datasets.symbols[].id values from https://api.tardis.dev/v1/exchanges/:exchange API response, see details below
Headers
Authorization
optional
string
For authenticated requests provide Authorization header with value: 'Bearer YOUR_API_KEY'. Without API key historical datsets for the first day of each month are available to download.
Response
200: OK
gzip compressed CSV dataset

Sample requests