Downloadable CSV files

Historical datasets in daily intervals split by exchange, data type and symbol available via Datasets API

Quick start

CSV datasets are available via dedicated datasets API that allows downloading incremental order book L2 updates, tick-by-tick trades, options chains, quotes and derivative tickers data in daily intervals split by exchange, data type and symbol.

Both Node.js and Python clients have built-in functions to efficiently download whole date range of data.

Historical datasets for the first day of each month are available to download without API key.

Python
Node.js
cURL
Python
# pip install tardis-dev
# requires Python >=3.6
from tardis_dev import datasets
datasets.download(
exchange="deribit",
data_types=[
"incremental_book_L2",
"trades",
"quotes",
"derivative_ticker"
],
from_date="2019-11-01",
to_date="2019-11-02",
symbols=["BTC-PERPETUAL", "ETH-PERPETUAL"],
api_key="YOUR API KEY (optionally)",
)

See full example that shows all available download options (download path customization, filenames conventions and more).

Node.js
// npm install [email protected]
// requires node version >=12
const { downloadDatasets } = require('tardis-dev')
;(async () => {
await downloadDatasets({
exchange: 'deribit',
dataTypes: [
'incremental_book_L2',
'trades',
'quotes',
'derivative_ticker'
],
from: '2019-11-01',
to: '2019-11-02',
symbols: ['BTC-PERPETUAL', 'ETH-PERPETUAL'],
apiKey: 'YOUR API KEY (optionally)'
})
})()

See full example that shows all available download options (download path customization, filenames conventions and more).

cURL
curl -o deribit_trades_2019-11-01_BTC-PERPETUAL.csv.gz https://datasets.tardis.dev/v1/deribit/trades/2019/11/01/BTC-PERPETUAL.csv.gz

See datasets API reference.

CSV format details

  • columns delimiter: , (comma)

  • new line marker: \n (LF)

  • decimal mark: . (dot)

  • date time format: microseconds since epoch (https://www.epochconverter.com/)

  • date time timezone: UTC

Data types

incremental_book_L2

If you'd like to access L3 data (market by order) provided by few exchanges, contact us.

Incremental order book L2 updates collected from exchanges' real-time WebSocket order book L2 data feeds - data as deep and granular as underlying real-time data source. Each CSV file includes an initial snapshot (rows with is_snapshot=true) and then incremental updates to the order book on each row. Additionally, snapshot are also provided for events when connection that was recording the historical data got disconnected.

As exchanges real-time feeds usually publish multiple order book levels updates via single message you can recognize that by grouping rows by timestamp field if needed.

CSV incremental_book_L2 schema
dataset preview
CSV incremental_book_L2 schema

column name

description

exchange

exchange id, one of https://api.tardis.dev/v1/exchanges ([].id field)

symbol

instrument symbol as provided by exchange (always uppercase)

timestamp

timestamp provided by exchange in microseconds since epoch - if exchange does not provide one local_timestamp value is used as a fallback

local_timestamp

message arrival timestamp in microseconds since epoch

is_snapshot

possible values:

  • true - if update was a part of initial order book snapshot

  • false - if update was not a part of initial order book snapshot

If last update was not a snapshot and current one is, then existing order book state must be discarded (all existing levels removed)

side

determines to which side of the order book update belongs to:

  • bid - bid side of the book, buy orders

  • ask - ask side of the book, sell orders

price

price identifying book level being updated

amount

updated price level amount as provided by exchange, not a delta - an amount of 0 indicates that the price level can be removed

dataset preview

exchange

symbol

timestamp

local_timestamp

is_snapshot

side

price

amount

deribit

BTC-PERPETUAL

1585699209920000

1585699209934201

false

ask

6443.5

38640

deribit

BTC-PERPETUAL

1585699209947000

1585699209957629

false

bid

6311.5

0

deribit

BTC-PERPETUAL

1585699209950000

1585699209963464

false

ask

6428

13210

deribit

BTC-PERPETUAL

1585699209967000

1585699209979152

false

bid

6311.5

750

deribit

BTC-PERPETUAL

1585699209970000

1585699209983585

false

bid

6327

16010

deribit

BTC-PERPETUAL

1585699209970000

1585699209983585

false

bid

6325

210530

deribit

BTC-PERPETUAL

1585699209972000

1585699209983691

false

bid

6351

810

deribit

BTC-PERPETUAL

1585699209972000

1585699209983691

false

bid

6352.5

18830

deribit

BTC-PERPETUAL

1585699209974000

1585699209983703

false

ask

6492

100

● book_snapshot_25

We're in the process of rolling out this data type, it's currently available only for BitMEX, Deribit and Bitfinex Derivatives exchanges.

● book_snapshot_5

We're in the process of rolling out this data type, it's currently available only for BitMEX, Deribit and Bitfinex Derivatives exchanges.

● trades

Tick-by-tick trades data collected from exchanges' real-time WebSocket trades data feeds - in contrary to pooling periodically the REST API, see the trade-offs.

CSV trades schema
dataset preview
CSV trades schema

column name

description

exchange

exchange id, one of https://api.tardis.dev/v1/exchanges ([].id field)

symbol

instrument symbol as provided by exchange (always uppercase)

timestamp

timestamp provided by exchange in microseconds since epoch - if exchange does not provide one local_timestamp value is used as a fallback

local_timestamp

message arrival timestamp in microseconds since epoch

id

trade id as provided by exchange, empty if exchange does not provide one - different exchanges provide id's as numeric values, GUID's or other strings, and some do not provide that information at all

side

liquidity taker side (aggressor), possible values:

  • buy - liquidity taker was buying

  • sell - liquidity taker was selling

  • unknown - exchange did not provide that information

price

trade price as provided by exchange

amount

trade amount as provided by exchange

dataset preview

exchange

symbol

timestamp

local_timestamp

id

side

price

amount

bitmex

XBTUSD

1585699202957000

1585699203089980

d20...

buy

6425.5

12

bitmex

XBTUSD

1585699202980000

1585699203095276

619...

sell

6425

150

bitmex

XBTUSD

1585699203002000

1585699203099299

751...

sell

6425

25

bitmex

XBTUSD

1585699203092000

1585699203122233

3c1...

buy

6425.5

1

bitmex

XBTUSD

1585699203092000

1585699203122233

b9b...

buy

6425.5

1

bitmex

XBTUSD

1585699203092000

1585699203122233

433...

buy

6425.5

1

bitmex

XBTUSD

1585699203092000

1585699203122233

d16...

buy

6425.5

1

bitmex

XBTUSD

1585699203092000

1585699203122233

402...

buy

6425.5

1

bitmex

XBTUSD

1585699203092000

1585699203122233

2f8...

buy

6425.5

1

● options_chain

Tick-level options summary info (strike prices, expiration dates, open interest, implied volatility, greeks etc.) for all active options instruments collected from exchanges' real-time WebSocket options tickers data feeds. Options chain data is available for Deribit (sourced from ticker channel) and OKEx Options (sourced from option/summary and index/ticker channels).

For options_chain data type only 'OPTIONS' symbol is available (one file per day for all options instruments).

CSV options_chain schema
dataset preview
CSV options_chain schema

column name

description

exchange

exchange id, one of https://api.tardis.dev/v1/exchanges ([].id field)

symbol

instrument symbol as provided by exchange (always uppercase)

timestamp

ticker timestamp provided by exchange in microseconds since epoch

local_timestamp

ticker message arrival timestamp in microseconds since epoch

type

option type, possible values:

  • put

  • call

strike_price

option strike price

expiration

option expiration date in microseconds since epoch

open_interest

current open interest, empty is exchange does not provide one

last_price

price of the last trade, empty if there weren't any trades yet

bid_price

current best bid price, empty if there aren't any bids

bid_amount

current best bid amount, empty if there aren't any bids

bid_iv

implied volatility for best bid, empty if there aren't any bids

ask_price

current best ask price, empty if there aren't any asks

ask_amount

current best ask amount, empty if there aren't any asks

ask_iv

implied volatility for best ask, empty if there aren't any asks

mark_price

mark price, empty is exchange does not provide one

mark_iv

implied volatility for mark price, empty is exchange does not provide one

underlying_index

underlying index name that option contract is based upon

underlying_price

underlying price, empty is exchange does not provide one, empty is exchange does not provide one

delta

delta value for the option, empty is exchange does not provide one

gamma

gamma value for the option, empty is exchange does not provide one

vega

vega value for the option, empty is exchange does not provide one

theta

theta value for the option, empty is exchange does not provide one

rho

rho value for the option, empty is exchange does not provide one

dataset preview

exchange

symbol

timestamp

local_timestamp

type

strike_price

expiration

open_interest

last_price

bid_price

bid_amount

bid_iv

ask_price

ask_amount

ask_iv

mark_price

mark_iv

underlying_index

underlying_price

delta

gamma

vega

theta

rho

deribit

BTC-9JUN20-9875-P

1591574399413000

1591574400196008

put

9875

1591689600000000

0.1

0.0295

0.0205

15.0

55.91

0.0235

15.0

68.94

0.02210436

62.89

SYN.BTC-9JUN20

9756.36

-0.61752

0.00103

2.24964

-53.05655

-0.22796

deribit

BTC-9JUN20-9875-P

1591574404454000

1591574404473112

put

9875

1591689600000000

0.1

0.0295

0.0205

15.0

55.91

0.0235

15.0

68.94

0.02209480

62.86

SYN.BTC-9JUN20

9756.37

-0.61757

0.00103

2.24954

-53.02754

-0.22798

deribit

BTC-9JUN20-9875-C

1591574397505000

1591574400196010

call

9875

1591689600000000

44.3

0.0080

0.0095

0.5

61.00

0.0105

20.0

65.33

0.00992836

62.87

SYN.BTC-9JUN20

9756.25

0.38232

0.00103

2.24933

-53.03038

0.13272

deribit

BTC-9JUN20-9750-C

1591574399414000

1591574400196011

call

9750

1591689600000000

30.5

0.0145

0.0145

0.3

58.80

0.0160

20.0

65.02

0.01527998

62.05

SYN.BTC-9JUN20

9756.36

0.51442

0.00109

2.35092

-54.69903

0.17789

deribit

BTC-9JUN20-9750-P

1591574397562000

1591574400196012

put

9750

1591689600000000

0.8

0.0185

0.0140

0.3

59.40

0.0155

0.3

65.63

0.01464260

62.06

SYN.BTC-9JUN20

9756.25

-0.48570

0.00109

2.35092

-54.70775

-0.17832

deribit

BTC-9JUN20-9625-P

1591574397824000

1591574400197202

put

9625

1591689600000000

9.5

0.0130

0.0090

0.4

61.64

0.0105

0.4

68.29

0.00975848

65.01

SYN.BTC-9JUN20

9756.25

-0.35780

0.00097

2.20136

-53.66975

-0.13100

deribit

BTC-9JUN20-9625-C

1591574397359000

1591574400197208

call

9625

1591689600000000

18.0

0.0220

0.0215

15.0

57.39

0.0235

20.0

66.31

0.02320750

65.02

SYN.BTC-9JUN20

9756.18

0.64212

0.00097

2.20150

-53.67532

0.22058

deribit

BTC-9JUN20-9500-P

1591574397940000

1591574400197209

put

9500

1591689600000000

51.5

0.0065

0.0060

0.5

66.31

0.0065

17.1

68.91

0.00625625

67.64

SYN.BTC-9JUN20

9756.30

-0.25091

0.00080

1.87744

-47.62196

-0.09165

deribit

BTC-9JUN20-9500-C

1591574399413000

1591574400197211

call

9500

1591689600000000

43.8

0.0165

0.0315

20.0

62.19

0.0350

0.4

80.06

0.03252520

67.64

SYN.BTC-9JUN20

9756.36

0.74914

0.00080

1.87725

-47.61618

0.25540

● quotes

Top of the book (best bid/ask) data sourced from reconstructed L2 order book state - best bid/ask recorded every time top of the book has changed. We on purpose choose this solution over native exchanges real-time quotes feeds as those vary a lot between exchanges, can be throttled, some are absent at all, often are delayed and published in batches in comparison to more granular L2 updates which are the basis for our quotes dataset.

CSV quotes schema
dataset preview
CSV quotes schema

column name

description

exchange

exchange id, one of https://api.tardis.dev/v1/exchanges ([].id field)

symbol

instrument symbol as provided by exchange (always uppercase)

timestamp

timestamp provided by exchange in microseconds since epoch - if exchange does not provide one local_timestamp value is used as a fallback

local_timestamp

message arrival timestamp in microseconds since epoch

ask_amount

best ask amount as provided by exchange

ask_price

best ask price as provided by exchange

bid_price

best bid price as provided by exchange

bid_amount

best bid amount as provided by exchange

dataset preview

exchange

symbol

timestamp

local_timestamp

ask_amount

ask_price

bid_price

bid_amount

huobi-dm-swap

BTC-USD

1585699201147000

1585699201270777

86

6423

6422.9

112

huobi-dm-swap

BTC-USD

1585699201175000

1585699201292111

86

6423

6422.9

114

huobi-dm-swap

BTC-USD

1585699201257000

1585699201373479

84

6423

6422.9

219

huobi-dm-swap

BTC-USD

1585699201279000

1585699201495667

64

6423

6422.9

219

huobi-dm-swap

BTC-USD

1585699201295000

1585699201495715

64

6423

6422.9

229

huobi-dm-swap

BTC-USD

1585699201447000

1585699201564788

2

6423

6422.9

229

huobi-dm-swap

BTC-USD

1585699201556000

1585699201677770

64

6423

6422.9

229

huobi-dm-swap

BTC-USD

1585699201668000

1585699201784213

64

6423

6422.9

235

huobi-dm-swap

BTC-USD

1585699201747000

1585699201865051

2

6423

6422.9

235

● derivative_ticker

Derivative instrument ticker info (open interest, funding, mark price, index price) collected from exchanges' real-time WebSocket instruments & tickers data feeds. Anytime any of the tracked values has changed data was added to final dataset.

CSV derivative_ticker schema
dataset preview
CSV derivative_ticker schema

column name

description

exchange

exchange id, one of https://api.tardis.dev/v1/exchanges ([].id field)

symbol

instrument symbol as provided by exchange (always uppercase)

timestamp

timestamp provided by exchange in microseconds since epoch - if exchange does not provide one local_timestamp value is used as a fallback

local_timestamp

message arrival timestamp in microseconds since epoch

funding_timestamp

timestamp of the next funding event in microseconds since epoch, empty if exchange does not provide one

funding_rate

current funding rate, empty if exchange does not provide one

predicted_funding_rate

estimated next funding rate, empty if exchange does not provide one

open_interest

current open interest, empty if exchange does not provide one

last_price

last instrument price, empty if exchange does not provide one

index_price

index price of the instrument, empty if exchange does not provide one

mark_price

mark price of the instrument, empty if exchange does not provide one

dataset preview

1

exchange

symbol

timestamp

local_timestamp

funding_timestamp

funding_rate

predicted_funding_rate

open_interest

last_price

index_price

mark_price

2

bitmex

ETHUSD

1585699199651000

1585699202577291

1585713600000000

0.0001

0.001654

45921455

133.25

133.14

133.15

3

bitmex

ETHUSD

1585699200000000

1585699204834359

1585713600000000

0.0001

0.001654

45921455

133.25

133.12

133.13

4

bitmex

ETHUSD

1585699202925000

1585699205076090

1585713600000000

0.0001

0.001654

45921455

133.3

133.12

133.13

5

bitmex

ETHUSD

1585699202925000

1585699205090339

1585713600000000

0.0001

0.001654

45883853

133.3

133.12

133.13

6

bitmex

ETHUSD

1585699203465000

1585699205274555

1585713600000000

0.0001

0.001654

45883853

133.25

133.12

133.13

7

bitmex

ETHUSD

1585699204439000

1585699205951209

1585713600000000

0.0001

0.001654

45883853

133.15

133.12

133.13

8

bitmex

ETHUSD

1585699205000000

1585699206389317

1585713600000000

0.0001

0.001654

45883853

133.15

133.09

133.1

9

bitmex

ETHUSD

1585699207279000

1585699207490211

1585713600000000

0.0001

0.001654

45883853

133.2

133.09

133.1

10

bitmex

ETHUSD

1585699207279000

1585699208084951

1585713600000000

0.0001

0.001654

45867677

133.2

133.09

133.1

Datasets API details

Download via client libraries

Historical datasets for the first day of each month are available to download without API key.

Python
Node.js
Python
# pip install tardis-dev
# requires Python >=3.6
from tardis_dev import datasets, get_exchange_details
import logging
# comment out to disable debug logs
logging.basicConfig(level=logging.DEBUG)
# function used by default if not provided via options
def default_file_name(exchange, data_type, date, symbol, format):
return f"{exchange}_{data_type}_{date.strftime('%Y-%m-%d')}_{symbol}.{format}.gz"
# customized get filename function - saves data in nested directory structure
def file_name_nested(exchange, data_type, date, symbol, format):
return f"{exchange}/{data_type}/{date.strftime('%Y-%m-%d')}_{symbol}.{format}.gz"
# returns data available at https://api.tardis.dev/v1/exchanges/deribit
deribit_details = get_exchange_details("deribit")
# print(deribit_details)
datasets.download(
# one of https://api.tardis.dev/v1/exchanges with supportsDatasets:true - use 'id' value
exchange="deribit",
# accepted data types - 'datasets.symbols[].dataTypes' field in https://api.tardis.dev/v1/exchanges/deribit,
# or get those values from 'deribit_details["datasets"]["symbols][]["dataTypes"] dict above
data_types=["incremental_book_L2", "trades", "quotes", "derivative_ticker"],
# change date ranges as needed to fetch full month or year for example
from_date="2019-11-01",
# to date is non inclusive
to_date="2019-11-02",
# accepted values: 'datasets.symbols[].id' field in https://api.tardis.dev/v1/exchanges/deribit
symbols=["BTC-PERPETUAL", "ETH-PERPETUAL",],
# (optional) your API key to get access to non sample data as well
api_key="YOUR API KEY",
# (optional) path where data will be downloaded into, default dir is './datasets'
# download_dir="./datasets",
# (optional) - one can customize downloaded file name/path (flat dir strucure, or nested etc) - by default function 'default_file_name' is used
# get_filename=default_file_name,
# (optional) file_name_nested will download data to nested directory structure (split by exchange and data type)
# get_filename=file_name_nested,
)

If you're running into RuntimeError: This event loop is already running error try solution from https://github.com/ipython/ipython/issues/11338#issuecomment-646539516 (adding nest_asyncio).

Node.js
// npm install [email protected]
// requires node version >=12
// remove it to disable debug logs
process.env.DEBUG = 'tardis-dev*'
const { downloadDatasets, getExchangeDetails } = require('tardis-dev')
;(async () => {
// returns data available at https://api.tardis.dev/v1/exchanges/deribit
const deribitDetails = await getExchangeDetails('deribit')
// console.log(deribitDetails.datasets)
await downloadDatasets({
exchange: 'deribit', // one of https://api.tardis.dev/v1/exchanges with supportsDatasets:true - use 'id' value
dataTypes: ['incremental_book_L2', 'trades', 'quotes', 'derivative_ticker'], // accepted data types - 'datasets.symbols[].dataTypes' field in https://api.tardis.dev/v1/exchanges/deribit, or get those values from 'deribitDetails.datasets.symbols[].dataTypes' object above
from: '2019-11-01', // change date ranges as needed to fetch full month or year for example
to: '2019-11-02', // to date is non inclusive
symbols: ['BTC-PERPETUAL', 'ETH-PERPETUAL'], // accepted values: 'datasets.symbols[].id' field in https://api.tardis.dev/v1/exchanges/deribit, or `deribitDetails.datasets.symbols[].id` from object above
apiKey: 'YOUR_API_KEY', // (optional) your API key to get access to non sample data as well
// downloadDir:'./datasets', // (optional) path where data will be downloaded into, default dir is './datasets'
// getFilename: getFilenameDefault, // (optional) - one can customize downloaded file name/path (flat dir strucure, or nested etc) - by default function 'getFilenameDefault' is used
// getFilename: getFilenameCustom // (optional) getFilenameCustom will download data to nested directory structure (split by exchange and data type)
})
})().catch((e) => {
console.log('download error', e)
})
// function used by default if not provided via options
function getFilenameDefault({ exchange, dataType, format, date, symbol }) {
return `${exchange}_${dataType}_${date.toISOString().split('T')[0]}_${symbol}.${format}.gz`
}
// customized get filename function - saves data in nested directory structure
function getFilenameCustom({ exchange, dataType, format, date, symbol }) {
return `${exchange}/${dataType}/${date.toISOString().split('T')[0]}_${symbol}.${format}.gz`
}

get
Datasets API reference

https://datasets.tardis.dev/v1/:exchange/:dataType/:year/:month/:day/:symbol.csv.gz
Returns gzip compressed CSV dataset for given exchange, data type, date (year, month, day) and symbol.
Request
Response
Request
Path Parameters
exchange
required
string
one of https://api.tardis.dev/v1/exchanges (field id, only exchanges with "supportsDatasets":true)
dataType
required
string
one of datasets.symbols[].dataTypes values from https://api.tardis.dev/v1/exchanges/:exchange API response
year
required
string
year in format YYYY (four-digit year)
month
required
string
month in format MM (two-digit month of the year)
day
optional
string
day in format DD (two-digit day of the month)
symbol
required
string
one of datasets.symbols[].id values from https://api.tardis.dev/v1/exchanges/:exchange API response, see details below
Headers
Authorization
optional
string
For authenticated requests provide Authorization header with value: 'Bearer YOUR_API_KEY'. Without API key historical datsets for the first day of each month are available to download.
Response
200: OK
gzip compressed CSV dataset

Sample requests