Tardis.dev Documentation
Downloadable CSV filesAPI DocsHistorical Data DetailsPublic Roadmap
Search…
Welcome
Frequently Asked Questions
Downloadable CSV files
Historical Data Details
API
Getting Started
Python Client
Node.js Client
Tardis Machine Server
Instruments Metadata API
HTTP API Reference
Legal
Privacy Policy
Terms of Service
External Links
Public Roadmap
Open Source
Newsletter
Contact Us
Get API Key
Tardis.dev Home
Miscellaneous
Affiliate Program
Powered By GitBook
Node.js Client
Convenient access to tick-level historical and real-time cryptocurrency market data via Node.js

Introduction

Node.js tardis-dev library provides convenient access to tick-level historical and real-time cryptocurrency market data both in exchange-native and normalized formats. Instead of callbacks it relies on async iteration (for await ...of) enabling composability features like seamless switching between real-time data streaming and historical data replay or computing derived data locally.
​
historical market data replay
real-time market data streaming
1
const tardis = require('tardis-dev')
2
const { replayNormalized, normalizeTrades, normalizeBookChanges } = tardis
3
​
4
const messages = replayNormalized(
5
{
6
exchange: 'bitmex',
7
symbols: ['XBTUSD', 'ETHUSD'],
8
from: '2019-05-01',
9
to: '2019-05-02'
10
},
11
normalizeTrades,
12
normalizeBookChanges
13
)
14
​
15
for await (const message of messages) {
16
console.log(message)
17
}
Copied!
1
const tardis = require('tardis-dev')
2
const { streamNormalized, normalizeTrades, normalizeBookChanges } = tardis
3
​
4
const messages = streamNormalized(
5
{
6
exchange: 'bitmex',
7
symbols: ['XBTUSD', 'ETHUSD']
8
},
9
normalizeTrades,
10
normalizeBookChanges
11
)
12
​
13
for await (const message of messages) {
14
console.log(message)
15
}
Copied!
​​
​​
​

Features

Tardis-dev GitHub repository
​

Examples (TOC)

​
​
​

Installation

Requires Node.js v12+ installed.
1
npm install tardis-dev --save
Copied!
​
​
​

Debugging and logging

tardis-dev lib uses debug package for verbose logging and debugging purposes that can be enabled via DEBUG environment variable set to tardis-dev*.
​
​
​

Usage with TypeScript

Simply change from require
1
const { replay, stream } = require('tardis-dev')
Copied!
to ES Modules import
1
import { replay, stream } from 'tardis-dev'
Copied!
to enjoy first class TypeScript typings.
​
​
​

Replaying historical market data

See historical data details page to get detailed information about historical market data available for each exchange.
​

replay(options)

Replays historical market data messages for given replay options in exchange-native format. Historical market data is being fetched efficiently (in parallel) from the Tardis.dev HTTP API and cached locally. Returns async iterable.
1
const { replay } = require('tardis-dev')
2
​
3
const messages = replay({
4
exchange: 'bitmex',
5
filters: [
6
{ channel: 'trade', symbols: ['XBTUSD'] },
7
{ channel: 'orderBookL2', symbols: ['XBTUSD'] }
8
],
9
from: '2019-05-01',
10
to: '2019-05-02'
11
})
12
​
13
for await (const message of messages) {
14
console.log(message)
15
}
Copied!
​​
​​
​
​stream(options) is the real-time counterpart of replayfunction, returning real-time market data in the same format.
​

replay options

name
type
default
​
exchange
string
-
requested exchange id - one of allowed values​
filters
{channel:string, symbols?: string[]}[]
[]
optional filters of requested historical data feed - use getExchangeDetails function to get allowed channels and symbols ids for requested exchange
from
string
-
replay period start date (UTC) in a format recognized by the Date.parse(), e.g., 2019-04-01
to
string
-
replay period end date (UTC) in a format recognized by the Date.parse(), e.g., 2019-04-02
skipDecoding
boolean | undefined
undefined
when set to true returns messages as buffers instead of decoding them to objects
withDisconnects
boolean | undefined
undefined
when set to true returns message with value undefined for events when connection that was recording the historical data got disconnected
apiKey
string | undefined
undefined
API key for Tardis.dev HTTP API - if not provided only first day of each month of historical data is accessible. It can also be set via init function for all replay calls.
​

type of messages provided by replay iterator (for await ...of)

1
type Message =
2
| {
3
localTimestamp: Date // local timestamp when message has been received
4
message: any // message in exchange-native data format
5
}
6
// when skipDecoding is set to true
7
| {
8
localTimestamp: Buffer
9
message: Buffer
10
}
11
​
12
// when withDisconnects is set to true whole message can be undefined (disconnect)
13
Message | undefined
Copied!
​

sample message

1
{
2
localTimestamp: 2019-05-01T00:00:07.013Z,
3
message: {
4
table: 'orderBookL2',
5
action: 'update',
6
data: [ { symbol: 'XBTUSD', id: 8799474100, side: 'Buy', size: 167340 } ]
7
}
8
}
Copied!
​
​
​

replayNormalized(options, ...normalizers)

Replays historical market data messages for given replay options and normalizes messages using normalizers provided as rest arguments. Historical market data is being fetched efficiently (in parallel) from the Tardis.dev HTTP API and cached locally. Returns async iterable.
1
const tardis = require('tardis-dev')
2
const { replayNormalized, normalizeTrades, normalizeBookChanges } = tardis
3
​
4
const messages = replayNormalized(
5
{
6
exchange: 'bitmex',
7
symbols: ['XBTUSD', 'ETHUSD'],
8
from: '2019-05-01',
9
to: '2019-05-02'
10
},
11
normalizeTrades,
12
normalizeBookChanges
13
)
14
​
15
for await (const message of messages) {
16
console.log(message)
17
}
Copied!
​​
​​
​
​streamNormalized(options, ...normalizers) is the real-time counterpart of replayNormalized function, returning real-time market data in the same format.
​

replay normalized options

name
type
default
​
exchange
string
-
requested exchange id - one of allowed values​
symbols
string[] | undefined
undefined
optional symbols for requested data feed - use getExchangeDetails function to get allowed symbols ids for requested exchange
from
string
-
replay period start date (UTC) in a format recognized by the Date.parse(), e.g., 2019-04-01
to
string
-
replay period end date (UTC) in a format recognized by the Date.parse() e.g., 2019-04-02
withDisconnectMessages
boolean | undefined
undefined
when set to true returns disconnect messages for events when connection that was recording the historical data got disconnected
apiKey
string | undefined
undefined
API key for Tardis.dev HTTP API - if not provided only first day of each month of historical data is accessible. It can also be set via init function for all replayNormalized calls.
​

Built-in normalizers

replayNormalized function accepts any number of normalizers as rest parameters that map from exchange-native format to normalized data format. tardis-dev ships with built in ones that normalize trades, order book and derivative ticker data but also allows adding custom ones.
​

types of messages provided by replayNormalized iterator (for await ...of)

Message types and formats depend on specific normalizers provided to replayNormalized function and are documented in detail in data normalization section.
​

sample message

Sample message produced by normalizeBookChanges
1
{
2
type: 'book_change',
3
symbol: 'XBTUSD',
4
exchange: 'bitmex',
5
isSnapshot: false,
6
bids: [{ price: 5263.5, amount: 1780043 }],
7
asks: [],
8
timestamp: 2019-05-01T00:00:04.430Z,
9
localTimestamp: 2019-05-01T00:00:04.430Z
10
}
Copied!
​
​
​

Streaming real-time market data

stream(options)

Streams real-time market data messages for given stream options in exchange-native format. It connects directly to exchanges WebSocket APIs and transparently restarts closed, broken or stale connections (open connections without data being send for specified amount of time). Returns async iterable.
1
const { stream } = require('tardis-dev')
2
​
3
const messages = stream({
4
exchange: 'bitmex',
5
filters: [
6
{ channel: 'trade', symbols: ['XBTUSD'] },
7
{ channel: 'orderBookL2', symbols: ['XBTUSD'] }
8
]
9
})
10
​
11
for await (const message of messages) {
12
console.log(message)
13
}
Copied!
​​
​​
​
​replay(options) is the historical market data counterpart of stream function, returning historical market data in the same format.
​

stream options

name
type
default
​
exchange
string
-
requested exchange id - one of allowed values​
filters
{channel:string, symbols?: string[]}[]
[]
optional filters of requested real-time data feed - use getExchangeDetails to get allowed channels and symbols ids for requested exchange
skipDecoding
boolean | undefined
undefined
when set to true returns messages as buffers instead of decoding them to objects
withDisconnects
boolean | undefined
undefined
when set to true returns message with value undefined for real-time stream disconnect events
timeoutIntervalMS
number
10000
specifies time in milliseconds after which connection is restarted if no message has been received from the exchange
onError
(err) => void | undefined
undefined
Optional callback invoked when real-time WebSocket connection error occurs, useful for custom error logging etc.
​

type of messages provided by stream iterator (for await ...of)

1
type Message =
2
| {
3
localTimestamp: Date // local timestamp when message has been received
4
message: any // message in exchange-native data format
5
}
6
// when skipDecoding is set to true
7
| {
8
localTimestamp: Buffer
9
message: Buffer
10
}
11
​
12
// when withDisconnects is set to true whole message can be undefined
13
Message | undefined
Copied!
​

sample message

1
{
2
localTimestamp: 2019-10-22T09:24:51.025Z,
3
message: {
4
table: 'orderBookL2',
5
action: 'update',
6
data: [
7
{ symbol: 'XBTUSD', id: 8799172400, side: 'Sell', size: 369501 }
8
]
9
}
10
}
Copied!
​
​
​

streamNormalized(options, ...normalizers)

Streams real-time market data messages for given stream options and normalizes messages using provided normalizers provided as rest arguments. It connects directly to exchanges WebSocket APIs and transparently restarts closed, broken or stale connections (open connections without data being send for specified amount of time). Returns async iterable.
1
const tardis = require('tardis-dev')
2
const { streamNormalized, normalizeTrades, normalizeBookChanges } = tardis
3
​
4
const messages = streamNormalized(
5
{
6
exchange: 'bitmex',
7
symbols: ['XBTUSD', 'ETHUSD']
8
},
9
normalizeTrades,
10
normalizeBookChanges
11
)
12
​
13
for await (const message of messages) {
14
console.log(message)
15
}
Copied!
​​
​​
​
​replayNormalized(options) is the historical counterpart of streamNormalized function, returning historical market data in the same format.
​

stream normalized options

name
type
default
​
exchange
string
-
requested exchange id - one of allowed values​
symbols
string[] | undefined
undefined
instruments symbols for requested data feed
withDisconnectMessages
boolean | undefined
undefined
when set to true returns disconnect messages for real-time stream disconnect events
timeoutIntervalMS
number
10000
specifies time in milliseconds after which connection is restarted if no message has been received from the exchange
onError
((err) => void) | undefined
undefined
Optional callback invoked when real-time WebSocket connection or mapping error occurs, useful for custom error logging etc.
​

Built-in normalizers

streamNormalized function can accept any number of custom normalizers as rest parameters that map from exchange-native format to normalized data format. tardis-dev ships with built in ones that normalize trades, order book and derivative ticker data but also allows adding custom ones.
​

types of messages provided by streamNormalized iterator (for await ...of)

Message types and formats depend on specific normalizers provided to streamNormalized function and are documented in detail in data normalization section.
​

sample message

Sample message produced by normalizeTrades
1
{
2
type: 'trade',
3
symbol: 'XBTUSD',
4
exchange: 'bitmex',
5
id: 'b1f4b309-80e2-1ffb-340b-2f7576f6ef0d',
6
price: 7447.5,
7
amount: 100,
8
side: 'buy',
9
timestamp: 2019-10-24T13:28:07.867Z,
10
localTimestamp: 2019-10-24T13:28:07.938Z
11
}
Copied!
​
​
​

Historical market data helpers

init(options)

This function doesn't affect real-time streaming functionality in any way, it's useful only for historical data replay.
When working with market data viareplay and replayNormalized functions by default only first day of each month of historical data is available for replay as well as locally cached historical data is stored in default location on disk (OS temp dir).
Init function allows providing apiKey received via email after ordering historical market data access via Tardis.dev website as well as customcacheDir. ApiKey can also be provided directly via options of replay and replayNormalized functions - that overrides anything that was provided via init.
1
const { init } = require('tardis-dev')
2
​
3
init({
4
apiKey: 'YOUR API KEY',
5
cacheDir: 'CUSTOM CACHE DIR PATH'
6
})
Copied!
​

init options

name
type
defaults
​
apiKey
string | undefined
undefined
API key for Tardis.dev HTTP API - if not provided only first day of each month of historical data is accessible
cacheDir
string
<os.tmpdir>/.tardis-cache
path to local dir that will be used as cache location - if not provided default temp dir for given OS will be used
​
​
​

getExchangeDetails(exchange)

Given exchange id provides exchange details (available symbols, availability dates, available channels, pricing info etc) provided by exchanges/:exchange API endpoint.
1
const { getExchangeDetails } = require('tardis-dev')
2
​
3
const bitmexExchangeDetails = await getExchangeDetails('bitmex')
Copied!
​​
​​
​

type of response returned by awaiting on getExchangeDetails

1
{
2
id: string
3
name: string
4
enabled: boolean
5
filterable: boolean
6
availableSince: string
7
availableSymbols: {
8
id: string
9
type: 'spot' | 'future' | 'perpetual' | 'option'
10
availableSince: string
11
availableTo?: string
12
}[]
13
availableChannels: string[]
14
incidentReports: {
15
from: string
16
to: string
17
status: string
18
details: string
19
}
20
}
Copied!
​
​
​

getApiKeyAccessInfo(apiKey?)

Given apiKey provided as optional parameter or provided in init function provides information about what historical data is available for it - exchanges, date ranges, symbols.
1
const { getApiKeyAccessInfo } = require('tardis-dev')
2
​
3
const details = await getApiKeyAccessInfo('YOUR_API_KEY')
Copied!
​

type of response returned by awaiting on getApiKeyAccessInfo()

1
{
2
exchange: string
3
from: string
4
to: string
5
symbols: string[]
6
}[]
Copied!
​
​
​

clearCache()

Clears local data cache dir.
1
const { clearCache } = require('tardis-dev')
2
​
3
await clearCache()
Copied!
​
​
​

Data normalization

Data normalization allows consuming market data feeds from various exchanges in consistent format.
tardis-dev has following built-in normalizers that can be provided to replayNormalized or streamNormalized functions:
If you're interested in how exactly data is mapped from exchange-native format to normalized one, please follow code in tardis-dev GitHub repository for each exchange and if you determined that mapping should be done differently please read "modifying built-in and adding custom normalizers" section.
1
const { streamNormalized, normalizeTrades, normalizeBookChanges,
2
normalizeDerivativeTickers } = require('tardis-dev')
3
​
4
// or replayNormalized to replay normalized historical data
5
const messages = streamNormalized(
6
{
7
exchange: 'deribit',
8
symbols: ['BTC-PERPETUAL']
9
},
10
normalizeTrades,
11
normalizeBookChanges,
12
normalizeDerivativeTickers
13
)
14
​
15
for await (const message of messages) {
16
if (message.type === 'book_change') {
17
// process normalized book change
18
}
19
if (message.type === 'trade') {
20
// process normalized trade
21
}
22
if (message.type === 'derivative_ticker') {
23
// process normalized derivative_ticker
24
}
25
}
Copied!
​​
​​
​
​

normalizeTrades

When passed as an arg toreplayNormalized or streamNormalized function provides normalized trade data for all supported exchanges.
sample message
type definition
1
{
2
type: 'trade',
3
symbol: 'XBTUSD',
4
exchange: 'bitmex',
5
id: '282a0445-0e3a-abeb-f403-11003204ea1b',
6
price: 7996,
7
amount: 50,
8
side: 'sell',
9
timestamp: 2019-10-23T10:32:49.669Z,
10
localTimestamp: 2019-10-23T10:32:49.740Z
11
}
Copied!
1
{
2
type: 'trade'
3
symbol: string
4
exchange: string
5
id: string | undefined
6
price: number
7
amount: number
8
side: 'buy' | 'sell' | 'unknown' // liquidity taker side (aggressor)
9
timestamp: Date
10
localTimestamp: Date
11
}
Copied!
​
​

normalizeBookChanges

When passed as an arg toreplayNormalized or streamNormalized function provides normalized book_change data for all supported exchanges.
Provides initial L2 (market by price) order book snapshots (isSnapshot=true) plus incremental updates for each order book change. Please note that amount is the updated amount at that price level, not a delta. An amount of 0 indicates the price level can be removed.
sample message
type definition
1
{
2
type: 'book_change',
3
symbol: 'XBTUSD',
4
exchange: 'bitmex',
5
isSnapshot: false,
6
bids: [],
7
asks: [{ price: 7985, amount: 283318 }],
8
timestamp: 2019-10-23T11:29:53.469Z,
9
localTimestamp: 2019-10-23T11:29:53.469Z
10
}
Copied!
1
{
2
type: 'book_change'
3
symbol: string
4
exchange: string
5
isSnapshot: boolean
6
bids: { price: number; amount: number }[]
7
asks: { price: number; amount: number }[]
8
timestamp: Date
9
localTimestamp: Date
10
}
Copied!
​
​

normalizeDerivativeTickers

When passed as an arg toreplayNormalized or streamNormalized function provides normalized derivative_ticker data for supported exchanges that trade derivative instruments.
sample message
type definition
1
{
2
type: 'derivative_ticker',
3
symbol: 'BTC-PERPETUAL',
4
exchange: 'deribit',
5
lastPrice: 7987.5,
6
openInterest: 84129491,
7
fundingRate: -0.00001568,
8
indexPrice: 7989.28,
9
markPrice: 7987.56,
10
timestamp: 2019-10-23T11:34:29.302Z,
11
localTimestamp: 2019-10-23T11:34:29.416Z
12
}
Copied!
1
{
2
type: 'derivative_ticker'
3
symbol: string
4
exchange: string
5
lastPrice: number | undefined
6
openInterest: number | undefined
7
fundingRate: number | undefined
8
indexPrice: number | undefined
9
markPrice: number | undefined
10
timestamp: Date
11
localTimestamp: Date
12
}
Copied!
​
​

disconnect message

When replayNormalized or streamNormalized functions options have withDisconnectMessages flag set to true and disconnect event occurred (eg.: WebSocket connection close) then disconnect message is being returned.
sample message
type definition
1
{
2
type: 'disconnect',
3
exchange: 'deribit',
4
localTimestamp: 2019-10-23T11:34:29.416Z
5
}
Copied!
1
{
2
type: 'disconnect'
3
exchange: string
4
localTimestamp: Date
5
}
Copied!
​
​
​

Modifying built-in and adding custom normalizers

Intardis-dev data normalization is implemented via normalize factory functions provided to replayNormalized and streamNormalized functions. This design gives lots of flexibility by allowing replacing, extending and modifying built-in normalizers or adding new ones for new normalized data types without the need of forking the whole library.
Any normalize function provided to replayNormalized and streamNormalized functions needs to have following signature:
1
(exchange: string, localTimestamp: Date) => Mapper
Copied!
Exchange is an exchange id for which mapper object needs to be returned for, localTimestamp is a date for which mapper is created (and is created after each disconnection). In most cases localTimestamp is not necessary for anything, but in certain cases like for example exchange API change it can be used to switch to different mapping logic like using new data channel that wasn't available until certain date.
Returned Mapper object has following signature:
1
{
2
canHandle: (message: any) => boolean
3
​
4
map(message: any, localTimestamp: Date): IterableIterator<Data> | undefined
5
​
6
getFilters: (symbols?: string[]) => {channel: string, symbols?: string[]}[]
7
}
Copied!
​
On every disconnection event that occurs normalize factory functions are called again to provide new Mapper objects with clean state if required (stateful mapping like BitMEX order book data that needs to persist state of mapping between price level id and price level and needs to 'reset' for each new connection). If mapper object is stateful it's required to always return new clean state object from normalize factory function or reset it's state in one way or another.
Normalized data returned by iterable iterator of Mapper.map method is expected to have a shape that has at least fields as described in normalized data type section below to play well with other tardis-dev functions like combine or compute.

normalized data type

1
{
2
type: string
3
symbol: string
4
exchange: string
5
timestamp: Date
6
localTimestamp: Date
7
name? : string | undefined
8
}
Copied!
​

Adding custom normalizeLiquidations normalizer

Example implementation of custom normalizeLiquidations function that normalizes liquidations data for deribit exchange. Implementations for for other exchanges are left as an exercise for the reader.
​
type of messages provided by normalizeLiquidations
1
{
2
type: 'liquidation'
3
symbol: string
4
exchange: string
5
side: 'buy' | 'sell'
6
amount: number
7
price: number
8
timestamp: Date
9
localTimestamp: Date
10
}
Copied!
implementation of deribitLiquidations mapper and normalizeLiquidations
1
// object that maps from deribit trades real-time channel to our custom
2
// liquidations messages if determines that trade was caused by liquidation
3
​
4
const deribitLiquidationsMapper = {
5
6
// function that given message in deribit native format
7
// https://docs.deribit.com/v2/#trades-instrument_name-interval
8
// determines if such message can be mapped/normalized
9
canHandle(message) {
10
const params = message.params
11
const channel = params !== undefined ? params.channel : undefined
12
if (channel === undefined) {
13
return false
14
}
15
return (
16
channel.startsWith('trades') &&
17
params.data.some(trade => trade.liquidation !== undefined)
18
)
19
},
20
21
// given symbols returns filters that are provided
22
// as filters to replay & stream functions options
23
// in our case we're interested in trades deribit channel
24
// but it could be multiple channels in certain scenarions as well
25
getFilters(symbols) {
26
return [
27
{
28
channel: 'trades',
29
symbols
30
}
31
]
32
},
33
​
34
// map message that was determined that is can be handled via canHandle
35
// to normalized liquidation message
36
*map(message, localTimestamp) {
37
for (const deribitLiquidationTrade of message.params.data) {
38
if (deribitLiquidationTrade.liquidation === undefined) {
39
continue
40
}
41
​
42
yield {
43
type: 'liquidation',
44
symbol: deribitLiquidationTrade.instrument_name,
45
exchange: 'deribit',
46
price: deribitLiquidationTrade.price,
47
amount: deribitLiquidationTrade.amount,
48
side: deribitLiquidationTrade.direction,
49
timestamp: new Date(deribitLiquidationTrade.timestamp),
50
localTimestamp: localTimestamp
51
}
52
}
53
}
54
}
55
​
56
// provides factory function that given exchange returns mapper for it
57
// if such mapper if implemented
58
// in our case deribitLiquidationsMapper is stateless so we can return the same
59
// instance every time
60
​
61
const normalizeLiquidations = (exchange, localTimestamp) => {
62
if (exchange === 'deribit') {
63
return deribitLiquidationsMapper
64
}
65
throw new Error(`normalizeLiquidations: ${exchange} not supported`)
66
}
Copied!
​
normalizeLiquidations usage example
We could as well provide the same normalizeLiquidations function to streamNormalized function or use it together it with other normalizers (normalizeTrades etc.).
1
const { replayNormalized } = require('tardis-dev')
2
​
3
const liquidations = replayNormalized(
4
{
5
exchange: 'deribit',
6
symbols: ['BTC-PERPETUAL'],
7
from: '2019-07-01',
8
to: '2019-07-02'
9
},
10
normalizeLiquidations
11
)
12
​
13
for await (const liquidation of liquidations) {
14
console.log(liquidation)
15
}
Copied!
​​
​​
​

Changing normalizeTrades for Binance exchange

Let's assume that default normalization of Binance exchange trades data doesn't fit our use case and we need to use @aggTrade stream as source of trade data instead of used by default@trade stream.
1
// add custom binance trades mapper that used @aggTrade stream as source of
2
// normalized trades
3
const customBinanceTradesMapper = {
4
canHandle(message) {
5
return message.stream && message.stream.endsWith('@aggTrade')
6
},
7
​
8
getFilters(symbols) {
9
if (symbols !== undefined) {
10
symbols = symbols.map(s => s.toLocaleLowerCase())
11
}
12
// binance api expects all symbols to be lower cased
13
return [
14
{
15
channel: 'aggTrade',
16
symbols
17
}
18
]
19
},
20
​
21
*map(message, localTimestamp) {
22
const binanceAggTrade = message.data
23
​
24
yield {
25
type: 'trade',
26
symbol: binanceAggTrade.s,
27
exchange: 'binance',
28
id: String(binanceAggTrade.a),
29
price: Number(binanceAggTrade.p),
30
amount: Number(binanceAggTrade.q),
31
side: binanceAggTrade.m ? 'sell' : 'buy',
32
timestamp: new Date(binanceAggTrade.T),
33
localTimestamp: localTimestamp
34
}
35
}
36
}