Node.js Client
Convenient access to tick-level historical and real-time cryptocurrency market data via Node.js
Introduction
Node.js tardis-dev
library provides convenient access to tick-level historical and real-time cryptocurrency market data both in exchange-native and normalized formats. Instead of callbacks it relies on async iteration (for await ...of) enabling composability features like seamless switching between real-time data streaming and historical data replay or computing derived data locally.
Features
historical tick-level market data replay backed by Tardis.dev HTTP API — includes full order book depth snapshots plus incremental updates, trades, historical open interest, funding, index, mark prices, liquidations and more
consolidated real-time data streaming API connecting directly to exchanges' public WebSocket APIs
support for both exchange-native and normalized market data formats (unified format for accessing market data across all supported exchanges — normalized trades, order book and ticker data)
seamless switching between real-time streaming and historical market data replay thanks to
async iterables
providing unified way of consuming data messagestransparent historical local data caching (cached data is stored on disk in compressed GZIP format and decompressed on demand when reading the data)
support for top cryptocurrency exchanges: BitMEX, Deribit, Binance, Binance Futures, FTX, OKEx, Huobi Global, Huobi DM, bitFlyer, Bitstamp, Coinbase Pro, Kraken Futures, Gemini, Kraken, Bitfinex, Bybit, OKCoin, CoinFLEX and more
automatic closed connections and stale connections reconnection logic for real-time streams
combining multiple exchanges feeds into single one via
combine
helper function — synchronized historical market data replay and consolidated real-time data streaming from multiple exchangescomputing derived data locally like order book imbalance, customizable trade bars, book snapshots and more via
compute
helper function andcomputables
, e.g., volume based bars, top 20 levels order book snapshots taken every 10 ms etcfull limit order book reconstruction both for real-time and historical data via
OrderBook
objectfast and lightweight architecture — low memory footprint and no heavy in-memory buffering
extensible mapping logic that allows adjusting normalized formats for specific needs
built-in TypeScript support
Examples (TOC)
Installation
Requires Node.js v12+ installed.
Debugging and logging
tardis-dev
lib uses debug package for verbose logging and debugging purposes that can be enabled via DEBUG
environment variable set to tardis-dev*
.
Usage with TypeScript
Simply change from require
to ES Modules import
to enjoy first class TypeScript typings.
Replaying historical market data
See historical data details page to get detailed information about historical market data available for each exchange.
replay(options)
replay(options)
Replays historical market data messages for given replay options in exchange-native format. Historical market data is being fetched efficiently (in parallel) from the Tardis.dev HTTP API and cached locally. Returns async iterable
.
stream(options) is the real-time counterpart of replay
function, returning real-time market data in the same format.
replay options
type of messages provided by replay
iterator (for await ...of
)
replay
iterator (for await ...of
)sample message
replayNormalized(options, ...normalizers)
replayNormalized(options, ...normalizers)
Replays historical market data messages for given replay options and normalizes messages using normalizers provided as rest arguments. Historical market data is being fetched efficiently (in parallel) from the Tardis.dev HTTP API and cached locally. Returns async iterable
.
streamNormalized(options, ...normalizers) is the real-time counterpart of replayNormalized
function, returning real-time market data in the same format.
replay normalized options
Built-in normalizers
replayNormalized
function accepts any number of normalizers as rest parameters that map from exchange-native format to normalized data format. tardis-dev
ships with built in ones that normalize trades, order book and derivative ticker data but also allows adding custom ones.
types of messages provided by replayNormalized
iterator (for await ...of
)
replayNormalized
iterator (for await ...of
)Message types and formats depend on specific normalizers provided to replayNormalized
function and are documented in detail in data normalization section.
sample message
Sample message produced by normalizeBookChanges
Streaming real-time market data
stream(options)
stream(options)
Streams real-time market data messages for given stream options in exchange-native format. It connects directly to exchanges WebSocket APIs and transparently restarts closed, broken or stale connections (open connections without data being send for specified amount of time). Returns async iterable
.
replay(options) is the historical market data counterpart of stream
function, returning historical market data in the same format.
stream options
type of messages provided by stream
iterator (for await ...of
)
stream
iterator (for await ...of
)sample message
streamNormalized(options, ...normalizers)
streamNormalized(options, ...normalizers)
Streams real-time market data messages for given stream options and normalizes messages using provided normalizers provided as rest arguments. It connects directly to exchanges WebSocket APIs and transparently restarts closed, broken or stale connections (open connections without data being send for specified amount of time). Returns async iterable
.
replayNormalized(options) is the historical counterpart of streamNormalized
function, returning historical market data in the same format.
stream normalized options
Built-in normalizers
streamNormalized
function can accept any number of custom normalizers as rest parameters that map from exchange-native format to normalized data format. tardis-dev
ships with built in ones that normalize trades, order book and derivative ticker data but also allows adding custom ones.
types of messages provided by streamNormalized
iterator (for await ...of
)
streamNormalized
iterator (for await ...of
)Message types and formats depend on specific normalizers provided to streamNormalized
function and are documented in detail in data normalization section.
sample message
Sample message produced by normalizeTrades
Historical market data helpers
init(options)
init(options)
This function doesn't affect real-time streaming functionality in any way, it's useful only for historical data replay.
When working with market data viareplay
and replayNormalized
functions by default only first day of each month of historical data is available for replay as well as locally cached historical data is stored in default location on disk (OS temp dir).
Init
function allows providing apiKey
received via email after ordering historical market data access via Tardis.dev website as well as customcacheDir
. ApiKey
can also be provided directly via options of replay
and replayNormalized
functions - that overrides anything that was provided via init
.
init options
getExchangeDetails(exchange)
getExchangeDetails(exchange)
Given exchange id provides exchange details (available symbols, availability dates, available channels, pricing info etc) provided by exchanges/:exchange API endpoint.
type of response returned by awaiting on getExchangeDetails
getExchangeDetails
getApiKeyAccessInfo(apiKey?)
getApiKeyAccessInfo(apiKey?)
Given apiKey
provided as optional parameter or provided in init
function provides information about what historical data is available for it - exchanges, date ranges, symbols.
type of response returned by awaiting on getApiKeyAccessInfo()
getApiKeyAccessInfo()
clearCache()
clearCache()
Clears local data cache dir.
Data normalization
Data normalization allows consuming market data feeds from various exchanges in consistent format.
tardis-dev
has following built-in normalizers that can be provided to replayNormalized
or streamNormalized
functions:
normalizeTrades
- provides normalizedtrade
datanormalizeBookChanges
- provides normalizedbook_change
datanormalizeDerivativeTickers
- provides normalized funding, index and mark price data
If you're interested in how exactly data is mapped from exchange-native format to normalized one, please follow code in tardis-dev
GitHub repository for each exchange and if you determined that mapping should be done differently please read "modifying built-in and adding custom normalizers" section.
normalizeTrades
normalizeTrades
When passed as an arg toreplayNormalized
or streamNormalized
function provides normalized trade
data for all supported exchanges.
normalizeBookChanges
normalizeBookChanges
When passed as an arg toreplayNormalized
or streamNormalized
function provides normalized book_change
data for all supported exchanges.
Provides initial L2 (market by price) order book snapshots (isSnapshot=true
) plus incremental updates for each order book change. Please note that amount
is the updated amount at that price level, not a delta. An amount
of 0
indicates the price level can be removed.
normalizeDerivativeTickers
normalizeDerivativeTickers
When passed as an arg toreplayNormalized
or streamNormalized
function provides normalized derivative_ticker
data for supported exchanges that trade derivative instruments.
disconnect
message
disconnect
messageWhen replayNormalized
or streamNormalized
functions options have withDisconnectMessages
flag set to true
and disconnect event occurred (eg.: WebSocket connection close) then disconnect
message is being returned.
Modifying built-in and adding custom normalizers
Intardis-dev
data normalization is implemented via normalize
factory functions provided to replayNormalized
and streamNormalized
functions. This design gives lots of flexibility by allowing replacing, extending and modifying built-in normalizers or adding new ones for new normalized data types without the need of forking the whole library.
Any normalize function provided to replayNormalized
and streamNormalized
functions needs to have following signature:
Exchange
is an exchange id for which mapper object needs to be returned for, localTimestamp
is a date for which mapper is created (and is created after each disconnection). In most cases localTimestamp
is not necessary for anything, but in certain cases like for example exchange API change it can be used to switch to different mapping logic like using new data channel that wasn't available until certain date.
Returned Mapper
object has following signature:
On every disconnection event that occurs normalize factory functions are called again to provide new Mapper
objects with clean state if required (stateful mapping like BitMEX order book data that needs to persist state of mapping between price level id and price level and needs to 'reset' for each new connection). If mapper object is stateful it's required to always return new clean state object from normalize factory function or reset it's state in one way or another.
Normalized data returned by iterable iterator
of Mapper.map
method is expected to have a shape that has at least fields as described in normalized data type section below to play well with other tardis-dev
functions like combine
or compute
.
normalized data type
Adding custom normalizeLiquidations
normalizer
normalizeLiquidations
normalizerExample implementation of custom normalizeLiquidations
function that normalizes liquidations data for deribit
exchange. Implementations for for other exchanges are left as an exercise for the reader.
type of messages provided by normalizeLiquidations
implementation of deribitLiquidations
mapper and normalizeLiquidations
normalizeLiquidations
usage example
We could as well provide the same normalizeLiquidations
function to streamNormalized
function or use it together it with other normalizers (normalizeTrades
etc.).
Changing normalizeTrades
for Binance exchange
normalizeTrades
for Binance exchangeLet's assume that default normalization of Binance exchange trades data doesn't fit our use case and we need to use @aggTrade
stream as source of trade data instead of used by default@trade
stream.
normalizeTradesWithBinancePatch
usage example
Limit order book reconstruction
tardis-dev
exports OrderBook
class that when instantiated can process normalized book_change
messages with order book snapshots and incremental updates and allows maintaining full local order book (level 2 - aggregated market-by-price) state both for real-time data as well as reconstructing historical order book state at any past point in time. It waits for first book_change
message that is a snapshot (isSnaphot = true
) and then applies subsequent updates to it. Single orderBook
object can maintain order book state only for single symbol/instrument. It uses Red-Black tree data structure under the hood to efficiently maintain it's local state in sorted order.
orderBook.update(bookChange)
orderBook.update(bookChange)
Processes normalized book_change
messages to update it's internal local state that maintains ordered bids and asks sets. It ignores any non snapshot book_change
messages before initial snapshot is received. It should be called for every book_change
message received for given symbol we'd like to reconstruct order book for.
orderBook.bestBid()
orderBook.bestBid()
Returns book price level object for highest bid order (best bid) in order book or undefined
if book doesn't have any bids (not initialized yet with initial snapshot).
orderBook.bestAsk()
orderBook.bestAsk()
Returns book price level object for lowest ask order (best ask) in order book or undefined
if book doesn't have any asks (not initialized yet with initial snapshot).
orderBook.asks()
orderBook.asks()
Returns iterable iterator of book price level objects for all asks available ordered from the lowest to highest ask.
orderBook.bids()
orderBook.bids()
Returns iterable iterator of book price level objects for all bids available ordered from highest to lowest bid.
book price level type
Combining data streams
combine(...iterators)
combine(...iterators)
Combine
function given multiple async iterators
combines them into single one. That allows synchronized historical market data replay and consolidated streaming of real-time data for multiple exchanges via single for await ...of
loop.
Accepts async iterables
of normalized messages as rest parameters and combines them returning single async iteratable
.
For historical data replay it combines input async iterables
messages by sorting them by localTimestamp
in ascending order, this allows synchronized/ordered market data replay for multiple exchanges.
For real-time market data streaming it combines input async iterables
messages in FIFO order by using Promise.race
.
Computing derived data locally
compute(iterator, ...computables)
compute(iterator, ...computables)
Compute
function allows computing various derived data locally via so called computables
like:
computeTradeBars
- computes various trade bars (OHLC, volume based bars, tick based bars) based on normalized trade datacomputeBookSnapshots
- computes various order book snapshots based on normalized order book data
If you're interested in adding custom computables
like for example order book imbalance, volume imbalance, open interest or funding rate based bars please read "adding custom computable" section.
Compute
function accepts async iterable
producing normalized messages together withcomputables
as a rest parameters and returns async iterable
with normalized messages produced by provided iterable and additionally all computed messages based on provided computable
functions. It computes and produces separate computed normalized messages for each symbol and exchange combination. When disconnect message
is returned by provided async iterable
it discards existing pending computables and starts from computing them from scratch.
computeTradeBars(options)
computeTradeBars(options)
When provided to compute
function, computes normalized trade_bar
messages based on normalized trade data.
compute trade bars options
type of message provided by computeTradeBars
computeTradeBars
sample normalized trade_bar
message
trade_bar
messagecomputeBookSnapshots(options)
computeBookSnapshots(options)
When provided to compute
function, computes normalized book_snapshot
messages based on normalized order book data. It produces new snapshots only if there is an actual change in order book state for requested depth
.
compute book snapshots options
type of message provided by