Node.js Client
Convenient access to tick-level historical and real-time cryptocurrency market data via Node.js
Last updated
Convenient access to tick-level historical and real-time cryptocurrency market data via Node.js
Last updated
Node.js tardis-dev
library provides convenient access to tick-level historical and real-time cryptocurrency market data both in . Instead of callbacks it relies on enabling composability features like or .
transparent historical local data caching (cached data is stored on disk in compressed GZIP format and decompressed on demand when reading the data)
automatic closed connections and stale connections reconnection logic for real-time streams
fast and lightweight architecture — low memory footprint and no heavy in-memory buffering
built-in TypeScript support
Requires Node.js v12+ installed.
Simply change from require
to ES Modules import
to enjoy first class TypeScript typings.
replay(options)
name
type
default
exchange
string
-
filters
{channel:string, symbols?: string[]}[]
[]
from
string
-
to
string
-
skipDecoding
boolean | undefined
undefined
when set to true
returns messages as buffers instead of decoding them to objects
withDisconnects
boolean | undefined
undefined
when set to true
returns message with value undefined
for events when connection that was recording the historical data got disconnected
apiKey
string | undefined
undefined
replayNormalized(options, ...normalizers)
name
type
default
exchange
string
-
symbols
string[] | undefined
undefined
from
string
-
to
string
-
withDisconnectMessages
boolean | undefined
undefined
apiKey
string | undefined
undefined
stream(options)
name
type
default
exchange
string
-
filters
{channel:string, symbols?: string[]}[]
[]
skipDecoding
boolean | undefined
undefined
when set to true
returns messages as buffers instead of decoding them to objects
withDisconnects
boolean | undefined
undefined
when set to true
returns message with value undefined
for real-time stream disconnect events
timeoutIntervalMS
number
10000
specifies time in milliseconds after which connection is restarted if no message has been received from the exchange
onError
(err) => void | undefined
undefined
Optional callback invoked when real-time WebSocket connection error occurs, useful for custom error logging etc.
streamNormalized(options, ...normalizers)
name
type
default
exchange
string
-
symbols
string[] | undefined
undefined
instruments symbols for requested data feed
withDisconnectMessages
boolean | undefined
undefined
timeoutIntervalMS
number
10000
specifies time in milliseconds after which connection is restarted if no message has been received from the exchange
onError
((err) => void) | undefined
undefined
Optional callback invoked when real-time WebSocket connection or mapping error occurs, useful for custom error logging etc.
init(options)
name
type
defaults
apiKey
string | undefined
undefined
cacheDir
string
<os.tmpdir>/.tardis-cache
path to local dir that will be used as cache location - if not provided default temp
dir for given OS will be used
getExchangeDetails(exchange)
getExchangeDetails
getApiKeyAccessInfo(apiKey?)
getApiKeyAccessInfo()
clearCache()
Clears local data cache dir.
Data normalization allows consuming market data feeds from various exchanges in consistent format.
normalizeTrades
normalizeBookChanges
normalizeDerivativeTickers
disconnect
messageExchange
is an exchange id for which mapper object needs to be returned for, localTimestamp
is a date for which mapper is created (and is created after each disconnection). In most cases localTimestamp
is not necessary for anything, but in certain cases like for example exchange API change it can be used to switch to different mapping logic like using new data channel that wasn't available until certain date.
Returned Mapper
object has following signature:
On every disconnection event that occurs normalize factory functions are called again to provide new Mapper
objects with clean state if required (stateful mapping like BitMEX order book data that needs to persist state of mapping between price level id and price level and needs to 'reset' for each new connection). If mapper object is stateful it's required to always return new clean state object from normalize factory function or reset it's state in one way or another.
normalizeLiquidations
normalizerExample implementation of custom normalizeLiquidations
function that normalizes liquidations data for deribit
exchange. Implementations for for other exchanges are left as an exercise for the reader.
type of messages provided by normalizeLiquidations
implementation of deribitLiquidations
mapper and normalizeLiquidations
normalizeLiquidations
usage example
normalizeTrades
for Binance exchangenormalizeTradesWithBinancePatch
usage example
orderBook.update(bookChange)
orderBook.bestBid()
orderBook.bestAsk()
orderBook.asks()
orderBook.bids()
combine(...iterators)
For historical data replay it combines input async iterables
messages by sorting them by localTimestamp
in ascending order, this allows synchronized/ordered market data replay for multiple exchanges.
compute(iterator, ...computables)
computeTradeBars(options)
name
type
default
kind
| 'time'
| 'volume'
| 'tick'
-
determines the way trades within a bar will be aggregated.
time
- classic OHLC candles aggregated by time
volume
- volume based trade bars agg by sum of trades amount
tick
- tick based trade bars, aggregated by trades count
interval
number
-
determines interval to aggregate by - for time based bars it's number of milliseconds, for volume based bars it's accumulated volume, for tick it's count of trades
name
string | undefined
undefined
optional custom name of trade_bar
, if not specified computed name will be provided based on kind and interval options
computeTradeBars
trade_bar
messagecomputeBookSnapshots(options)
name
type
default
depth
number
-
number of closest bids and asks levels to provide snaphot for
interval
number
-
snapshot interval in milliseconds, if 0
is provided it computes snapshots real-time any time there is a change in order book state for requested depth
name
string | undefined
undefined
optional custom name of book_snapshot
, if not specified computed name will be provided based on depth and interval options
computeBookSnaphots
book_snapshot
messagecomputable
where returned Computable
object has following signature:
computeOrderBookImbalanceRatio()
type of messages produced by computeOrderBookImbalanceRatio
BookImbalanceRatioComputable
computable
and computeOrderBookImbalanceRatio
factory function.computeOrderBookImbalanceRatio
usage exampleGiven implementation above we can compute book imbalance ratio for BitMEX real-time XBTUSD
message stream. For this example we compute top 5 levels, 2 second book snapshots as a source to our custom computable. We need to have async iterable
that produces book snapshots as a source to our book imbalance computable, hence two invocations of compute.
Example showing replaying large historical trades across multiple exchanges as those happened.
Example showing simple pattern of providing async iterable
of market data messages to the function that can process them no matter if it's is real-time or historical market data. That effectively enables having the same 'data pipeline' for backtesting and live trading.
Example showing how to quickly display real-time funding rate and open interest info across multiple exchanges at once.
Example showing how to write Deribit exchange historical funding, index and open interest data into CSV.
Example showing implementation of SimpleMovingAverageComputable
that calculates average of trade bar closes prices for specified rolling window in incremental way. It uses CircularBuffer
under the hood.
tick-level backed by Tardis.dev — includes full order book depth snapshots plus incremental updates, trades, historical open interest, funding, index, mark prices, liquidations and more
consolidated connecting directly to exchanges' public WebSocket APIs
support for both and formats (unified format for accessing market data across all supported exchanges — normalized trades, order book and ticker data)
thanks to providing unified way of consuming data messages
support for top cryptocurrency exchanges: , , , , , , , , , , , , , , , , , and more
via helper function — synchronized historical market data replay and consolidated real-time data streaming from multiple exchanges
like order book imbalance, customizable trade bars, book snapshots and more via helper function and computables
, e.g., volume based bars, top 20 levels order book snapshots taken every 10 ms etc
both for real-time and historical data via OrderBook
object
that allows adjusting normalized formats for specific needs
tardis-dev
lib uses package for verbose logging and debugging purposes that can be enabled via DEBUG
environment variable set to tardis-dev*
.
See page to get detailed information about historical market data available for each exchange.
Replays historical market data messages for given replay options in . Historical market data is being fetched efficiently (in parallel) from the and cached locally. Returns .
is the real-time counterpart of replay
function, returning real-time market data in the same format.
requested exchange id - one of
optional filters of requested historical data feed - use function to get allowed channels and symbols ids for requested exchange
replay period start date (UTC) in a format recognized by the , e.g., 2019-04-01
replay period end date (UTC) in a format recognized by the , e.g., 2019-04-02
API key for Tardis.dev - if not provided only first day of each month of historical data is accessible. It can also be set via function for all replay calls.
Replays historical market data messages for given replay options and normalizes messages using normalizers provided as . Historical market data is being fetched efficiently (in parallel) from the and cached locally. Returns .
is the real-time counterpart of replayNormalized
function, returning real-time market data in the same format.
requested exchange id - one of
optional symbols for requested data feed - use function to get allowed symbols ids for requested exchange
replay period start date (UTC) in a format recognized by the , e.g., 2019-04-01
replay period end date (UTC) in a format recognized by the e.g., 2019-04-02
when set to true
returns messages for events when connection that was recording the historical data got disconnected
API key for Tardis.dev - if not provided only first day of each month of historical data is accessible. It can also be set via function for all replayNormalized
calls.
replayNormalized
function accepts any number of that map from to normalized data format. tardis-dev
ships with but also allows ones.
Message types and formats depend on specific normalizers provided to replayNormalized
function and are documented in detail in .
Sample message produced by
Streams real-time market data messages for given stream options in . It connects directly to exchanges WebSocket APIs and transparently restarts closed, broken or stale connections (open connections without data being send for specified amount of time). Returns .
is the historical market data counterpart of stream
function, returning historical market data in the same format.
requested exchange id - one of
optional filters of requested real-time data feed - use to get allowed channels and symbols ids for requested exchange
Streams real-time market data messages for given stream options and normalizes messages using provided normalizers provided as . It connects directly to exchanges WebSocket APIs and transparently restarts closed, broken or stale connections (open connections without data being send for specified amount of time). Returns .
is the historical counterpart of streamNormalized
function, returning historical market data in the same format.
requested exchange id - one of
when set to true
returns messages for real-time stream disconnect events
streamNormalized
function can accept any number of custom normalizers as that map from to normalized data format. tardis-dev
ships with but also allows .
Message types and formats depend on specific normalizers provided to streamNormalized
function and are documented in detail in .
Sample message produced by
When working with market data via and functions by default only first day of each month of historical data is available for replay as well as locally cached historical data is stored in default location on disk (OS temp dir).
Init
function allows providing apiKey
received via email after ordering historical market data access via as well as customcacheDir
. ApiKey
can also be provided directly via options of and functions - that overrides anything that was provided via init
.
API key for Tardis.dev - if not provided only first day of each month of historical data is accessible
Given exchange id provides exchange details (available symbols, availability dates, available channels, pricing info etc) provided by API endpoint.
Given apiKey
provided as optional parameter or provided in function provides information about what historical data is available for it - exchanges, date ranges, symbols.
tardis-dev
has following built-in normalizers that can be provided to or functions:
- provides normalized trade
data
- provides normalizedbook_change
data
- provides normalized funding, index and mark price data
If you're interested in how exactly data is mapped from to normalized one, please follow code in for each exchange and if you determined that mapping should be done differently please read "" section.
When passed as an arg to or function provides normalized trade
data for all supported exchanges.
When passed as an arg to or function provides normalized book_change
data for all supported exchanges.
Provides initial order book snapshots (isSnapshot=true
) plus incremental updates for each order book change. Please note that amount
is the updated amount at that price level, not a delta. An amount
of 0
indicates the price level can be removed.
When passed as an arg to or function provides normalized derivative_ticker
data for supported exchanges that trade derivative instruments.
When or functions options have withDisconnectMessages
flag set to true
and disconnect event occurred (eg.: WebSocket connection close) then disconnect
message is being returned.
Intardis-dev
data normalization is implemented via normalize
factory functions provided to and functions. This design gives lots of flexibility by allowing replacing, extending and modifying built-in normalizers or adding new ones for new normalized data types without the need of forking the whole library.
Any normalize function provided to and functions needs to have following signature:
Normalized data returned by of Mapper.map
method is expected to have a shape that has at least fields as described in below to play well with other tardis-dev
functions like or .
We could as well provide the same normalizeLiquidations
function to function or use it together it with other normalizers (normalizeTrades
etc.).
Let's assume that default normalization of Binance exchange trades data doesn't fit our use case and we need to use stream as source of trade data instead of used by default stream.
tardis-dev
exports OrderBook
class that when instantiated can process normalized messages with order book snapshots and incremental updates and allows maintaining full local order book (level 2 - aggregated market-by-price) state both for real-time data as well as reconstructing historical order book state at any past point in time. It waits for first book_change
message that is a snapshot (isSnaphot = true
) and then applies subsequent updates to it. Single orderBook
object can maintain order book state only for single symbol/instrument. It uses data structure under the hood to efficiently maintain it's local state in sorted order.
Processes normalized messages to update it's internal local state that maintains ordered bids and asks sets. It ignores any non snapshot book_change
messages before initial snapshot is received. It should be called for every book_change
message received for given symbol we'd like to reconstruct order book for.
Returns for highest bid order (best bid) in order book or undefined
if book doesn't have any bids (not initialized yet with initial snapshot).
Returns for lowest ask order (best ask) in order book or undefined
if book doesn't have any asks (not initialized yet with initial snapshot).
Returns of objects for all asks available ordered from the lowest to highest ask.
Returns of objects for all bids available ordered from highest to lowest bid.
function given multiple async iterators
combines them into single one. That allows synchronized historical market data replay and consolidated streaming of real-time data for multiple exchanges via single loop.
Accepts async iterables
of as and combines them returning single async iteratable
.
For real-time market data streaming it combines input async iterables
messages in FIFO order by using .
function allows computing various derived data locally via so called like:
- computes various trade bars (OHLC, volume based bars, tick based bars) based on data
- computes various order book snapshots based on normalized
If you're interested in adding custom like for example order book imbalance, volume imbalance, open interest or funding rate based bars please read section.
function accepts async iterable
producing together with as a and returns async iterable
with normalized messages produced by provided iterable and additionally all computed messages based on provided functions. It computes and produces separate computed normalized messages for each symbol and exchange combination. When is returned by provided async iterable
it discards existing pending computables and starts from computing them from scratch.
When provided to function, computes normalized messages based on data.
When provided to function, computes normalized messages based on normalized . It produces new snapshots only if there is an actual change in order book state for requested depth
.
Any computables
provided to need to be a factory functions with following signature:
Computable.compute
returned iterator is expected to provide objects that at least have fields as described in to play well with other tardis-dev
functions like .
Example implementation of custom computeOrderBookImbalanceRatio
function that as a source data type uses book snapshots and based on it computes ratio of asks amounts (sell orders) to bids amounts (buy orders) for given depth. It may be used to determine relative buy or sell pressure.
Example showing how to very easy display real-time spread and best bid/ask info across multiple exchanges at once. It can be easily adapted to do the same for historical data ( instead of ).