Replaying Historical Data

This page contains the detailed historical replay reference for the Node.js tardis-dev client. For a shorter onboarding flow with simple examples, see Getting Started.

See historical data details page to get detailed information about historical market data available for each exchange.

replay(options)

Replays historical market data messages for given replay options in exchange-native format. Historical market data is being fetched efficiently (in parallel) from the Tardis.dev HTTP API and cached locally. Returns async iterable.

const { replay } = require('tardis-dev')

const messages = replay({
  exchange: 'bitmex',
  filters: [
    { channel: 'trade', symbols: ['XBTUSD'] },
    { channel: 'orderBookL2', symbols: ['XBTUSD'] }
  ],
  from: '2019-05-01',
  to: '2019-05-02'
})

for await (const message of messages) {
  console.log(message)
}

stream(options) is the real-time counterpart of replay function, returning real-time market data in the same format.

When symbols array is empty or omitted in filters, data for all active symbols is returned.

replay options

name	type	default	description
exchange	string	-	requested exchange id - one of allowed values
filters	{channel:string, symbols?: string[]}[]	[]	optional filters of requested historical data feed - use getExchangeDetails function from Getting Started to get allowed channels and symbols ids for requested exchange
from	string	-	replay period start date (UTC) in a format recognized by the Date.parse(), e.g., 2019-04-01
to	string	-	replay period end date (UTC) in a format recognized by the Date.parse(), e.g., 2019-04-02
skipDecoding	boolean (optional)	undefined	when set to true returns messages as buffers instead of decoding them to objects
withDisconnects	boolean (optional)	undefined	when set to true returns message with value undefined for events when connection that was recording the historical data got disconnected
apiKey	string (optional)	undefined	API key for Tardis.dev HTTP API - if not provided only first day of each month of historical data is accessible. It can also be set via init function from Getting Started for all replay calls.
autoCleanup	boolean (optional)	undefined	when set to true, automatically removes cached data from disk after it has been processed — useful for large backfills where disk space is a concern. Not safe for concurrent replay — see warning below.
waitWhenDataNotYetAvailable	boolean (optional)	undefined	when set to true, if requested data is not yet available (e.g., requesting data from the last few minutes), the replay will wait until the data becomes available instead of returning an error

Each replay request triggers many parallel HTTP requests to the Tardis API (one per minute of data per channel/symbol combination). Running multiple concurrent replay processes can quickly exceed API rate limits, causing request failures.

autoCleanup concurrency: avoid using autoCleanup with concurrent replay — the cache path is keyed by exchange, a hash of the filters, and the calendar day, so any concurrent jobs sharing those three components will conflict (even with non-overlapping time windows within the same day) and cause file-not-found errors. Clean the cache manually after all jobs complete instead.

type of messages provided by replay iterator (for await ...of)

type Message =
  | {
      localTimestamp: Date // local timestamp when message has been received
      message: any // message in exchange-native data format
    }
    // when skipDecoding is set to true
  | {
      localTimestamp: Buffer
      message: Buffer
    }

// when withDisconnects is set to true whole message can be undefined (disconnect)
Message | undefined

sample message

{
  localTimestamp: 2019-05-01T00:00:07.013Z,
  message: {
    table: 'orderBookL2',
    action: 'update',
    data: [ { symbol: 'XBTUSD', id: 8799474100, side: 'Buy', size: 167340 } ]
  }
}

replayNormalized(options, ...normalizers)

Replays historical market data messages for given replay options and normalizes messages using normalizers provided as rest arguments. Historical market data is being fetched efficiently (in parallel) from the Tardis.dev HTTP API and cached locally. Returns async iterable.

const tardis = require('tardis-dev')
const { replayNormalized, normalizeTrades, normalizeBookChanges } = tardis

const messages = replayNormalized(
  {
    exchange: 'bitmex',
    symbols: ['XBTUSD', 'ETHUSD'],
    from: '2019-05-01',
    to: '2019-05-02'
  },
  normalizeTrades,
  normalizeBookChanges
)

for await (const message of messages) {
  console.log(message)
}

streamNormalized(options, ...normalizers) is the real-time counterpart of replayNormalized function, returning real-time market data in the same format.

replay normalized options

name	type	default	description
exchange	string	-	requested exchange id - one of allowed values
symbols	string[] (optional)	undefined	optional symbols for requested data feed - use getExchangeDetails function from Getting Started to get allowed symbols ids for requested exchange
from	string	-	replay period start date (UTC) in a format recognized by the Date.parse(), e.g., 2019-04-01
to	string	-	replay period end date (UTC) in a format recognized by the Date.parse() e.g., 2019-04-02
withDisconnectMessages	boolean (optional)	undefined	when set to true returns disconnect messages for events when connection that was recording the historical data got disconnected
apiKey	string (optional)	undefined	API key for Tardis.dev HTTP API - if not provided only first day of each month of historical data is accessible. It can also be set via init function from Getting Started for all replayNormalized calls.
autoCleanup	boolean (optional)	undefined	when set to true, automatically removes cached data from disk after it has been processed — useful for large backfills where disk space is a concern. Not safe for concurrent replay — see replay options warning.
waitWhenDataNotYetAvailable	boolean (optional)	undefined	when set to true, if requested data is not yet available (e.g., requesting data from the last few minutes), the replay will wait until the data becomes available instead of returning an error

Built-in normalizers

replayNormalized function accepts any number of normalizers as rest parameters that map from exchange-native format to normalized data format. tardis-dev ships with built in ones that normalize trades, order book and derivative ticker data but also allows adding custom ones.

types of messages provided by replayNormalized iterator (for await ...of)

Message types and formats depend on specific normalizers provided to replayNormalized function and are documented in detail in Normalization and Local Processing.

sample message

Sample message produced by normalizeBookChanges

{
  type: 'book_change',
  symbol: 'XBTUSD',
  exchange: 'bitmex',
  isSnapshot: false,
  bids: [{ price: 5263.5, amount: 1780043 }],
  asks: [],
  timestamp: 2019-05-01T00:00:04.430Z,
  localTimestamp: 2019-05-01T00:00:04.430Z
}