# Replaying Historical Data {% hint style="info" %} See [historical data details](https://docs.tardis.dev/historical-data-details/overview) page to get detailed information about historical market data available for each exchange. {% endhint %} ## `replay(options)` Replays historical market data messages for given replay options in [exchange-native format](https://docs.tardis.dev/faq/data#what-is-a-difference-between-exchange-native-and-normalized-data-format). Historical market data is fetched efficiently (in parallel) from the [Tardis.dev HTTP API](https://docs.tardis.dev/api/http-api-reference#data-feeds-exchange) and cached locally. Returns [`async iterable`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of). Replay requests use `zstd` compression by default. To force `gzip`, set `init({ dataFeedCompression: 'gzip' })` before calling `replay()` or `replayNormalized()`. ```javascript import { replay } from 'tardis-dev' const messages = replay({ exchange: 'binance', filters: [ { channel: 'trade', symbols: ['btcusdt'] }, { channel: 'depth', symbols: ['btcusdt'] } ], from: '2024-03-01', to: '2024-03-02' }) for await (const { localTimestamp, message } of messages) { console.log(localTimestamp, message) } ``` {% hint style="info" %} [stream(options)](https://docs.tardis.dev/streaming-real-time-data#stream-options) is the real-time counterpart of `replay` function, returning real-time market data in the same format. {% endhint %} When `symbols` array is empty or omitted in filters, data for all active symbols is returned. #### replay options | name | type | default | description | | ------------------------------- | ---------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **exchange** | string | - | requested exchange id - one of [allowed values](https://github.com/tardis-dev/tardis-node/blob/master/src/consts.ts#L1) | | **filters** | {channel:string, symbols?: string\[]}\[] | \[] | optional filters of requested historical data feed - use [`getExchangeDetails`](https://docs.tardis.dev/quickstart#historical-market-data-helpers) function from [Getting Started](https://docs.tardis.dev/quickstart#historical-market-data-helpers) to get allowed channels and symbols ids for requested exchange | | **from** | string | - | replay period start date (UTC) in a format recognized by the [Date.parse()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse), e.g., `2019-04-01` | | **to** | string | - | replay period end date (UTC) in a format recognized by the [Date.parse()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse), e.g., `2019-04-02` | | **skipDecoding** | boolean (optional) | undefined | when set to `true` returns messages as buffers instead of decoding them to objects | | **withDisconnects** | boolean (optional) | undefined | when set to `true` returns message with value `undefined` for events when connection that was recording the historical data got disconnected | | **apiKey** | string (optional) | undefined | API key for Tardis.dev[ HTTP API](https://docs.tardis.dev/api/http-api-reference#data-feeds-exchange) - if not provided only first day of each month of historical data is accessible. It can also be set via [`init`](https://docs.tardis.dev/quickstart#historical-market-data-helpers) function from [Getting Started](https://docs.tardis.dev/quickstart#historical-market-data-helpers) for all replay calls. | | **autoCleanup** | boolean (optional) | undefined | when set to `true`, automatically removes cached data from disk after it has been processed — useful for large backfills where disk space is a concern. Not safe for concurrent replay — see warning below. | | **waitWhenDataNotYetAvailable** | boolean or number (optional) | undefined | when set to `true`, waits for data that is not yet available. When set to a number, specifies the offset in minutes | | **withMicroseconds** | boolean (optional) | undefined | when set to `true`, `localTimestamp` is returned with microsecond precision instead of millisecond | {% hint style="warning" %} Each replay request triggers many parallel HTTP requests to the Tardis API (one per minute of data per channel/symbol combination). Running multiple concurrent replay processes can quickly exceed [API rate limits](https://docs.tardis.dev/faq/general#are-there-any-rate-limits-for-the-api), causing request failures. {% endhint %} {% hint style="warning" %} **`autoCleanup` concurrency**: avoid using `autoCleanup` with concurrent replay — the cache path is keyed by exchange, a hash of the filters, and the calendar day, so any concurrent jobs sharing those three components will conflict (even with non-overlapping time windows within the same day) and cause file-not-found errors. Clean the cache manually after all jobs complete instead. {% endhint %} #### type of messages provided by `replay` iterator ([`for await ...of`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of)) ```typescript type Message = | { localTimestamp: Date // local timestamp when message has been received message: any // message in exchange-native data format } // when skipDecoding is set to true | { localTimestamp: Buffer message: Buffer } // when withDisconnects is set to true whole message can be undefined (disconnect) Message | undefined ``` #### sample message ```javascript { localTimestamp: 2024-03-01T00:00:00.001Z, message: { stream: 'btcusdt@trade', data: { e: 'trade', E: 1709251199998, s: 'BTCUSDT', t: 3445374963, p: '61130.98000000', q: '0.00145000', b: 25230662315, a: 25230662808, T: 1709251199998, m: true, M: true } } } ``` ## `replayNormalized(options, ...normalizers)` Replays historical market data messages for given replay options and **normalizes messages** using normalizers provided as [rest arguments](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/rest_parameters). Historical market data is fetched efficiently (in parallel) from the [Tardis.dev HTTP API](https://docs.tardis.dev/api/http-api-reference#data-feeds-exchange) and cached locally. Returns [`async iterable`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of). ```javascript import { replayNormalized, normalizeTrades, normalizeBookChanges } from 'tardis-dev' const messages = replayNormalized( { exchange: 'binance', symbols: ['btcusdt'], from: '2024-03-01', to: '2024-03-02' }, normalizeTrades, normalizeBookChanges ) for await (const message of messages) { console.log(message) } ``` {% hint style="info" %} [streamNormalized(options, ...normalizers)](https://docs.tardis.dev/streaming-real-time-data#streamnormalized-options-...normalizers) is the real-time counterpart of `replayNormalized` function, returning real-time market data in the same format. {% endhint %} #### replay normalized options | name | type | default | description | | ------------------------------- | ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **exchange** | string | - | requested exchange id - one of [allowed values](https://github.com/tardis-dev/tardis-node/blob/master/src/consts.ts#L1) | | **symbols** | string\[] (optional) | undefined | optional symbols for requested data feed - use [`getExchangeDetails`](https://docs.tardis.dev/quickstart#historical-market-data-helpers) function from [Getting Started](https://docs.tardis.dev/quickstart#historical-market-data-helpers) to get allowed symbols ids for requested exchange | | **from** | string | - | replay period start date (UTC) in a format recognized by the [Date.parse()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse), e.g., `2019-04-01` | | **to** | string | - | replay period end date (UTC) in a format recognized by the [Date.parse()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse) e.g., `2019-04-02` | | **withDisconnectMessages** | boolean (optional) | undefined | when set to `true` returns [`disconnect`](https://docs.tardis.dev/normalization#disconnect-message) messages for events when connection that was recording the historical data got disconnected | | **apiKey** | string (optional) | undefined | API key for Tardis.dev[ HTTP API](https://docs.tardis.dev/api/http-api-reference#data-feeds-exchange) - if not provided only first day of each month of historical data is accessible. It can also be set via [`init`](https://docs.tardis.dev/quickstart#historical-market-data-helpers) function from [Getting Started](https://docs.tardis.dev/quickstart#historical-market-data-helpers) for all `replayNormalized` calls. | | **autoCleanup** | boolean (optional) | undefined | when set to `true`, automatically removes cached data from disk after it has been processed — useful for large backfills where disk space is a concern. Not safe for concurrent replay — see [replay options](#replay-options) warning. | | **waitWhenDataNotYetAvailable** | boolean or number (optional) | undefined | when set to `true`, waits for data that is not yet available. When set to a number, specifies the offset in minutes | #### Built-in normalizers `replayNormalized` function accepts any number of [normalizers](https://docs.tardis.dev/node-client/normalization) [as rest parameters](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/rest_parameters) that map from [exchange-native format](https://docs.tardis.dev/faq/data#what-is-a-difference-between-exchange-native-and-normalized-data-format) to normalized data format. `tardis-dev` ships with [built in ones that normalize trades, order book and derivative ticker data](https://docs.tardis.dev/node-client/normalization) but also allows [adding custom](https://docs.tardis.dev/normalization#modifying-built-in-and-adding-custom-normalizers) ones. #### types of messages provided by `replayNormalized` iterator ([`for await ...of`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of)) Message types and formats depend on specific normalizers provided to `replayNormalized` function and are documented in detail in [Normalization](https://docs.tardis.dev/node-client/normalization). #### sample message Sample message produced by [`normalizeTrades`](https://docs.tardis.dev/normalization#normalizetrades) ```javascript { type: 'trade', symbol: 'BTCUSDT', exchange: 'binance', id: '3445374963', price: 61130.98, amount: 0.00145, side: 'sell', timestamp: 2024-02-29T23:59:59.998Z, localTimestamp: 2024-03-01T00:00:00.001Z } ```