# Replaying Historical Data {% hint style="info" %} See [historical data details](/historical-data-details/overview.md) page to get detailed information about historical market data available for each exchange. {% endhint %} ## `replay(options)` Replays historical market data messages for given replay options in [exchange-native format](/faq/data.md#what-is-a-difference-between-exchange-native-and-normalized-data-format). Historical market data is fetched efficiently (in parallel) from the [Tardis.dev HTTP API](/api/http-api-reference.md#data-feeds-exchange) and cached locally. Returns [`async iterable`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of). Replay requests use `zstd` compression by default. To force `gzip`, set `init({ dataFeedCompression: 'gzip' })` before calling `replay()` or `replayNormalized()`. {% hint style="info" %} For filtered historical replay, the client automatically uses [API-suggested multi-minute raw data feed slices](/api/http-api-reference.md#data-feeds-exchange) to reduce the number of API requests. {% endhint %} ```javascript import { replay } from 'tardis-dev' const messages = replay({ exchange: 'binance', filters: [ { channel: 'trade', symbols: ['btcusdt'] }, { channel: 'depth', symbols: ['btcusdt'] } ], from: '2024-03-01', to: '2024-03-02' }) for await (const { localTimestamp, message } of messages) { console.log(localTimestamp, message) } ``` {% hint style="info" %} [stream(options)](/node-client/streaming-real-time-data.md#stream-options) is the real-time counterpart of `replay` function, returning real-time market data in the same format. {% endhint %} When `symbols` array is empty or omitted in filters, data for all active symbols is returned. #### replay options | name | type | default | description | | ------------------------------- | ---------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **exchange** | string | - | requested exchange id - one of [allowed values](https://github.com/tardis-dev/tardis-node/blob/master/src/consts.ts#L1) | | **filters** | {channel:string, symbols?: string\[]}\[] | \[] | optional filters of requested historical data feed - use [`getExchangeDetails`](/node-client/quickstart.md#historical-market-data-helpers) function from [Getting Started](/node-client/quickstart.md#historical-market-data-helpers) to get allowed channels and symbols ids for requested exchange | | **from** | string | - | replay period start date (UTC) in a format recognized by the [Date.parse()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse), e.g., `2019-04-01` | | **to** | string | - | replay period end date (UTC) in a format recognized by the [Date.parse()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse), e.g., `2019-04-02` | | **skipDecoding** | boolean (optional) | undefined | when set to `true` returns messages as buffers instead of decoding them to objects | | **withDisconnects** | boolean (optional) | undefined | when set to `true` returns message with value `undefined` for events when connection that was recording the historical data got disconnected | | **apiKey** | string (optional) | undefined | API key for Tardis.dev[ HTTP API](/api/http-api-reference.md#data-feeds-exchange) - if not provided only first day of each month of historical data is accessible. It can also be set via [`init`](/node-client/quickstart.md#historical-market-data-helpers) function from [Getting Started](/node-client/quickstart.md#historical-market-data-helpers) for all replay calls. | | **autoCleanup** | boolean (optional) | undefined | when set to `true`, automatically removes cached data from disk after it has been processed — useful for large backfills where disk space is a concern. Not safe for concurrent replay — see warning below. | | **waitWhenDataNotYetAvailable** | boolean or number (optional) | undefined | when set to `true`, waits for data that is not yet available using a 30-minute offset. When set to a number, specifies the offset in minutes (minimum effective value is 6 minutes) | | **withMicroseconds** | boolean (optional) | undefined | when set to `true`, `localTimestamp` is returned with microsecond precision instead of millisecond | {% hint style="warning" %} Each replay request triggers many parallel HTTP requests to the Tardis API (one per minute of data per channel/symbol combination). Running multiple concurrent replay processes can quickly exceed [API rate limits](/faq/general.md#are-there-any-rate-limits-for-the-api), causing request failures. {% endhint %} {% hint style="warning" %} **`autoCleanup` concurrency**: avoid using `autoCleanup` with concurrent replay — the cache path is keyed by exchange, a hash of the filters, and the calendar day, so any concurrent jobs sharing those three components will conflict (even with non-overlapping time windows within the same day) and cause file-not-found errors. Clean the cache manually after all jobs complete instead. {% endhint %} #### type of messages provided by `replay` iterator ([`for await ...of`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of)) ```typescript type Message = | { localTimestamp: Date // local timestamp when message has been received message: any // message in exchange-native data format } // when skipDecoding is set to true | { localTimestamp: Buffer message: Buffer } // when withDisconnects is set to true whole message can be undefined (disconnect) Message | undefined ``` #### sample message ```javascript { localTimestamp: 2024-03-01T00:00:00.001Z, message: { stream: 'btcusdt@trade', data: { e: 'trade', E: 1709251199998, s: 'BTCUSDT', t: 3445374963, p: '61130.98000000', q: '0.00145000', b: 25230662315, a: 25230662808, T: 1709251199998, m: true, M: true } } } ``` ## `replayNormalized(options, ...normalizers)` Replays historical market data messages for given replay options and **normalizes messages** using normalizers provided as [rest arguments](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/rest_parameters). Historical market data is fetched efficiently (in parallel) from the [Tardis.dev HTTP API](/api/http-api-reference.md#data-feeds-exchange) and cached locally. Returns [`async iterable`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of). ```javascript import { replayNormalized, normalizeTrades, normalizeBookChanges } from 'tardis-dev' const messages = replayNormalized( { exchange: 'binance', symbols: ['btcusdt'], from: '2024-03-01', to: '2024-03-02' }, normalizeTrades, normalizeBookChanges ) for await (const message of messages) { console.log(message) } ``` {% hint style="info" %} [streamNormalized(options, ...normalizers)](https://docs.tardis.dev/node-client/pages/NugzOsZ9SfKsMBMmecqk#streamnormalized-options-...normalizers) is the real-time counterpart of `replayNormalized` function, returning real-time market data in the same format. {% endhint %} #### replay normalized options | name | type | default | description | | ------------------------------- | ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **exchange** | string | - | requested exchange id - one of [allowed values](https://github.com/tardis-dev/tardis-node/blob/master/src/consts.ts#L1) | | **symbols** | string\[] (optional) | undefined | optional symbols for requested data feed - use [`getExchangeDetails`](/node-client/quickstart.md#historical-market-data-helpers) function from [Getting Started](/node-client/quickstart.md#historical-market-data-helpers) to get allowed symbols ids for requested exchange | | **from** | string | - | replay period start date (UTC) in a format recognized by the [Date.parse()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse), e.g., `2019-04-01` | | **to** | string | - | replay period end date (UTC) in a format recognized by the [Date.parse()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse) e.g., `2019-04-02` | | **withDisconnectMessages** | boolean (optional) | undefined | when set to `true` returns [`disconnect`](/node-client/normalization.md#disconnect-message) messages for events when connection that was recording the historical data got disconnected | | **apiKey** | string (optional) | undefined | API key for Tardis.dev[ HTTP API](/api/http-api-reference.md#data-feeds-exchange) - if not provided only first day of each month of historical data is accessible. It can also be set via [`init`](/node-client/quickstart.md#historical-market-data-helpers) function from [Getting Started](/node-client/quickstart.md#historical-market-data-helpers) for all `replayNormalized` calls. | | **autoCleanup** | boolean (optional) | undefined | when set to `true`, automatically removes cached data from disk after it has been processed — useful for large backfills where disk space is a concern. Not safe for concurrent replay — see [replay options](#replay-options) warning. | | **waitWhenDataNotYetAvailable** | boolean or number (optional) | undefined | when set to `true`, waits for data that is not yet available using a 30-minute offset. When set to a number, specifies the offset in minutes (minimum effective value is 6 minutes) | #### Built-in normalizers `replayNormalized` function accepts any number of [normalizers](/node-client/normalization.md) [as rest parameters](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/rest_parameters) that map from [exchange-native format](/faq/data.md#what-is-a-difference-between-exchange-native-and-normalized-data-format) to normalized data format. `tardis-dev` ships with [built in ones that normalize trades, order book and derivative ticker data](/node-client/normalization.md) but also allows [adding custom](/node-client/normalization.md#modifying-built-in-and-adding-custom-normalizers) ones. #### types of messages provided by `replayNormalized` iterator ([`for await ...of`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of)) Message types and formats depend on specific normalizers provided to `replayNormalized` function and are documented in detail in [Normalization](/node-client/normalization.md). #### sample message Sample message produced by [`normalizeTrades`](/node-client/normalization.md#normalizetrades) ```javascript { type: 'trade', symbol: 'BTCUSDT', exchange: 'binance', id: '3445374963', price: 61130.98, amount: 0.00145, side: 'sell', timestamp: 2024-02-29T23:59:59.998Z, localTimestamp: 2024-03-01T00:00:00.001Z } ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.tardis.dev/node-client/replaying-historical-data.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.