Quickstart

Install via Docker or npm and run your first replay

Tardis-machine is a locally runnable server with built-in data caching that uses Tardis.dev HTTP API under the hood. It provides both tick-level historical and consolidated real-time cryptocurrency market data via its HTTP and WebSocket APIs and is available via npm and Docker.

Features

efficient data replay API endpoints returning historical market data for whole time periods (in contrast to Tardis.dev HTTP API where single call returns data for single minute time period)
exchange-native market data APIs
- tick-by-tick historical market data replay in exchange-native format
- HTTP and WebSocket endpoints
- WebSocket API providing historical market data replay from any given past point in time with the same data format and 'subscribe' logic as real-time exchanges' APIs - in many cases existing exchanges' WebSocket clients can be used to connect to this endpoint
normalized market data APIs
- consistent format for accessing market data across multiple exchanges
- HTTP and WebSocket endpoints
- synchronized historical market data replay across multiple exchanges
- consolidated real-time data streaming connecting directly to exchanges' WebSocket APIs
- customizable order book snapshots and trade bars data types
seamless switching between real-time data streaming and historical data replay
transparent historical local data caching (cached data is stored on disk in compressed GZIP format and decompressed on demand when reading the data)
support for top cryptocurrency exchanges: BitMEX, Deribit, Binance Spot, Binance USDT Futures, OKX Spot, HTX Spot, bitFlyer, Bitstamp, Coinbase Exchange, Kraken Futures (Crypto Facilities), Gemini, Kraken, Bitfinex, Bybit Derivatives, OKCoin and more
Open Source

Real-time streaming via tardis-machine is available only in normalized format. Exchange-native (raw) real-time data is not provided. If you need raw real-time feeds, connect directly to exchange WebSocket APIs.

Installation

Docker

Pull and run latest version of tardisdev/tardis-machine image:

### running without persistent local cache
docker run -p 8000:8000 -p 8001:8001 -e "TM_API_KEY=YOUR_API_KEY" -d tardisdev/tardis-machine

Tardis-machine server's HTTP endpoints will be available on port 8000 and WebSocket API endpoints on port 8001. Your API key will be passed via environment variable (TM_API_KEY) — simply replace YOUR_API_KEY with the API key you've received via email.

The command above does not use persistent volumes for local caching (each Docker restart will result in losing local data cache). To use ./host-cache-dir as a persistent volume (bind mount) cache directory, run:

docker run -v ./host-cache-dir:/.cache -p 8000:8000 -p 8001:8001 -e "TM_API_KEY=YOUR_API_KEY" -d tardisdev/tardis-machine

Since using volumes can cause issues especially on Windows, it's fine to run Docker image without them with the caveat of potentially poor local cache ratio after each container's restart.

Config environment variables

You can set the following environment variables to configure tardis-machine server:

name

default

description

TM_API_KEY

API key for Tardis.dev HTTP API - if not provided only first day of each month of historical data is accessible

TM_PORT

8000

HTTP port on which server will be running, WebSocket port is always this value + 1 (8001 with port set to 8000)

TM_CACHE_DIR

/.cache

path to local dir that will be used as cache location

TM_CLUSTER_MODE

false

will launch cluster of Node.js processes to handle the incoming requests if set to true, by default server runs in single process mode

TM_DEBUG

false

server will print verbose debug logs to stdout if set to true

TM_CLEAR_CACHE

false

server will clear local cache dir on startup if set to true

The local cache grows without limit by default. For large backfills this can consume significant disk space. Use TM_CLEAR_CACHE=true to clear the cache on startup, or set autoCleanup: true in replay options to remove cached data after each replay completes. See the replay options section for autoCleanup concurrency caveats.

Tardis-machine also supports exchange-specific environment variables (exchange credentials, proxy, WebSocket URL overrides, Binance rate limit tuning) — see the Node.js client environment variables for the full list.

npm

Requires Node.js v12+ and git installed.

Install and runtardis-machine server via npx command:

npx tardis-machine --api-key=YOUR_API_KEY

or install globally via npm:

npm install -g tardis-machine

and then run:

tardis-machine --api-key=YOUR_API_KEY

Tardis-machine server's HTTP endpoints will be available on port 8000 and WebSocket API endpoints on port 8001. Your API key will be passed via --api-key config flag — simply replace YOUR_API_KEY with API key you've received via email.

CLI config flags

You can configure tardis-machine server via environment variables as described in Docker section as well.

You can set following CLI config flags when starting tardis-machine server installed via npm:

name

default

description

--api-key

API key for Tardis.dev HTTP API - if not provided only first day of each month of historical data is accessible

--port

8000

HTTP port on which server will be running, WebSocket port is always this value + 1 (8001 with port set to 8000)

--cache-dir

<os.tmpdir>/.tardis-cache

path to local dir that will be used as cache location - if not provided default temp dir for given OS will be used

--cluster-mode

false

will launch cluster of Node.js processes to handle the incoming requests if set to true, by default server runs in single process mode

--debug

false

server will print verbose debug logs to stdout if set to true

--clear-cache

false

server will clear local cache dir on startup is set to true

--help

shows CLI help

--version

shows tardis-machine version number

Examples

Historical normalized replay

import asyncio
import aiohttp
import json
import urllib.parse


async def replay_normalized(options):
    encoded_options = urllib.parse.quote_plus(json.dumps(options))
    url = f"http://localhost:8000/replay-normalized?options={encoded_options}"

    async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=0)) as session:
        async with session.get(url) as response:
            async for line in response.content:
                print(json.loads(line))


asyncio.run(
    replay_normalized(
        {
            "exchange": "bitmex",
            "from": "2019-10-01",
            "to": "2019-10-02",
            "symbols": ["XBTUSD"],
            "dataTypes": ["trade", "book_change"],
        }
    )
)

const fetch = require('node-fetch')
const split2 = require('split2')

const serialize = options => encodeURIComponent(JSON.stringify(options))

async function* replayNormalized(options) {
  const url = `http://localhost:8000/replay-normalized?options=${serialize(options)}`
  const response = await fetch(url)
  const lines = response.body.pipe(split2())

  for await (const line of lines) {
    yield JSON.parse(line)
  }
}

async function run() {
  const messages = replayNormalized({
    exchange: 'bitmex',
    from: '2019-10-01',
    to: '2019-10-02',
    symbols: ['XBTUSD'],
    dataTypes: ['trade', 'book_change']
  })

  for await (const message of messages) {
    console.log(message)
  }
}

run()

Replaying Historical Data

Real-time normalized streaming

import asyncio
import aiohttp
import json
import urllib.parse


async def run():
    stream_options = {
        "exchange": "bitmex",
        "symbols": ["XBTUSD"],
        "dataTypes": ["trade", "book_change"],
    }
    options = urllib.parse.quote_plus(json.dumps(stream_options))
    url = f"ws://localhost:8001/ws-stream-normalized?options={options}"

    async with aiohttp.ClientSession() as session:
        async with session.ws_connect(url) as websocket:
            async for message in websocket:
                print(message.data)


asyncio.run(run())

const WebSocket = require('ws')

const options = encodeURIComponent(
  JSON.stringify({
    exchange: 'bitmex',
    symbols: ['XBTUSD'],
    dataTypes: ['trade', 'book_change']
  })
)

const ws = new WebSocket(`ws://localhost:8001/ws-stream-normalized?options=${options}`)

ws.onmessage = message => {
  console.log(message.data)
}

Streaming Real-Time Data

PreviousNormalization NextReplaying Historical Data

Last updated 1 hour ago

hashtagFeatures

hashtagInstallation

hashtagDocker

hashtagConfig environment variables

hashtagnpm

hashtagCLI config flags

hashtagExamples

hashtagHistorical normalized replay

hashtagReal-time normalized streaming

Features

Installation

Docker

Config environment variables

npm

CLI config flags

Examples

Historical normalized replay

Real-time normalized streaming