Introduction

​Tardis-machine is a locally runnable server with built-in data caching that uses Tardis.dev HTTP API under the hood. It provides both tick-level historical and consolidated real-time cryptocurrency market data via it's HTTP and WebSocket APIs and is available via npm and Docker.

Features

• efficient data replay API endpoints returning historical market data for whole time periods (in contrast to Tardis.dev HTTP API where single call returns data for single minute time period)

• ​exchange-native market data APIs​

  • tick-by-tick historical market data replay in exchange-native format​

  • ​HTTP and WebSocket endpoints

  • ​WebSocket API providing historical market data replay from any given past point in time with the same data format and 'subscribe' logic as real-time exchanges' APIs - in many cases existing exchanges' WebSocket clients can be used to connect to this endpoint

• ​normalized market data APIs​

  • consistent format for accessing market data across multiple exchanges

  • ​HTTP and WebSocket endpoints

  • synchronized historical market data replay across multiple exchanges​

  • ​consolidated real-time data streaming connecting directly to exchanges' WebSocket APIs

  • customizable order book snapshots and trade bars data types

  • seamless switching between real-time data streaming and historical data replay

• transparent historical local data caching (cached data is stored on disk in compressed GZIP format and decompressed on demand when reading the data)

• support for top cryptocurrency exchanges: BitMEX, Deribit, Binance, Binance Futures, FTX, OKEx, Huobi Global, Huobi DM, bitFlyer, Bitstamp, Coinbase Pro, Kraken Futures, Gemini, Kraken, Bitfinex, Bybit, OKCoin, CoinFLEX and more

  ​

• ​Open Source​

​

​

​

Installation

Docker

Pull and run latest version of tardisdev/tardis-machine image:

### running without persitent local cache
docker run -p 8000:8000 -p 8001:8001 -e "TM_API_KEY=YOUR_API_KEY" -d tardisdev/tardis-machine

Tardis-machine server's HTTP endpoints will be available on port 8000 and WebSocket API endpoints on port 8001. Your API key will be passed via ENV variable (TM_API_KEY) — simply replace YOUR_API_KEY with API key you've received via email.

Command above does not use persistent volumes for local caching (each docker restart will result in loosing local data cache). In order to use for example./host-cache-dir as persistent volume (bind mount) cache directory, run:

docker run -v ./host-cache-dir:/.cache -p 8000:8000 -p 8001:8001 -e "TM_API_KEY=YOUR_API_KEY" -d tardisdev/tardis-machine

Since using volumes can cause issues especially on Windows, it's fine to run Docker image without them with the caveat of potentially poor local cache ratio after each container's restart.

​

Config environment variables

You can set following environment config variables to configure tardis-machine server:

​

​

npm

Install and runtardis-machine server via npx command:

npx tardis-machine --api-key=YOUR_API_KEY

or install globally via npm:

npm install -g tardis-machine

and then run:

tardis-machine --api-key=YOUR_API_KEY

Tardis-machine server's HTTP endpoints will be available on port 8000 and WebSocket API endpoints on port 8001. Your API key will be passed via --api-key config flag — simply replace YOUR_API_KEY with API key you've received via email.

​

CLI config flags

You can set following CLI config flags when starting tardis-machine server installed via npm:

​

​

​

Exchange-native market data APIs

Exchange-native market data API endpoints provide historical data in exchange-native format. The main difference between HTTP and WebSocket endpoints is the logic of requesting data:

• ​HTTP API accepts request options payload via query string param

• ​WebSocket API accepts exchanges' specific 'subscribe' messages that define what data will be then "replayed" and send to WebSocket client

​

● HTTP GET /replay?options={options}

Returns historical market data messages in exchange-native format for given replay options query string param. Single streaming HTTP response returns data for the whole requested time period as NDJSON.

​

import asyncio
import aiohttp
import json
import urllib.parse
​
​
async def replay_via_tardis_machine_machine(replay_options):
    timeout = aiohttp.ClientTimeout(total=0)
​
    async with aiohttp.ClientSession(timeout=timeout) as session:
        # url encode as json object options
        encoded_options = urllib.parse.quote_plus(json.dumps(replay_options))
​
        # assumes tardis-machine HTTP API running on localhost:8000
        url = f"http://localhost:8000/replay?options={encoded_options}"
​
        async with session.get(url) as response:
            # otherwise we may get line to long errors
            response.content._high_water = 100_000_000
​
            # returned data is in NDJSON format http://ndjson.org/
            # each line is separate message JSON encoded
            async for line in response.content:
                yield line
​
​
async def run():
    lines = replay_via_tardis_machine_machine(
        {
            "exchange": "bitmex",
            "from": "2019-10-01",
            "to": "2019-10-02",
            "filters": [
                {"channel": "trade", "symbols": ["XBTUSD", "ETHUSD"]},
                {"channel": "orderBookL2", "symbols": ["XBTUSD", "ETHUSD"]},
            ],
        }
    )
​
    async for line in lines:
        message = json.loads(line)
        # localTimestamp string marks timestamp when message was received
        # message is a message dict as provided by exchange real-time stream
        print(message["localTimestamp"], message["message"])
​
​
asyncio.run(run())

const fetch = require('node-fetch')
const split2 = require('split2')
​
​
const serialize = options => {
  return encodeURIComponent(JSON.stringify(options))
}
​
async function* replayViaTardisMachine(options) {
  // assumes tardis-machine HTTP API running on localhost:8000
  const url = `http://localhost:8000/replay?options=${serialize(options)}`
  const response = await fetch(url)
​
  // returned data is in NDJSON format http://ndjson.org/
  // each line is separate message JSON encoded
​
  // split response body stream by new lines
  const lines = response.body.pipe(split2())
​
  for await (const line of lines) {
    yield line
  }
}
​
async function run() {
  const options = {
    exchange: 'bitmex',
    from: '2019-10-01',
    to: '2019-10-02',
    filters: [
      {
        channel: 'trade',
        symbols: ['XBTUSD', 'ETHUSD']
      },
      {
        channel: 'orderBookL2',
        symbols: ['XBTUSD', 'ETHUSD']
      }
    ]
  }
​
  const lines = replayViaTardisMachine(options)
​
  for await (const line of lines) {
    // localTimestamp string marks timestamp when message was received
    // message is a message object as provided by exchange real-time stream
    const { message, localTimestamp } = JSON.parse(line)
​
    console.log(message, localTimestamp)
  }
}
​
run()

curl -g 'localhost:8000/replay?options={"exchange":"bitmex","filters":[{"channel":"orderBookL2","symbols":["XBTUSD","ETHUSD"]}],"from":"2019-07-01","to":"2019-07-02"}'

​

Replay options

​HTTP /replay endpoint accepts required options query string param in url encoded JSON format.

​

Response format

Streamed HTTP response provides data in NDJSON format (new line delimited JSON) - each response line is a JSON with market data message in exchange-native format plus local timestamp:

• localTimestamp - date when message has been received in ISO 8601 format

• message - JSON with exactly the same format as provided by requested exchange real-time feeds

​

Sample response

{"localTimestamp":"2019-05-01T00:09:42.2760012Z","message":{"table":"orderBookL2","action":"update","data":[{"symbol":"XBTUSD","id":8799473750,"side":"Buy","size":2333935}]}}
{"localTimestamp":"2019-05-01T00:09:42.2932826Z","message":{"table":"orderBookL2","action":"update","data":[{"symbol":"XBTUSD","id":8799474250,"side":"Buy","size":227485}]}}
{"localTimestamp":"2019-05-01T00:09:42.4249304Z","message":{"table":"trade","action":"insert","data":[{"timestamp":"2019-05-01T00:09:42.407Z","symbol":"XBTUSD","side":"Buy","size":1500,"price":5263,"tickDirection":"ZeroPlusTick","trdMatchID":"29d7de7f-27b6-9574-48d1-3ee9874831cc","grossValue":28501500,"homeNotional":0.285015,"foreignNotional":1500}]}}
{"localTimestamp":"2019-05-01T00:09:42.4249403Z","message":{"table":"orderBookL2","action":"update","data":[{"symbol":"XBTUSD","id":8799473700,"side":"Sell","size":454261}]}}
{"localTimestamp":"2019-05-01T00:09:42.4583155Z","message":{"table":"orderBookL2","action":"update","data":[{"symbol":"XBTUSD","id":8799473750,"side":"Buy","size":2333838},{"symbol":"XBTUSD","id":8799473800,"side":"Buy","size":547746}]}}

​

​

​

● WebSocket /ws-replay?exchange={exchange}&from={fromDate}&to={toDate}

Exchanges' WebSocket APIs are designed to publish real-time market data feeds, not historical ones. Tardis-machine WebSocket /ws-replay API fills that gap and allows "replaying" historical market data from any given past point in time with the same data format and 'subscribe' logic as real-time exchanges' APIs. In many cases existing exchanges' WebSocket clients can be used to connect to this endpoint just by changing URL, and receive historical market data in exchange-native format for date ranges specified in URL query string params.

After connection is established, client has 2 seconds to send subscriptions payloads and then market data replay starts.

If two clients connect at the same time requesting data for different exchanges and provide the same session key via query string param, then data being send to those clients will be synchronized (by local timestamp).

​

import asyncio
import aiohttp
import json
​
​
async def run():
    WS_REPLAY_URL = "ws://localhost:8001/ws-replay"
    URL = f"{WS_REPLAY_URL}?exchange=bitmex&from=2019-10-01&to=2019-10-02"
​
    async with aiohttp.ClientSession() as session:
        async with session.ws_connect(URL) as websocket:
​
            await websocket.send_str(
                json.dumps(
                    {
                        "op": "subscribe",
                        "args": [
                            "trade:XBTUSD",
                            "trade:ETHUSD",
                            "orderBookL2:XBTUSD",
                            "orderBookL2:ETHUSD",
                        ],
                    }
                )
            )
​
            async for msg in websocket:
                print(msg.data)
​
​
asyncio.run(run())

const WebSocket = require('ws')
​
​
const WS_REPLAY_URL = 'ws://localhost:8001/ws-replay'
​
const ws = new WebSocket(
  `${WS_REPLAY_URL}?exchange=bitmex&from=2019-10-01&to=2019-10-02`
)
​
ws.onmessage = message => {
  console.log(message.data)
}
​
ws.onopen = () => {
  ws.send(
    JSON.stringify({
      op: 'subscribe',
      args: [
        'trade:XBTUSD',
        'trade:ETHUSD',
        'orderBookL2:XBTUSD',
        'orderBookL2:ETHUSD'
      ]
    })
  )
}

const ccxws = require('ccxws')
const BASE_URL = 'ws://localhost:8001/ws-replay'
const WS_REPLAY_URL = `${BASE_URL}?exchange=bitmex&from=2019-10-01&to=2019-10-02`
​
const bitMEXClient = new ccxws.bitmex()
// only change required for ccxws client is to point it to /ws-replay URL
bitMEXClient._wssPath = WS_REPLAY_URL
​
const market = {
  id: 'XBTUSD',
  base: 'BTC',
  quote: 'USD'
}
​
bitMEXClient.on('l2snapshot', snapshot =>
  console.log('snapshot', snapshot.asks.length, snapshot.bids.length)
)
​
bitMEXClient.on('l2update', update => console.log(update))
​
bitMEXClient.on('trade', trade => console.log(trade))
​
bitMEXClient.subscribeTrades(market)
bitMEXClient.subscribeLevel2Updates(market)

​

Query string params

​

​

​

Normalized market data APIs

Normalized market data API endpoints provide data in unified format across all supported exchanges. Both HTTP /replay-normalized and WebSocket /ws-replay-normalized APIs accept the same replay options payload via query string param. It's mostly matter of preference when choosing which protocol to use, but WebSocket /ws-replay-normalized API has also it's real-time counterpart /ws-stream-normalized, which connects directly to exchanges' real-time WebSocket APIs. This opens the possibility of seamless switching between real-time streaming and historical normalized market data replay.

​

● HTTP GET /replay-normalized?options={options}

Returns historical market data for data types specified via query string. Single streaming HTTP response returns data for the whole requested time period as NDJSON. See supported data types which include normalized trade, order book change, customizable order book snapshots etc.

​

import asyncio
import aiohttp
import json
import urllib.parse
​
​
async def replay_normalized_via_tardis_machine_machine(replay_options):
    timeout = aiohttp.ClientTimeout(total=0)
​
    async with aiohttp.ClientSession(timeout=timeout) as session:
        # url encode as json object options
        encoded_options = urllib.parse.quote_plus(json.dumps(replay_options))
​
        # assumes tardis-machine HTTP API running on localhost:8000
        url = f"http://localhost:8000/replay-normalized?options={encoded_options}"
​
        async with session.get(url) as response:
            # otherwise we may get line to long errors
            response.content._high_water = 100_000_000
​
            # returned data is in NDJSON format http://ndjson.org/ streamed
            # each line is separate message JSON encoded
            async for line in response.content:
                yield line
​
​
async def run():
    lines = replay_normalized_via_tardis_machine_machine(
        {
            "exchange": "bitmex",
            "from": "2019-10-01",
            "to": "2019-10-02",
            "symbols": ["XBTUSD", "ETHUSD"],
            "withDisconnectMessages": True,
            # other available data types examples:
            # 'book_snapshot_10_100ms', 'derivative_ticker', 'quote',
            # 'trade_bar_10ms', 'trade_bar_10s'
            "dataTypes": ["trade", "book_change", "book_snapshot_10_100ms"],
        }
    )
​
    async for line in lines:
        normalized_message = json.loads(line)
        print(normalized_message)
​
​
asyncio.run(run())

const fetch = require('node-fetch')
const split2 = require('split2')
​
​
const serialize = options => {
  return encodeURIComponent(JSON.stringify(options))
}
​
async function* replayNormalizedViaTardisMachine(options) {
  // assumes tardis-machine HTTP API running on localhost:8000
  const url = `http://localhost:8000/replay-normalized?options=${serialize(
    options
  )}`
  const response = await fetch(url)
​
  // returned data is in NDJSON format http://ndjson.org/
  // each line is separate message JSON encoded
​
  // split response body stream by new lines
  const lines = response.body.pipe(split2())
​
  for await (const line of lines) {
    yield line
  }
}
​
async function run() {
  const options = {
    exchange: 'bitmex',
    from: '2019-10-01',
    to: '2019-10-02',
    symbols: ['XBTUSD', 'ETHUSD'],
    withDisconnectMessages: true,
    // other available data types examples:
    // 'book_snapshot_10_100ms', 'derivative_ticker', 'quote',
    // 'trade_bar_10ms', 'trade_bar_10s'
    dataTypes: ['trade', 'book_change', 'book_snapshot_10_100ms']
  }
​
  const lines = replayNormalizedViaTardisMachine(options)
​
  for await (const line of lines) {
    const normalizedMessage = JSON.parse(line)
​
    console.log(normalizedMessage)
  }
}
​
run()

curl -g 'http://localhost:8000/replay-normalized?options={"exchange":"bitmex","from":"2019-10-01","to":"2019-10-02","symbols":["XBTUSD","ETHUSD"],"withDisconnectMessages":true,"dataTypes":["trade","book_change","book_snapshot_10_100ms"]}'

​

Replay normalized options

​HTTP /replay-normalized endpoint accepts required options query string param in url encoded JSON format.

Options JSON needs to be an object or an array of objects with fields as specified below. If array is provided, then data requested for multiple exchanges is returned synchronized (by local timestamp).

​

Response format & sample messages

See normalized data types.

​

​

​

● WebSocket /ws-replay-normalized?options={options}

Sends normalized historical market data for data types specified via query string. See supported data types which include normalized trade, order book change, customizable order book snapshots etc.

​

import asyncio
import aiohttp
import json
import urllib.parse
​
​
async def run():
    replay_options = {
        "exchange": "bitmex",
        "from": "2019-10-01",
        "to": "2019-10-02",
        "symbols": ["XBTUSD", "ETHUSD"],
        "withDisconnectMessages": True,
        # other available data types examples:
        # 'book_snapshot_10_100ms', 'derivative_ticker', 'quote',
        # 'trade_bar_10ms', 'trade_bar_10s'
        "dataTypes": ["trade", "book_change", "book_snapshot_10_100ms"],
    }
​
    options = urllib.parse.quote_plus(json.dumps(replay_options))
​
    URL = f"ws://localhost:8001/ws-replay-normalized?options={options}"
​
    async with aiohttp.ClientSession() as session:
        async with session.ws_connect(URL) as websocket:
            async for msg in websocket:
                print(msg.data)
​
​
asyncio.run(run())

const WebSocket = require('ws')
​
​
const serialize = options => {
  return encodeURIComponent(JSON.stringify(options))
}
​
const replayOptions = {
  exchange: 'bitmex',
  from: '2019-10-01',
  to: '2019-10-02',
  symbols: ['XBTUSD', 'ETHUSD'],
  withDisconnectMessages: true,
  // other available data types examples:
  // 'book_snapshot_10_100ms', 'derivative_ticker', 'quote',
  // 'trade_bar_10ms', 'trade_bar_10s'
  dataTypes: ['trade', 'book_change', 'book_snapshot_10_100ms']
}
​
const options = serialize(replayOptions)
const URL = `ws://localhost:8001/ws-replay-normalized?options=${options}`
const ws = new WebSocket(URL)
​
ws.onmessage = message => {
  console.log(message.data)
}
​

​

Replay normalized options

​WebSocket /ws-replay-normalized endpoint accepts required options query string param in url encoded JSON format.

Options JSON needs to be an object or an array of objects with fields as specified below. If array is provided, then data requested for multiple exchanges is being send synchronized (by local timestamp).

​

Response format & sample messages

See normalized data types.

​

​

​

● WebSocket /ws-stream-normalized?options={options}

Sends normalized real-time market data for data types specified via query string. See supported data types which include normalized trade, order book change, customizable order book snapshots etc.

Doesn't requires API key as connects directly to exchanges real-time WebSocket APIs and transparently restarts closed, broken or stale connections (open connections without data being send for specified amount of time).

Provides consolidated real-time market data streaming functionality with options as an array - provides single consolidated real-time data stream for all exchanges specified in options array.

​

import asyncio
import aiohttp
import json
import urllib.parse
​
​
async def run():
    data_types = ["trade", "book_change", "book_snapshot_10_100ms"]
​
    stream_options = [
        {
            "exchange": "bitmex",
            "symbols": ["XBTUSD"],
            "dataTypes": data_types,
        },
        {
            "exchange": "deribit",
            "symbols": ["BTC-PERPETUAL"],
            "dataTypes": data_types,
        },
    ]
​
    options = urllib.parse.quote_plus(json.dumps(stream_options))
​
    URL = f"ws://localhost:8001/ws-stream-normalized?options={options}"
    # real-time normalized data for two exchanges via single connection
    async with aiohttp.ClientSession() as session:
        async with session.ws_connect(URL) as websocket:
​
            async for msg in websocket:
                print(msg.data)
​
​
asyncio.run(run())