Tardis-Machine

Introduction

Tardis-machine is a fast, locally runnable server with built-in local caching, that uses lower level HTTP API under the hood. It can be installed via NPM or run as a Docker Container (image available on Docker hub).

Locally cached data is stored on disk in compressed form (GZIP) and decompressed on demand when reading the data.

Tardis-machine provides on-demand tick-level market data replay API from any moment in time in exchange's WebSocket data format both via streaming HTTP and WebSocket endpoints. In many cases existing exchanges WebSocket clients can be used to consume historical message stream as if it was real-time one.

Installation

via Docker

Run command below in order to pull and run latest version of tardisdev/tardis-machine image. Tardis-machine server will be available on host via 8000 port. Your API key will be passed via env variable (TM_API_KEY) — simply replace <YOUR_API_KEY> with API key you've received.

### running without persitent local cache
docker run -p 8000:8000 -e "TM_API-KEY=<YOUR_API_KEY>" -d tardisdev/tardis-machine

Command above does not use persistent volumes for local caching (each docker restart will result in loosing local cache). In order to use for example./host-cache-dir as persistent volume (bind mount) cache directory, run:

docker run -v ./host-cache-dir:/.cache -p 8000:8000 -e "TM_API-KEY=<YOUR_API_KEY>" -d tardisdev/tardis-machine

Since using volumes can cause issues especially on Windows, it's perfectly fine to run docker without them, with the caveat of potentially poor local cache ratio after container restart.

● via npx

Requires Node.js v12+ installed.

Run command below to install and start tardis-machine server running on port 8000 by default (port can be changed --port CLI flag)

npx tardis-machine --api-key=<YOUR_API_KEY>

If you'd like to try server on sample data (first full day of each month) run the command without --api-key option.

npx tardis-machine

Run npx tardis-machine --help to see all available CLI options (setting custom cache directory, enabling debug logs etc.)

● via npm

Requires Node.js v12+ installed.

npm install -g tardis-machine

Run command below to start tardis-machine server running on port 8000 by default (port can be changed --port CLI flag)

tardis-machine --api-key=<YOUR_API_KEY>

If you'd like to try server on sample data (first full day of each month) run the command without --api-key option.

tardis-machine

Run tardis-machine --help to see all available CLI options (setting custom cache directory, enabling debug logs etc.)

Debugging and logging

In order to see detailed logs & debug messages when using Docker, run:

docker run -v ./host-cache-dir:/.cache -p 8000:8000 -e "TM_API-KEY=YOUR_API_KEY" -e="DEBUG=tardis-dev:machine*" -d tardisdev/tardis-machine

Setting DEBUG=tardis-dev:machine* env variable will cause to printing debug logs to stdout.

When running via npm installation or npx you can pass --debug flag when starting CLI or set DEBUG environment variable to tardis-dev:machine*.

npx tardis-machine --debug
# or when installed globally via npm
tardis-machine --debug

Available HTTP API endpoints

get
/replay

http://localhost:8000/replay?options={options}
Request
Response
Path Parameters
optional
string
200: OK

get
/replay-normalized

http://localhost:8000/replay-normalized?options={options}
Request
Response
Query Parameters
options
required
string
Replay normalized options as url encoded JSON string
200: OK

Replay normalized options JSON string needs to be url encoded when provided via query string

Replay normalized options JSON

{
exchange: string
symbols?: string[] | undefined
from: string
to: string
withDisconnectMessages?: boolean | undefined
apiKey?: boolean | undefined
dataTypes: string[]
}
// or array of options

Example request

const serialize = options => {
return encodeURIComponent(JSON.stringify(options))
}
const options = {
exchange: 'bitmex',
symbols: ['ETHUSD'],
from: '2019-06-01',
to: '2019-06-02',
dataTypes: ['trade', 'book_change', 'derivative_ticker']
}
fetch(`http://localhost:8000/replay-normalized?options=${serialize(options)}`)

Accepted data types strings

  • trade

  • book_change

  • derivative_ticker

  • quote

  • quote_{suffix}

  • book_snapshot_{numberof_top_book_levels}_{snapshot_interval}{suffix}

    • Available suffixes:

      • ms

      • s

      • m

  • trade_bar_{aggregation_interval}{suffix}

    • Available suffixes:

      • ms

      • s

      • m

      • ticks

      • vol

Available WebSocket API endpoints

/ws-replay

...

/ws-replay-normalized

...

/ws-stream-normalized

...

Normalized JSON data format

trade

{
type: 'trade'
symbol: string
exchange: string
id: string | undefined
price: number
amount: number
side: 'buy' | 'sell' | 'unknown' // liquidity taker side (aggressor)
timestamp: string
localTimestamp: string
}

sample normalized

trade JSON message

{
"type": "trade",
"symbol": "XBTUSD",
"exchange": "bitmex",
"id": "282a0445-0e3a-abeb-f403-11003204ea1b",
"price": 7996,
"amount": 50,
"side": "sell",
"timestamp": "2019-10-23T10:32:49.669Z",
"localTimestamp": "2019-10-23T10:32:49.740Z"
}

book_change

{
type: 'book_change'
symbol: string
exchange: string
isSnapshot: boolean
bids: { price: number; amount: number }[]
asks: { price: number; amount: number }[]
timestamp: string
localTimestamp: string
}

sample normalized book_change JSON message

{
"type": "book_change",
"symbol": "XBTUSD",
"exchange": "bitmex",
"isSnapshot": false,
"bids": [],
"asks": [
{
"price": 7985,
"amount": 283318
}
],
"timestamp": "2019-10-23T11:29:53.469Z",
"localTimestamp": "2019-10-23T11:29:53.469Z"
}

derivative_ticker

{
type: 'derivative_ticker'
symbol: string
exchange: string
lastPrice: number | undefined
openInterest: number | undefined
fundingRate: number | undefined
indexPrice: number | undefined
markPrice: number | undefined
timestamp: string
localTimestamp: string
}

sample normalized derivative_ticker JSON message

{
"type": "derivative_ticker",
"symbol": "BTC-PERPETUAL",
"exchange": "deribit",
"lastPrice": 7987.5,
"openInterest": 84129491,
"fundingRate": -0.00001568,
"indexPrice": 7989.28,
"markPrice": 7987.56,
"timestamp": "2019-10-23T11:34:29.302Z",
"localTimestamp": "2019-10-23T11:34:29.416Z"
}

trade_bar

{
type: 'trade_bar'
symbol: string
exchange: string
name: string
interval: number
kind: 'time' | 'volume' | 'tick'
open: number
high: number
low: number
close: number
volume: number
buyVolume: number
sellVolume: number
trades: number
vwap: number
openTimestamp: Date
closeTimestamp: Date
timestamp: Date
localTimestamp: Date
}

sample normalized trade_bar JSON message

{
"type": "trade_bar",
"symbol": "XBTUSD",
"exchange": "bitmex",
"name": "trade_bar_10000ms",
"interval": 10000,
"kind": "time",
"open": 7623.5,
"high": 7623.5,
"low": 7623,
"close": 7623.5,
"volume": 37034,
"buyVolume": 24244,
"sellVolume": 12790,
"trades": 9,
"vwap": 7623.327320840309,
"openTimestamp": "2019-10-25T13:11:31.574Z",
"closeTimestamp": "2019-10-25T13:11:39.212Z",
"localTimestamp": "2019-10-25T13:11:40.369Z",
"timestamp": "2019-10-25T13:11:40.000Z"
}

book_snapshot

{
type: 'book_snapshot'
symbol: string
exchange: string
name: string
depth: number
interval: number
bids: { price: number; amount: number }[]
asks: { price: number; amount: number }[]
timestamp: Date
localTimestamp: Date
}

sample normalized book_snapshot JSON message

{
"type": "book_snapshot",
"symbol": "XBTUSD",
"exchange": "bitmex",
"name": "book_snapshot_2_50ms",
"depth": 2,
"interval": 50,
"bids": [
{
"price": 7633.5,
"amount": 1906067
},
{
"price": 7633,
"amount": 65319
}
],
"asks": [
{
"price": 7634,
"amount": 1467849
},
{
"price": 7634.5,
"amount": 67939
}
],
"timestamp": "2019-10-25T13:39:46.950Z",
"localTimestamp": "2019-10-25T13:39:46.961Z"
}

Examples

Example below illustrates tardis-machine executing inline from code and official Node.js BitMEX WebSocket client communicating with it and consuming historical data as if it was real-time stream - it can as well run as CLI installed via NPM and as Docker container.

tardis-machine example working with official Node.js BitMEX WebSocket client

Try example above using run button - first change Node version to v12 in dropdown on the left

tardis-machine example working together with ccxws lib

Try example above using run button - first change Node version to v12 in dropdown on the left