Tardis-Machine

TODO: description

This is in progress draft of tardis-machine v2 documentation. Please contact us in case of any questions. We're fully aware documentation is not yet complete, working hard to finish it as soon as possible.

Introduction

Tardis-machine is a fast, locally runnable server with built-in local caching, that uses lower level HTTP API under the hood. It can be installed via NPM or run as a Docker Container (image available on Docker hub).

Locally cached data is stored on disk in compressed form (GZIP) and decompressed on demand when reading the data.

Tardis-machine provides on-demand tick-level market data replay API from any moment in time in exchange's WebSocket data format both via streaming HTTP and WebSocket endpoints. In many cases existing exchanges WebSocket clients can be used to consume historical message stream as if it was real-time one.

TODO : list of features

// transparent open source real-time data , hosted locally

See historical data details page to get detailed information about historical market data available for each exchange.

Installation

via Docker

Run command below in order to pull and run latest version of tardisdev/tardis-machine image. Tardis-machine server will be available on host via 8000 port. Your API key will be passed via env variable (TM_API_KEY) — simply replace <YOUR_API_KEY> with API key you've received.

### running without persitent local cache
docker run -p 8000:8000 -e "TM_API_KEY=<YOUR_API_KEY>" -d tardisdev/tardis-machine

Command above does not use persistent volumes for local caching (each docker restart will result in loosing local cache). In order to use for example./host-cache-dir as persistent volume (bind mount) cache directory, run:

docker run -v ./host-cache-dir:/.cache -p 8000:8000 -e "TM_API_KEY=<YOUR_API_KEY>" -d tardisdev/tardis-machine

Since using volumes can cause issues especially on Windows, it's perfectly fine to run docker without them, with the caveat of potentially poor local cache ratio after container restart.

via npx

Requires Node.js v12+ installed.

Run command below to install and start tardis-machine server running on port 8000 by default (port can be changed --port CLI flag)

npx tardis-machine --api-key=<YOUR_API_KEY>

If you'd like to try server on sample data (first full day of each month) run the command without --api-key option.

npx tardis-machine

Run npx tardis-machine --help to see all available CLI options (setting custom cache directory, enabling debug logs etc.)

via npm

Requires Node.js v12+ installed.

npm install -g tardis-machine

Run command below to start tardis-machine server running on port 8000 by default (port can be changed --port CLI flag)

tardis-machine --api-key=<YOUR_API_KEY>

If you'd like to try server on sample data (first full day of each month) run the command without --api-key option.

tardis-machine

Run tardis-machine --help to see all available CLI options (setting custom cache directory, enabling debug logs etc.)

Debugging and logging

In order to see detailed logs & debug messages when using Docker, run:

docker run -v ./host-cache-dir:/.cache -p 8000:8000 -e "TM_API_KEY=YOUR_API_KEY" -e "DEBUG=tardis-dev:machine*" -d tardisdev/tardis-machine

Setting DEBUG=tardis-dev:machine* env variable will cause to printing debug logs to stdout.

When running via npm installation or npx you can pass --debug flag when starting CLI or set DEBUG environment variable to tardis-dev:machine*.

npx tardis-machine --debug
# or when installed globally via npm
tardis-machine --debug

HTTP endpoints

Base HTTP API Endpoint: http://localhost:8000 (assuming default options when starting tardis-machine)

GET /replay?options={options}

Replays historical market data messages for given replay options query param in exchange native format. Historical raw market data is being fetched efficiently (in parallel) from the HTTP API and cached locally by tardis-machine.

Accepts required options query string param in url encoded JSON format.

Returns messages in NDJSON format (new line delimited JSON) - each line is JSON with message in exchange native format.

replay options

name

type

default

exchange

string

-

requested exchange id - one of allowed values

filters

{channel:string, symbols?: string[]}[]

[]

optional filters of requested historical data feed - use getExchangeDetails function to get allowed channels and symbols ids for requested exchange

from

string

-

replay period start date (UTC) in a format recognized by the Date.parse(), e.g., 2019-04-01

to

string

-

replay period end date (UTC) in a format recognized by the Date.parse(), e.g., 2019-04-02

skipDecoding

boolean | undefined

undefined

when set to true returns messages as buffers instead of decoding them to objects

withDisconnects

boolean | undefined

undefined

when set to true returns message with value undefined for events when connection that was recording the historical data was closed

apiKey

string | undefined

undefined

API key for tardis.dev API - if not provided only first day of each month of historical data is accessible. It can also be set via init function for all replay calls.

TODO: response format

TODO: example response

GET /replay-normalized?options={options}

Replay normalized options JSON string needs to be url encoded when provided via query string

Replay normalized options JSON

{
exchange: string
symbols?: string[] | undefined
from: string
to: string
withDisconnectMessages?: boolean | undefined
apiKey?: boolean | undefined
dataTypes: string[]
}
// or array of options

Example request

const serialize = options => {
return encodeURIComponent(JSON.stringify(options))
}
const options = {
exchange: 'bitmex',
symbols: ['ETHUSD'],
from: '2019-06-01',
to: '2019-06-02',
dataTypes: ['trade', 'book_change', 'derivative_ticker']
}
fetch(`http://localhost:8000/replay-normalized?options=${serialize(options)}`)

WebSocket Endpoints

Base WebSocket API Endpoint: ws://localhost:8000 (assuming default options when starting tardis-machine)

WS /ws-replay

...

WS /ws-replay-normalized

...

WS /ws-stream-normalized

...

Normalized data types

Accepted data types strings

  • trade

  • book_change

  • derivative_ticker

  • quote

  • quote_{suffix}

  • book_snapshot_{numberof_top_book_levels}_{snapshot_interval}{suffix}

    • Available suffixes:

      • ms

      • s

      • m

  • trade_bar_{aggregation_interval}{suffix}

    • Available suffixes:

      • ms

      • s

      • m

      • ticks

      • vol

trade

{
type: 'trade'
symbol: string
exchange: string
id: string | undefined
price: number
amount: number
side: 'buy' | 'sell' | 'unknown' // liquidity taker side (aggressor)
timestamp: string
localTimestamp: string
}

sample normalized

trade JSON message

{
"type": "trade",
"symbol": "XBTUSD",
"exchange": "bitmex",
"id": "282a0445-0e3a-abeb-f403-11003204ea1b",
"price": 7996,
"amount": 50,
"side": "sell",
"timestamp": "2019-10-23T10:32:49.669Z",
"localTimestamp": "2019-10-23T10:32:49.740Z"
}

book_change

{
type: 'book_change'
symbol: string
exchange: string
isSnapshot: boolean
bids: { price: number; amount: number }[]
asks: { price: number; amount: number }[]
timestamp: string
localTimestamp: string
}

sample normalized book_change JSON message

{
"type": "book_change",
"symbol": "XBTUSD",
"exchange": "bitmex",
"isSnapshot": false,
"bids": [],
"asks": [
{
"price": 7985,
"amount": 283318
}
],
"timestamp": "2019-10-23T11:29:53.469Z",
"localTimestamp": "2019-10-23T11:29:53.469Z"
}

derivative_ticker

{
type: 'derivative_ticker'
symbol: string
exchange: string
lastPrice: number | undefined
openInterest: number | undefined
fundingRate: number | undefined
indexPrice: number | undefined
markPrice: number | undefined
timestamp: string
localTimestamp: string
}

sample normalized derivative_ticker JSON message

{
"type": "derivative_ticker",
"symbol": "BTC-PERPETUAL",
"exchange": "deribit",
"lastPrice": 7987.5,
"openInterest": 84129491,
"fundingRate": -0.00001568,
"indexPrice": 7989.28,
"markPrice": 7987.56,
"timestamp": "2019-10-23T11:34:29.302Z",
"localTimestamp": "2019-10-23T11:34:29.416Z"
}

trade_bar

{
type: 'trade_bar'
symbol: string
exchange: string
name: string
interval: number
kind: 'time' | 'volume' | 'tick'
open: number
high: number
low: number
close: number
volume: number
buyVolume: number
sellVolume: number
trades: number
vwap: number
openTimestamp: Date
closeTimestamp: Date
timestamp: Date
localTimestamp: Date
}

sample normalized trade_bar JSON message

{
"type": "trade_bar",
"symbol": "XBTUSD",
"exchange": "bitmex",
"name": "trade_bar_10000ms",
"interval": 10000,
"kind": "time",
"open": 7623.5,
"high": 7623.5,
"low": 7623,
"close": 7623.5,
"volume": 37034,
"buyVolume": 24244,
"sellVolume": 12790,
"trades": 9,
"vwap": 7623.327320840309,
"openTimestamp": "2019-10-25T13:11:31.574Z",
"closeTimestamp": "2019-10-25T13:11:39.212Z",
"localTimestamp": "2019-10-25T13:11:40.369Z",
"timestamp": "2019-10-25T13:11:40.000Z"
}

book_snapshot

{
type: 'book_snapshot'
symbol: string
exchange: string
name: string
depth: number
interval: number
bids: { price: number; amount: number }[]
asks: { price: number; amount: number }[]
timestamp: Date
localTimestamp: Date
}

sample normalized book_snapshot JSON message

{
"type": "book_snapshot",
"symbol": "XBTUSD",
"exchange": "bitmex",
"name": "book_snapshot_2_50ms",
"depth": 2,
"interval": 50,
"bids": [
{
"price": 7633.5,
"amount": 1906067
},
{
"price": 7633,
"amount": 65319
}
],
"asks": [
{
"price": 7634,
"amount": 1467849
},
{
"price": 7634.5,
"amount": 67939
}
],
"timestamp": "2019-10-25T13:39:46.950Z",
"localTimestamp": "2019-10-25T13:39:46.961Z"
}

Examples

Example below illustrates tardis-machine executing inline from code and official Node.js BitMEX WebSocket client communicating with it and consuming historical data as if it was real-time stream - it can as well run as CLI installed via NPM and as Docker container.

tardis-machine example working with official Node.js BitMEX WebSocket client

Try example above using run button - first change Node version to v12 in dropdown on the left

tardis-machine example working together with ccxws lib

Try example above using run button - first change Node version to v12 in dropdown on the left