Getting Started

Overview

Tardis-machine is a locally runnable server with built-in data caching that uses Tardis.dev HTTP API under the hood. It provides both tick-level historical and consolidated real-time cryptocurrency market data via its HTTP and WebSocket APIs and is available via npm and Docker.

Installation and configuration stay on this page.
Full historical replay docs are on Replaying Historical Data.
Full real-time streaming docs are on Streaming Real-Time Data.
Normalized message schemas and computed data types are on Output Data Types.

Real-time streaming via tardis-machine is available only in normalized format. Exchange-native (raw) real-time data is not provided. If you need raw real-time feeds, connect directly to exchange WebSocket APIs.

Installation

Docker

Pull and run latest version of tardisdev/tardis-machine image:

### running without persistent local cache
docker run -p 8000:8000 -p 8001:8001 -e "TM_API_KEY=YOUR_API_KEY" -d tardisdev/tardis-machine

Tardis-machine server's HTTP endpoints will be available on port 8000 and WebSocket API endpoints on port 8001. Your API key will be passed via environment variable (TM_API_KEY) — simply replace YOUR_API_KEY with the API key you've received via email.

The command above does not use persistent volumes for local caching (each Docker restart will result in losing local data cache). To use ./host-cache-dir as a persistent volume (bind mount) cache directory, run:

docker run -v ./host-cache-dir:/.cache -p 8000:8000 -p 8001:8001 -e "TM_API_KEY=YOUR_API_KEY" -d tardisdev/tardis-machine

Since using volumes can cause issues especially on Windows, it's fine to run Docker image without them with the caveat of potentially poor local cache ratio after each container's restart.

Config environment variables

You can set the following environment variables to configure tardis-machine server:

name

default

description

TM_API_KEY

API key for Tardis.dev HTTP API - if not provided only first day of each month of historical data is accessible

TM_PORT

8000

HTTP port on which server will be running, WebSocket port is always this value + 1 (8001 with port set to 8000)

TM_CACHE_DIR

/.cache

path to local dir that will be used as cache location

TM_CLUSTER_MODE

false

will launch cluster of Node.js processes to handle the incoming requests if set to true, by default server runs in single process mode

TM_DEBUG

false

server will print verbose debug logs to stdout if set to true

TM_CLEAR_CACHE

false

server will clear local cache dir on startup if set to true

The local cache grows without limit by default. For large backfills this can consume significant disk space. Use TM_CLEAR_CACHE=true to clear the cache on startup, or set autoCleanup: true in replay options to remove cached data after each replay completes. See the replay options section for autoCleanup concurrency caveats.

Tardis-machine also supports exchange-specific environment variables (exchange credentials, proxy, WebSocket URL overrides, Binance rate limit tuning) — see the Node.js client environment variables for the full list.

npm

Requires Node.js v12+ and git installed.

Install and runtardis-machine server via npx command:

npx tardis-machine --api-key=YOUR_API_KEY

or install globally via npm:

npm install -g tardis-machine

and then run:

tardis-machine --api-key=YOUR_API_KEY

Tardis-machine server's HTTP endpoints will be available on port 8000 and WebSocket API endpoints on port 8001. Your API key will be passed via --api-key config flag — simply replace YOUR_API_KEY with API key you've received via email.

CLI config flags

You can configure tardis-machine server via environment variables as described in Docker section as well.

You can set following CLI config flags when starting tardis-machine server installed via npm:

name

default

description

--api-key

API key for Tardis.dev HTTP API - if not provided only first day of each month of historical data is accessible

--port

8000

HTTP port on which server will be running, WebSocket port is always this value + 1 (8001 with port set to 8000)

--cache-dir

<os.tmpdir>/.tardis-cache

path to local dir that will be used as cache location - if not provided default temp dir for given OS will be used

--cluster-mode

false

will launch cluster of Node.js processes to handle the incoming requests if set to true, by default server runs in single process mode

--debug

false

server will print verbose debug logs to stdout if set to true

--clear-cache

false

server will clear local cache dir on startup is set to true

--help

shows CLI help

--version

shows tardis-machine version number

Quick examples

Historical normalized replay

import asyncio
import aiohttp
import json
import urllib.parse


async def replay_normalized(options):
    encoded_options = urllib.parse.quote_plus(json.dumps(options))
    url = f"http://localhost:8000/replay-normalized?options={encoded_options}"

    async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=0)) as session:
        async with session.get(url) as response:
            async for line in response.content:
                print(json.loads(line))


asyncio.run(
    replay_normalized(
        {
            "exchange": "bitmex",
            "from": "2019-10-01",
            "to": "2019-10-02",
            "symbols": ["XBTUSD"],
            "dataTypes": ["trade", "book_change"],
        }
    )
)

const fetch = require('node-fetch')
const split2 = require('split2')

const serialize = options => encodeURIComponent(JSON.stringify(options))

async function* replayNormalized(options) {
  const url = `http://localhost:8000/replay-normalized?options=${serialize(options)}`
  const response = await fetch(url)
  const lines = response.body.pipe(split2())

  for await (const line of lines) {
    yield JSON.parse(line)
  }
}

async function run() {
  const messages = replayNormalized({
    exchange: 'bitmex',
    from: '2019-10-01',
    to: '2019-10-02',
    symbols: ['XBTUSD'],
    dataTypes: ['trade', 'book_change']
  })

  for await (const message of messages) {
    console.log(message)
  }
}

run()

See Replaying Historical Data for exchange-native replay, normalized replay over HTTP and WebSocket, and full replay options.

Real-time normalized streaming

import asyncio
import aiohttp
import json
import urllib.parse


async def run():
    stream_options = {
        "exchange": "bitmex",
        "symbols": ["XBTUSD"],
        "dataTypes": ["trade", "book_change"],
    }
    options = urllib.parse.quote_plus(json.dumps(stream_options))
    url = f"ws://localhost:8001/ws-stream-normalized?options={options}"

    async with aiohttp.ClientSession() as session:
        async with session.ws_connect(url) as websocket:
            async for message in websocket:
                print(message.data)


asyncio.run(run())

const WebSocket = require('ws')

const options = encodeURIComponent(
  JSON.stringify({
    exchange: 'bitmex',
    symbols: ['XBTUSD'],
    dataTypes: ['trade', 'book_change']
  })
)

const ws = new WebSocket(`ws://localhost:8001/ws-stream-normalized?options=${options}`)

ws.onmessage = message => {
  console.log(message.data)
}

See Streaming Real-Time Data for full stream options, multi-exchange streaming, timeout tuning, and error messages.

PreviousNormalization and Local Processing NextReplaying Historical Data

Last updated 3 minutes ago

hashtagOverview

hashtagInstallation

hashtagDocker

hashtagConfig environment variables

hashtagnpm

hashtagCLI config flags

hashtagQuick examples

hashtagHistorical normalized replay

hashtagReal-time normalized streaming

Overview

Installation

Docker

Config environment variables

npm

CLI config flags

Quick examples

Historical normalized replay

Real-time normalized streaming