Use your own historical market data

This Pro workflow feeds captured market data through the Replay Engine, and optionally records your own captures with Session Recorder. Start from Data, storage, and replay for the shorter operating procedure; this page is the full reference, including the capture-format specification and a working converter.

How replay works

Historical data is re-injected through the same path a live exchange connector uses. The Replay Engine derives from the same BasePluginDataRetriever base class as the Kraken and Binance connectors, publishes through the same HelperOrderBook and HelperTrade singletons, and reaches every study through the same synchronous dispatch. Almost nothing downstream knows about replay: studies, charts and view models consume the feed they always consume. The one exception is an opt-in hook, IPostStartInitializable, that a plugin implements only if its child view models must re-read settings after a routed restart.

The only per-message difference is the provider identity stamped on the data: ProviderID = 98, "Historical File". That single decision is what makes everything on this page work, and it is why the routing step in Driving the replay console exists.

What VisualHFT reads today

Four formats are selectable in the Replay Engine’s format picker. This is the complete list as shipped, not a roadmap.

Format string	What it is	Typical extension	Multi-file support
pcap-TV-NASDAQ-5	NASDAQ TotalView-ITCH 5.0 over MoldUDP64, captured to pcap	.pcap	Yes, one whole directory, concatenated in filename order
FIX 4.2	FIX 4.2 tag=value text logs	any text file	Yes, several files merged by timestamp
VHFT-BIN-1	VisualHFT’s own binary capture format	.vhftbin	Single file
VHFT-SESS-1	A Session Recorder session container	.vhftsess	Single file; the container holds several streams but replay reads only single-stream sessions

One caveat worth stating plainly: a fifth format, VHFT-BIN-2 (the L3 message-by-order variant), is registered in the parser factory but is deliberately not offered in the picker. If you see it referenced elsewhere, it is not user-selectable today.

Everything else routes through one of those four, or through a converter you write. Both routes are covered below.

Feature availability

This page names a lot of surfaces, so here is the entitlement map up front, taken from the shipping catalog rather than from marketing copy. Two things are tracked separately: the tier floor (the plan you must be on) and whether the item is bundled with that plan or a separate paid add-on.

Capability	Tier floor	Bundled or add-on
Replay Engine (Paths A, B and D below)	Pro	Bundled, no extra charge
Session Recorder (.vhftsess capture plus export)	Pro	Separate paid add-on
Event Capture Recorder (.vhftbin trigger capture)	Pro	Separate paid add-on
Microstructure Diagnostics (execution-log TCA)	Pro	Separate paid add-on
LOB Imbalance, VPIN, Market Resilience and its Bias variant, Order-to-Trade Ratio, Time and Sales, Quotes and Level 2 DOM, LOB TimeSeries, Position Manager	Community	Bundled
Market Ratios, Trade-to-Order Ratio, Market Event Stats, Market Latencies Stats, Network Performance Monitoring	Core	Bundled

Read that table as the answer to “what do I need to buy to do the thing in this section”. Confirm current plan and add-on availability before relying on a paid surface. The whole replay half of this page needs Pro and nothing else. Only the recording half (Path C) and the TCA surface are separately purchased.

One mechanical consequence worth knowing: paid add-ons are never packaged in the installer. They are delivered per user as an encrypted assembly after purchase, so a paid tile is simply absent, not greyed out, for an account that has never owned it.

Path A: you already have exchange-native captures

Tier: included with Pro. The Replay Engine is bundled; nothing here is a separate purchase.

If you have NASDAQ TotalView-ITCH 5.0 pcap files, you are done choosing formats. Point the engine at them. This section uses a real capture: a full trading day of NASDAQ TotalView-ITCH for 22 August 2023, 103 pcap files, 37.0 GB.

The configuration

Open the Replay Engine settings from the gear on the replay console, set your symbols, choose pcap-TV-NASDAQ-5, tick Is Directory?, and pick any file inside the folder. VisualHFT stores the folder, enumerates every *.pcap in it, and plays them as one continuous session.

Three things decide whether an ITCH capture actually produces data, and all three are easy to get wrong.

Symbols must match the ITCH Stock field exactly. The comparison is ordinal and case sensitive. AAPL works; AAPL.US typed on its own does not. The normalization form does work, though: write AAPL(AAPL.US) and the parser matches on the raw left side while the dashboard displays the right one.

The capture must include its StockDirectory segment. ITCH announces each instrument with an 'R' message. The parser keeps a dictionary keyed on the 2-byte StockLocate, and admits an entry only when two gates pass: the announcement’s financial-status byte is literally 'N', and the symbol is one you configured. Every later message (adds, executions, cancels, deletes, replaces, trades, crosses) is then looked up by that numeric locate and dropped if it is missing. Until the 'R' for your symbol arrives, the replay runs to completion emitting nothing.

Directory order is filename order, not timestamp order. The engine sorts *.pcap by name and concatenates. That is correct for session logs written as sequential time segments, which is how these captures are normally chunked. It is wrong if you drop unrelated files with clashing names into the same folder.

The trap nobody warns you about: most of a capture day is empty

A full-day capture starts at midnight, and midnight has no activity. If you load the whole directory and press Play, you will watch an empty dashboard for a very long time.

Measuring the real corpus makes the point precisely. Scanning the segments directly:

Segment (file name is UTC)	ITCH messages	StockDirectory	AddOrder	Data window (Eastern)
...T000000.pcap	247,829	0	0	20:00:00 to 20:05:00
...T070000.pcap	295,894	11,328	0	03:02:01 to 03:09:00
...T071000 through ...T075000	0 to 5	0	0	almost nothing
...T133000.pcap	20,288,210	0	8,889,490	09:29:59 to 09:39:59

Two facts fall out of that table, and both matter.

First, the file names are UTC while the ITCH clock inside them is Eastern. The T000000 segment carries data stamped 20:00, because ITCH 5.0 timestamps are nanoseconds since midnight Eastern and the capture host rolled its files on UTC. The engine projects those nanoseconds onto the pcap’s own UTC calendar date without converting between the two zones, which is why the virtual clock does not match the file name. For a segment straddling the UTC and Eastern day boundary the date is off as well, so prefer segments from inside the trading session.

Second, the entire symbol directory arrives in one burst early in the session. In this capture the 'R' blast is in the T070000 segment (03:02 to 03:09 Eastern), which is why that one file is 31.9 MB while each of the five segments after it is under 50 KB. AAPL is announced there at 03:06:44 with StockLocate = 32, MarketCategory = 'Q', FinancialStatusIndicator = 'N', round lot 100. That last field is load-bearing, and the test is blunt: the parser accepts the entry only if the byte is exactly 'N', so anything else, including a blank status, is refused.

How to skip to where the action is

Because directory replay is plain filename-ordered concatenation, you can compose a smaller directory containing only the segments you care about. The rule is simply that the segment carrying the StockDirectory must come first.

For the opening auction, a three-file directory is enough:

ny4-xnas-tvitch-a-20230822T070000.pcap    the StockDirectory blast, 31.9 MB
ny4-xnas-tvitch-a-20230822T132000.pcap    the pre-open ramp, 349 MB
ny4-xnas-tvitch-a-20230822T133000.pcap    the opening auction, 1.85 GB

Copy or hard-link those three into a folder of their own and point the engine at it. The engine registers AAPL from the first segment, paces through the quiet stretch, warms the book on the pre-open ramp, and arrives at the auction.

The one honest caveat: the resting book built during the hours you skipped is not there. Orders that were placed before your first segment and are cancelled or executed inside it produce messages the parser cannot resolve, so it drops them rather than inventing a level. On the open that self-heals within seconds, because the auction floods the book with fresh orders. If you need a fully warm book, include the segments from the session start and accept the extra minutes.

The other lever is the speed control. The gap between the directory blast and the pre-open is over six hours of data time, which at 1000x is about twenty seconds of wall time. Set a high speed to cross the dead zone, then drop back to 1x or 10x when you reach the window you actually want to watch.

Path B: FIX 4.2 text logs

Tier: included with Pro.

Select FIX 4.2 and add your log files individually. Unlike pcap, the text path does a real streaming k-way merge across files: it keeps at most one buffered message per file and repeatedly emits the globally earliest timestamp. That is result-equivalent to a global sort as long as each file is internally chronological, at O(number of files) memory instead of O(total messages).

Snapshot, incremental refresh, new order single, execution report and cancel-replace are all handled on this path.

Two practical constraints: all active files must share one extension (the settings dialog blocks a save that mixes them, and the engine re-validates at play time), and a directory feed is a pcap-only feature. FIX files are selected individually.

Path C: record it inside VisualHFT

Tier note. Both recorders below are separate paid add-ons on top of Pro. Everything else in this guide is bundled with Pro.

If the data you want does not exist yet, capture it from a live connector and replay it later. VisualHFT ships two recorders, and they write different things for different reasons.

Session Recorder does continuous, session-long capture to a single .vhftsess container. It records full order-book frames and trades across one or more provider/symbol/aggregation streams, plus the decimal-exact value of every study metric you select, recorded on change alongside the raw feed. Sessions land in Documents\VisualHFT\Sessions by default. Capture is crash-safe: a write-probe refuses to start when the target is not writable, periodic checkpoint frames keep a torn session partly recoverable, and a disk-full condition finalizes and stops rather than taking the app down.

Event Capture Recorder does trigger-driven forensic capture to .vhftbin (VHFT-BIN-1). It continuously buffers live streams in a ring and, when a TriggerEngine rule fires, writes only the pre/post-event slice. Files land in Documents\VisualHFT\Recordings, named from the template {trigger}_{symbol}_{provider}_{utc:yyyyMMdd-HHmmss}.vhftbin, so the file name tells you which rule produced it and when.

The difference is intent. Session Recorder is “keep everything for this session”. Event Capture Recorder is “keep the sixty seconds around every liquidity shock and throw the rest away”.

The recorder lives on the shell footer next to the replay console, with its own status rail: a state dot, a transport button, a live stream and study count, and a warning chip while it still needs configuration.

Its settings window is where the actual decisions are made, and it is worth walking through because each block maps to something you will care about later:

• Metric Studies. A grid of 26 selectable studies (LOB Imbalance, VPIN, MR, MRB, OTR, TTO, OER, OCR, MOR, QTR, CTR, Adds, Updates, Cancels, Trades, TOB, Crosses, latency and reconnect counters, and so on). Whatever you tick is recorded, decimal-exact and on change, alongside the raw feed. Studies that are not configured are greyed out with a warning icon explaining why; a separate tick widens the list to include studies that emit no metrics at all, which are otherwise hidden entirely.
• Capture Streams. One row per provider, symbol and aggregation level. This is the list that decides what raw book and trade data lands in the file.
• Capture cadences. Snapshot resync defaults to 1000 ms, the durability checkpoint to 5000 ms, and the write watchdog to 3600 s. The first controls how often a full keyframe is forced; the second is what makes a torn file partly recoverable.
• Export. The output folder (defaulting to your Documents VisualHFT\Sessions), a Test Write button that runs the write probe and reports Write probe: OK, and the file-name template {symbol}_{provider}_{utc:yyyyMMdd-HHmmss}.vhftsess with a live preview of the name it will produce.

The dialog will not let you save an unusable configuration. With no stream selected it shows 1 validation error: above a bulleted Add at least one capture stream. and keeps Save disabled. Once armed, stopping finalizes off the UI thread, because the stop path blocks on a drain of the write queue before it writes the footer.

One real constraint on the replay side of .vhftsess: multi-stream sessions are rejected today with an explicit error. The reader replays single-stream sessions only. If you record three streams into one session and try to replay it, you get a clear failure, not silent corruption. Record one stream per session if replay is the goal.

Worth knowing about the on-disk shape: the Session Recorder is snapshot-based. Inside the .vhftsess container every book record is a 0x10 envelope wrapping a VHFT-BIN-1 body (metrics are 0x04, checkpoints 0x20). The recorder stamps that inner body 0x01 on the first frame and every resync interval and 0x02 in between, but unlike a hand-written VHFT-BIN-1 file both carry the complete book: the recorder never emits an incremental delta. That is stated in the export schema sidecar the exporter emits alongside every export, and it means you can reconstruct the book at any timestamp by taking the latest frame at or before it. No replay state machine required.

Path D: everything else, via a converter

Tier: included with Pro. Writing the converter is your side; reading the result is bundled.

This is the path most people will actually use, because most data is not ITCH pcap, not FIX 4.2, and not something VisualHFT recorded. It is a vendor tick store, a kdb table, a Parquet dump, a CSV export, or a proprietary binary feed.

The supported route is to convert it to VHFT-BIN-1. The format is small enough to implement in an afternoon in any language, and the rest of this section is the complete contract plus a working converter.

The file layout

A 256-byte header, then a sequence of frames, then a 64-byte footer. Everything is little-endian. All timestamps are nanoseconds since the Unix epoch, as signed 64-bit integers.

Header (256 bytes), the fields that matter:

Offset	Size	Field
0	8	Magic, ASCII VHFTBIN1
8	2	FormatMajor (uint16), currently 1
10	2	FormatMinor (uint16), currently 0
12	4	HeaderSize (uint32), 256
16	8	CreatedAtUtcNs (int64)
48	4	ProviderID (int32)
52	32	Symbol, UTF-8, null-padded
84	4	AggregationLevel (int32)
88	4	PriceDecimalPlaces (int32)
92	4	SizeDecimalPlaces (int32)
96	8	SymbolMultiplier (float64)
104	4	DepthLevels (int32)

The remaining fields, at offsets 24, 32, 40 and 108 through 195, carry trigger provenance (which rule fired, its pre and post windows, its threshold, and the metric value at fire time) and are only meaningful for Event Capture Recorder output. Leave those zero.

PriceDecimalPlaces and SizeDecimalPlaces are load-bearing. VHFT-BIN-1 carries its own precision rather than assuming one, so prices and sizes are stored as scaled integers and recovered as priceTicks / 10^PriceDecimalPlaces and sizeRaw / 10^SizeDecimalPlaces. Get these wrong and your book renders at the wrong order of magnitude.

Book frames, discriminator 0x01 (full snapshot) or 0x02 (delta):

byte    0        discriminator (0x01 or 0x02)
int64   1..8     tsNs          exchange timestamp
int64   9..16    localTsNs     local receive timestamp
uint16  17..18   nAsks
uint16  19..20   nBids
then nAsks levels, then nBids levels, 16 bytes each:
  int64  priceTicks
  int64  sizeRaw

Asks come first, then bids. On a 0x02 delta frame, a level with sizeRaw == 0 is the wire signal for “this level was removed”.

Trade frames, discriminator 0x03, fixed 35 bytes:

byte    0        0x03
int64   1..8     tsNs
int64   9..16    localTsNs
int64   17..24   priceTicks
int64   25..32   sizeRaw
byte    33       side       0 unknown, 1 buy, 2 sell
byte    34       aggressor  0 unknown, 1 buy, 2 sell

Write 0 for anything your source does not tell you. Do not infer an aggressor the source did not provide; VisualHFT records unknown as unknown deliberately, because a fabricated aggressor flag silently corrupts trade classification downstream.

Footer (64 bytes):

byte    0        0xFF
int64   1..8     FrameCount
uint32  9..12    CRC32
bytes   13..63   reserved, zero

The CRC is classical CRC-32 (IEEE 802.3, polynomial 0xEDB88320, reflected input and output, init and xor-out 0xFFFFFFFF) computed over the file from offset 0 through byte 8 of the footer inclusive. In other words, header plus all frames plus the footer’s discriminator and frame count. The CRC field itself and the reserved tail are excluded.

Three behaviours to design against:

• Bad magic or bad CRC is a hard reject. The parser throws. This is deliberate: silent acceptance of a corrupt capture is worse than a loud failure.
• A missing or partial footer is treated as truncation, not corruption. The parser emits every frame that decoded cleanly, reports Truncated, and the app shows a warning. Crash-interrupted captures stay useful.
• A full snapshot must precede any delta. The parser treats a delta arriving before its anchoring snapshot as a corruption signal. Start every file with a 0x01.

A working converter

Here is a complete converter, in plain Python with no dependencies, that turns two ordinary CSVs into a playable capture. The book CSV needs ts_ns_utc, local_ts_ns, side, level_index, price, size; the trade CSV needs ts_ns_utc, local_ts_ns, price, size, side, aggressor.

#!/usr/bin/env python3
"""csv_to_vhftbin.py - convert order-book and trade CSVs into a VHFT-BIN-1 capture."""
import argparse, csv, struct, sys, time, zlib
from collections import defaultdict

MAGIC, HEADER_SIZE = b"VHFTBIN1", 256
SNAPSHOT_FULL, TRADE, FOOTER = 0x01, 0x03, 0xFF
SIDE = {"unknown": 0, "buy": 1, "sell": 2}


def build_header(symbol, provider_id, aggregation, price_dp, size_dp, depth):
    h = bytearray(HEADER_SIZE)
    h[0:8] = MAGIC
    struct.pack_into("<H", h, 8, 1)                         # FormatMajor
    struct.pack_into("<H", h, 10, 0)                        # FormatMinor
    struct.pack_into("<I", h, 12, HEADER_SIZE)              # HeaderSize
    struct.pack_into("<q", h, 16, int(time.time() * 1e9))   # CreatedAtUtcNs
    struct.pack_into("<i", h, 48, provider_id)              # ProviderID
    sym = symbol.encode("utf-8")[:32]                       # 32-byte UTF-8 slot
    h[52:52 + len(sym)] = sym
    struct.pack_into("<i", h, 84, aggregation)              # AggregationLevel
    struct.pack_into("<i", h, 88, price_dp)                 # PriceDecimalPlaces
    struct.pack_into("<i", h, 92, size_dp)                  # SizeDecimalPlaces
    struct.pack_into("<d", h, 96, float(10 ** price_dp))    # SymbolMultiplier
    struct.pack_into("<i", h, 104, depth)                   # DepthLevels
    return bytes(h)


def book_frame(ts, lts, asks, bids):
    out = bytearray([SNAPSHOT_FULL])
    out += struct.pack("<qqHH", ts, lts, len(asks), len(bids))
    for px, sz in asks + bids:
        out += struct.pack("<qq", px, sz)
    return bytes(out)


def trade_frame(ts, lts, px, sz, side, aggressor):
    return struct.pack("<BqqqqBB", TRADE, ts, lts, px, sz, side, aggressor)


def read_book(path, pscale, sscale):
    frames = defaultdict(lambda: {"ask": [], "bid": []})
    with open(path, newline="", encoding="utf-8-sig") as f:
        for row in csv.DictReader(f):
            key = (int(row["ts_ns_utc"]), int(row["local_ts_ns"]))
            side = "bid" if row["side"].strip().lower() in ("bid", "1") else "ask"
            frames[key][side].append((int(round(float(row["price"]) * pscale)),
                                      int(round(float(row["size"]) * sscale)),
                                      int(row["level_index"])))
    out = []
    for (ts, lts), s in frames.items():
        pick = lambda rows: [(p, z) for p, z, _ in sorted(rows, key=lambda r: r[2])]
        out.append((ts, lts, pick(s["ask"]), pick(s["bid"])))
    out.sort(key=lambda r: r[0])
    return out


def read_trades(path, pscale, sscale):
    out = []
    with open(path, newline="", encoding="utf-8-sig") as f:
        for r in csv.DictReader(f):
            out.append((int(r["ts_ns_utc"]), int(r["local_ts_ns"]),
                        int(round(float(r["price"]) * pscale)),
                        int(round(float(r["size"]) * sscale)),
                        SIDE.get(r.get("side", "").strip().lower(), 0),
                        SIDE.get(r.get("aggressor", "").strip().lower(), 0)))
    out.sort(key=lambda r: r[0])
    return out


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--book", required=True)
    ap.add_argument("--trades")
    ap.add_argument("--symbol", required=True)
    ap.add_argument("--out", required=True)
    ap.add_argument("--price-dp", type=int, default=2)
    ap.add_argument("--size-dp", type=int, default=8)
    ap.add_argument("--provider-id", type=int, default=0)
    ap.add_argument("--aggregation", type=int, default=0)
    # Use when the CSV already holds scaled integer ticks rather than decimals.
    ap.add_argument("--raw-ticks", action="store_true")
    a = ap.parse_args()

    pscale = 1 if a.raw_ticks else 10 ** a.price_dp
    sscale = 1 if a.raw_ticks else 10 ** a.size_dp

    books = read_book(a.book, pscale, sscale)
    trades = read_trades(a.trades, pscale, sscale) if a.trades else []
    if not books:
        sys.exit("no book frames parsed")
    depth = max(max(len(x), len(y)) for _, _, x, y in books)

    # One timestamp-ordered stream; book frames win ties so a snapshot always
    # precedes the trades at the same instant.
    stream = [(ts, 0, ("book", lts, ask, bid)) for ts, lts, ask, bid in books]
    stream += [(t[0], 1, ("trade",) + t[1:]) for t in trades]
    stream.sort(key=lambda r: (r[0], r[1]))

    crc, frames = 0, 0
    with open(a.out, "wb") as f:
        header = build_header(a.symbol, a.provider_id, a.aggregation,
                              a.price_dp, a.size_dp, depth)
        f.write(header); crc = zlib.crc32(header, crc)
        for ts, _, rec in stream:
            blob = (book_frame(ts, rec[1], rec[2], rec[3]) if rec[0] == "book"
                    else trade_frame(ts, rec[1], rec[2], rec[3], rec[4], rec[5]))
            f.write(blob); crc = zlib.crc32(blob, crc); frames += 1
        # 0xFF and FrameCount are inside the CRC window; the CRC field is not.
        tail = struct.pack("<Bq", FOOTER, frames)
        crc = zlib.crc32(tail, crc)
        f.write(tail)
        f.write(struct.pack("<I", crc & 0xFFFFFFFF))
        f.write(b"\x00" * 51)

    print("wrote %s: %d frames, depth %d, crc32=0x%08X"
          % (a.out, frames, depth, crc & 0xFFFFFFFF))


if __name__ == "__main__":
    main()

Running it against a real recorded session exported to CSV:

$ python csv_to_vhftbin.py --book book.csv --trades trades.csv \
      --symbol "BTC/USD" --price-dp 1 --size-dp 8 --raw-ticks \
      --provider-id 4 --aggregation 3 --out BTCUSD-20260625.vhftbin

wrote BTCUSD-20260625.vhftbin: 2495 frames, depth 25, crc32=0x8BC0E7C0

That is 1,948 book frames and 547 trades, 1.59 MB on disk, covering 217 seconds of tape.

Loading it

Open the replay settings, set the symbol to match what you wrote into the header, choose VHFT-BIN-1, and add the file. The picker opens unfiltered for this format, so watch what you select.

Then arm exclusive replay mode and press Play.

A few seconds later the converted file is driving the whole dashboard: full depth ladder, depth chart, LOB time series with resting-order bubbles, time and sales, and every study tile computing.

Nothing in that picture knows the data started life as a CSV.

Driving the replay console

Tier: included with Pro.

The replay console is a persistent rail in the shell footer. Collapsed, it shows state, virtual clock, progress, requested and achieved speed, the routing chip, the bookmark button, settings, and a chevron that opens the details flyout.

Expanded, it is the whole control surface in one panel:

Start, current and end timestamps with a progress bar; Play/Pause, Stop and Restart; requested speed against actual speed; the provider mode; and the bookmark list. The red banner is deliberate: while exclusive routing is on, realtime market data is suppressed system-wide, and the panel says so rather than leaving you to discover it.

The settings dialog

Six things are configurable, and the OK button stays disabled with an inline reason until the configuration is structurally valid:

• Provider ID and Provider Name. Leave these at 98 and Historical File. The engine re-forces that reserved identity every time it loads its settings, but it does NOT re-apply it when you press OK, so a value you type here sticks until the next restart while the clone-swap routing still targets 98. The two then disagree and the routed studies receive nothing. (The field’s tooltip still says “default 6”; that copy is stale.)
• Symbols, comma separated, in the venue’s own notation. Parentheses normalize a venue symbol to a canonical one, for example BTCUSDT(BTC/USD).
• Depth Levels, default 25. This is how many levels per side the engine reconstructs, and it costs CPU and memory.
• Format, which applies to the files you add next. Set it before clicking Load File. Note what is actually enforced: the dialog and the engine both check that active files share one extension, never that they share a format, and a mixed set silently replays using the first active file’s parser.
• Is Directory?, which turns the next selection into a whole-folder feed in filename order. Only the pcap path implements this. The checkbox is not disabled for the other formats, so ticking it with a .vhftbin or .vhftsess is allowed and then fails at load rather than being blocked up front.
• The file list, where each row has its own Is Active tick and a Remove button. Unticking a row keeps it in the list but excludes it from the run, which is the quick way to swap between two captures without re-adding files.

Pressing OK saves and restarts the replay feed immediately. The dialog validates structure only. It does not re-check that the files still exist, so a capture that has moved since you configured it fails loudly at Play rather than at save.

Exclusive replay mode, and why nothing appears without it

This is the step people skip, and skipping it is why a replay can show “Playing”, advance the progress bar, and leave every tile empty.

Every study filters incoming events against its own saved provider and symbol, so replay data stamped with provider 98 is silently rejected by a study configured for Kraken. Rather than make you hand-edit every tile, VisualHFT engages the process-wide live mute first, so no live connector can emit into a half-swapped study during the transition, then per plugin clones its settings onto the replay identity, stops it, swaps in the clone, and restarts it. A commit that fails rolls back and releases the mute, so a failure never leaves live stranded dark. On deactivation the originals come back.

The clone is what makes this recoverable. Your original settings object is held by reference for the restore, and the restore path then sweeps the settings dictionary and re-registers the original if a clone leaked in, so replay values do not end up serialized over your real configuration.

If a plugin’s symbol cannot be resolved against the capture, that plugin is simply not routed. It keeps its own identity and receives nothing, rather than being silently pointed at the wrong symbol. The routing chip in the console is where you inspect and fix that.

Transport, and exactly when each control works

The transport is deliberately strict about when each button is live, so nothing is ever enabled-but-inert:

Control	Enabled when
Play / Pause	A session is loaded AND its catalog scan has finished, the same gate as Restart. The icon becomes Restart once the file completes.
Stop	Playing, Paused, Completed or Error. Not in Ready or with no session.
Restart	A session is loaded and its catalog scan has finished.
Mark (add bookmark)	Playing or Paused only.
Jump (to a bookmark)	Exclusive replay mode is ON and a session is loaded (Ready, Playing, Paused or Completed).

Stop is worth one extra sentence, because its interaction with the mute surprises people. A clean Stop, or reaching end of file, with the exclusive toggle still on deliberately keeps the mute engaged. Routing survives transport events, and tiles hold their last replayed state instead of flickering back to live data. The mute is released when you switch the toggle off, on a fault stop, and on plugin teardown including the crash path.

Speed, and the honest readout

Eleven options, from -1000x to 1000x, defaulting to 1x. Positive values run faster than real time. Negative values are slow motion: internally -N maps to a factor of 1/N, so -10x plays at one tenth speed. Pacing is anchored to an absolute wall-clock target per message rather than sleeping the raw inter-message gap, so per-message processing cost is absorbed into the gap instead of accumulating as drift.

The readout next to the selector is the part worth watching. REQ is what you asked for. ACT is what the chain actually achieved, computed as data-time over wall-time, and it turns amber when the achieved factor falls below 80 percent of the requested one. That number is honest, and it is there because the pipeline has real ceilings. Our own instrumentation puts the single-threaded produce path through order-book maintenance at roughly 110,000 messages per second with no consumers attached; with the usual study set wired up the sustained figure is nearer 50,000 to 58,000. Ask for 1000x on a busy tape and you will not get it. The readout tells you so instead of letting you believe otherwise.

Bookmarks

The Mark button captures a bookmark at the current virtual timestamp, including enough book state to restore it, so you can re-run the same forty seconds of a liquidity event as many times as you need.

How the jump resumes depends on the format: pcap and FIX re-open the producer at the stored byte offset, VHFT-BIN-1 reads its body in one pass so the restored virtual clock is the cut point, and .vhftsess does not support seek in this version. Bookmarks persist per capture in a keyed store under %LocalAppData%\VisualHFT\replay-bookmarks, so they survive a restart and follow the capture rather than the session.

The bookmark panel is never silently dead. When Jump is unavailable it says why, in words: “Enable replay mode (toggle) to use bookmarks” when a session is loaded but the toggle is off, and “Load a replay session to use bookmarks” otherwise.

What replay actually shows you

Tier: the Replay Engine is included with Pro. Every study named below is bundled too, from the Community tier upward, except market ratios, trade-to-order ratio, market event statistics and latency statistics, which need Core.

This is the whole point, and it deserves more than a feature list. Replaying a tape is not about watching a progress bar finish. It is about seeing microstructure happen at a speed your eyes can follow, on data where you already know what happened next.

Take the NASDAQ opening auction from Path A. Here is what the tape actually contains at that moment, measured rather than asserted. Scanning the first 300 MB of the T133000 segment covers roughly the first 39 seconds of the session and yields about 4.25 million ITCH messages across every instrument in the feed (the capture’s directory announces 11,328 of them), which is on the order of 109,000 messages per second at the bell. That is the burst rate; averaged over the segment’s full ten minutes it settles to about 34,000 per second. Filtered to AAPL alone, the window from 09:30:00 to 09:30:38 Eastern is 41,013 messages, roughly 1,100 per second:

ITCH message	Count in the first 38 seconds
33,303	Add Order
4,862	Order Delete
1,507	Order Replace
1,011	Order Executed
233	Trade (non-cross)
40	Add Order with MPID attribution
33	Order Executed with price
23	Order Cancel (partial)
1	Cross Trade: the opening auction print

That single 'Q' is the opening cross, and the bytes say so: its cross-type field is 'O', and it prints 587,385 shares at $177.0000 at 09:30:00 Eastern. VisualHFT surfaces it as a cross without distinguishing opening from closing (it parses the cross-type byte but does not act on it), so the timestamp and the type field are how you know which one you are looking at. Everything around it is the queue forming, and that is the thing you can finally watch.

Here is a single frame of that, because several things in it are worth pointing at:

AAPL is quoted 176.65 by 176.67 with 793 and 500 at the touch. The Position Manager’s session date reads 8/22/2023, not today, because the virtual clock is the capture’s clock. Time and Sales is printing real AAPL executions. LOB Imbalance is 0.2, VPIN 0.67, Market Resilience 0.6, order-to-trade ratio 77.0.

And look at the transport: REQ 1000x with ACT ≈ 7.2x at 97.5 percent progress. That is the honest readout doing its job. The request was a thousand times real time; the chain delivered about seven, because the opening auction is the densest part of the tape and book maintenance is the ceiling. Nothing lied about it, and nothing silently dropped data to keep up.

Slow the same window down and specific behaviours become legible:

• Queue build-up before the bell. The depth chart fills from the touch outward as orders stack at the auction price. Because the LOB time series plots resting size as bubble area against price and time, a level that is accumulating looks visibly different from one that is being lifted.
• Book skew. LOB Imbalance is a single number on a tile, but during the ramp it swings hard and then reverts. Watching the tile and the depth chart together is what tells you whether the skew was real depth or a thin quote that evaporated on the first execution.
• The cancel storm. Order Delete and Order Replace together are over 6,300 of those 41,013 messages. Order-to-Trade Ratio and Market Event Stats put numbers on that in real time, which is exactly the signal that separates a genuine book from a quoted one.
• The print itself. Time and Sales shows the cross landing, and every trade-driven study steps at the same instant.

Everything on the dashboard participates, with no configuration beyond the routing step: the order book view, depth chart, LOB time series, and time and sales all render from replayed data, and the study plugins compute normally, including LOB Imbalance, VPIN, Market Resilience and its bias variant, order-to-trade and trade-to-order ratios, market event statistics, market ratios, and latency statistics.

Time-dependent studies stay correct at speed because the virtual clock is advanced to each frame’s timestamp before that frame is dispatched, never after. VPIN’s volume buckets, Market Resilience’s shock timeouts, and every latency statistic read data time. This is tested with a rule that deliberately forbids introducing a mock clock seam, because the whole point is that studies read the same clock in replay as they do live. The Position Manager’s session date follows the capture too, so a replay of 25 June shows 25 June, not today.

TriggerEngine rules fire on replayed data as well. A routed study emits under the replay identity, and rules match on the plugin and the metric rather than on provider or symbol, so your existing alerting evaluates against the historical tape unchanged. That closes the loop: you can validate a rule against last Tuesday’s event before arming it on the live feed.

Getting the data back out

Tier: you need the Session Recorder add-on to RECORD the sessions, but not to export them. The exporter ships inside the bundled Replay Engine, so a .vhftsess someone hands you exports fine on plain Pro.

Replay is one direction. The other direction matters just as much for research, and Session Recorder sessions export to CSV or Parquet in three shapes.

Long is a faithful event stream, and it produces three tables plus a schema sidecar: metric values, book levels, and trades. Wide gives one row per book frame with every metric column forward-filled to that frame, plus a per-metric as-of age in milliseconds so you always know how stale a carried-forward value is. WideBook is Wide plus the book itself flattened into columns, bid1..bidN and ask1..askN price and size per frame. That last one is the DeepLOB-style panel: full book state and the feature matrix in one record per timestamp.

Export runs in the app with progress and cancel, and there is a standalone CLI for batch work. Both --format and --shape take comma-separated lists, and every format-by-shape combination is written:

Tools.VhftExport <session.vhftsess> --format csv,parquet --shape long,wide,widebook --out <dir> [--stream <key>] [--depth <N>]

Running that against a real recorded KuCoin BTC/USD session produced a long export split across three tables (171,894 metric observations across 2,173 distinct timestamps, plus the book and trade tables) and a 45-column WideBook panel of 1,948 rows, one per book frame, whose header reads:

book_ts_ns_utc,bid1_px,bid1_sz,ask1_px,ask1_sz, ... ,bid10_px,bid10_sz,ask10_px,ask10_sz,
value_LOB_Imbalance,asofage_ms_LOB_Imbalance,value_VPIN,asofage_ms_VPIN

That session happened to be one whose recorder was interrupted before it wrote its footer. The exporter detected the truncation and exported the valid prefix, marking every output .partial in its file name. That suffix is the signal you get: the exporter computes the torn byte offset internally but nothing currently surfaces it. A crashed capture is still a usable dataset. The _long_book.csv and _long_trade.csv tables from a Long export, not the WideBook panel shown above, are the input to the Path D converter, which is a convenient way to check your converter against data VisualHFT itself produced. One caveat: the recorder does not persist an aggressor flag, so that column exports as unknown throughout.

Every export ships a JSON schema sidecar next to the data describing column types, fill methods, per-stream precision, and the conventions the reader needs. Two of those conventions are worth repeating because they are the ones that trip people up: metric values are last-observation-carried-forward and never recomputed at export, and when a stream’s precision was not observable the sidecar sets PrecisionUnknown and the price and size columns hold raw integer ticks rather than decimals. The sidecar tells you which case you are in, per stream, so a downstream pipeline can branch on it instead of guessing.

Limits

A guide that only lists capabilities is a brochure. These are the constraints as they stand today:

• Directory replay is pcap-only. VHFT-BIN-1 and VHFT-SESS-1 replay a single file; FIX takes several files but selected individually, not as a folder.
• Multi-stream .vhftsess replay is not supported. It fails loudly. Record one stream per session if you intend to replay it.
• There is no symbol catalog scanner for pcap. VHFT-BIN-1 and VHFT-SESS-1 read their symbols from the header instantly, FIX pays a full-body scan for tag 55, and pcap returns an empty catalog so Play is not permanently gated. That is why typing the correct symbol matters most on the ITCH path: nothing will discover it for you.
• ITCH captures need their StockDirectory segment, and it must sort first in the directory.
• On a single-symbol ITCH replay, decoding is the constraint, not the order book. Per message our drill puts ITCH decoding at roughly 0.87 microseconds and 173 bytes, against 2.7 to 3.1 microseconds and 464 bytes for order-book maintenance. Book maintenance is the more expensive operation, but the two are not paid on the same messages: the parser decodes every instrument on the tape and only then discards the symbols you did not ask for. Filter a 11,328-instrument feed down to one name and you decode about a hundred messages for every one that reaches the book, so aggregate decode cost dominates. That is why the auction above returned an achieved 7.2x against a requested 1000x.
• Exclusive replay mode mutes live feeds process-wide while it is engaged. That is the intended isolation, and a clean Stop deliberately keeps it engaged so routing survives transport events.

What you need

The Replay Engine is bundled with the Pro tier at no additional cost, and it covers Paths A, B and D in full. Session Recorder and Event Capture Recorder are separate paid add-ons on top of Pro, as is Microstructure Diagnostics, the transaction-cost-analysis surface that carries its own ingestion layer for execution logs (FIX, CSV, and JSON or JSONL, with a field-mapping dialog for arbitrary column names and venue presets for common schemas). Microstructure Diagnostics is a different ingest path from the Replay Engine and is documented separately.

If you have a tape you have never been able to inspect properly, the shortest path is: convert it to VHFT-BIN-1 with the script above, load it, engage exclusive mode, and press Play. The dashboard does not know the difference.

Related: Data, storage, and replay for the shorter operating procedure, and Advanced analysis guides for controlled GPU, latency and cross-venue investigations.