exness-data-preprocess v2.0.0

Forex tick data preprocessing with ClickHouse backend. Downloads, stores, and queries tick data from Exness public repository.

Data Inventory (Validated 2025-12-28)

Total: 453.6M ticks, 2.87M OHLC bars in 5.99 GiB

Requirements

• Python 3.11+
• ClickHouse running on localhost:8123

Verify ClickHouse

# Check ClickHouse is running
clickhouse client --query "SELECT version()"

# Check exness database exists
clickhouse client --query "SELECT count() FROM exness.raw_spread_ticks"

Installation

# Using uv
uv pip install exness-data-preprocess

# From source
git clone https://github.com/Eon-Labs/exness-data-preprocess.git
cd exness-data-preprocess
uv sync

Python API

import exness_data_preprocess as edp

# Context manager (recommended)
with edp.ExnessDataProcessor() as processor:
    # Query ticks
    df = processor.query_ticks('EURUSD', 'raw_spread', '2024-01-01', '2024-01-31')

    # Query OHLC
    df = processor.query_ohlc('EURUSD', '1m', '2024-01-01', '2024-01-31')

    # Get coverage info
    cov = processor.get_data_coverage('EURUSD')

    # Download new data (incremental)
    result = processor.update_data('EURUSD', start_date='2022-01-01')

API Methods

Return Models

# UpdateResult
result.months_added        # int: months downloaded
result.raw_ticks_added     # int: raw spread ticks added
result.standard_ticks_added # int: standard ticks added
result.ohlc_bars           # int: OHLC bars generated
result.storage_bytes       # int: storage used

# CoverageInfo
cov.raw_spread_ticks       # int: total raw spread ticks
cov.standard_ticks         # int: total standard ticks
cov.ohlc_bars              # int: total OHLC bars
cov.earliest_date          # str: earliest timestamp
cov.latest_date            # str: latest timestamp
cov.storage_bytes          # int: storage used

Direct SQL Access

# Count ticks
clickhouse client --query "SELECT count() FROM exness.raw_spread_ticks WHERE instrument='EURUSD'"

# Query tick data
clickhouse client --query "
SELECT timestamp, bid, ask
FROM exness.raw_spread_ticks
WHERE instrument='EURUSD'
  AND timestamp >= '2024-01-01'
  AND timestamp < '2024-01-02'
LIMIT 10"

# Query OHLC
clickhouse client --query "
SELECT timestamp, open, high, low, close
FROM exness.ohlc_1m
WHERE instrument='EURUSD'
  AND timestamp >= '2024-01-01'
LIMIT 10"

# Resample to 1h
clickhouse client --query "
SELECT
    toStartOfHour(timestamp) AS ts,
    argMin(open, timestamp) AS open,
    max(high) AS high,
    min(low) AS low,
    argMax(close, timestamp) AS close
FROM exness.ohlc_1m
WHERE instrument='EURUSD' AND timestamp >= '2024-01-01'
GROUP BY ts
ORDER BY ts"

Schema

Tick Tables

exness.raw_spread_ticks, exness.standard_ticks:

OHLC Table

exness.ohlc_1m (27 columns):

Engine: ReplacingMergeTree (automatic deduplication)

Partition: toYYYYMM(timestamp)

Order: (instrument, timestamp)

Data Source

• URL: https://ticks.ex2archive.com/
• Format: Monthly ZIP files with CSV tick data
• Variants: Raw_Spread (zero spreads) + Standard (market spreads)
• Precision: Microsecond timestamps

Development

# Setup
uv sync --dev

# Test
uv run pytest

# Lint
uv run ruff check --fix .

# Type check
uv run mypy src/

Documentation

• ClickHouse User Guide - Detailed ClickHouse usage
• Database Schema - 27-column OHLC specification
• Module Architecture - 13 modules with SLOs
• ADR: ClickHouse Migration - Architecture decision

License

MIT