TurboQuant

Extreme compression for numerical data. 10-16x smaller. Inspired by Google's TurboQuant research.

Install

pip install turboquant  # coming soon — for now, copy turboquant.py

Usage

from turboquant import TurboQuant

# Your data — prices, sensors, metrics, anything numerical
data = [99.5, 100.2, 100.8, 101.1, 100.7, ...]

# Compress
tq = TurboQuant(bits=4)           # 4-bit = ~10x compression
compressed = tq.compress(data)     # returns bytes

# Decompress
restored = tq.decompress(compressed)

# Check
print(f"Size: {len(data)*8:,}B -> {len(compressed):,}B ({len(data)*8/len(compressed):.0f}x)")
print(f"Error: {tq.relative_error(data, restored):.2f}%")

Compression vs Accuracy

How it works

1. Delta encoding — stores differences between consecutive values (makes data near-zero-mean)
2. Randomized Hadamard rotation — spreads variance evenly across dimensions
3. Uniform quantization — maps rotated values to N-bit integers
4. Bit packing — stores efficiently

The rotation is the key: most data has uneven variance across dimensions. Rotation makes it uniform, so simple quantization works without per-block scaling overhead.

When to use it

Good for:

• Storing months/years of time-series in memory instead of days
• Compressing database columns (metrics, logs, sensor readings)
• Reducing Redis/cache memory usage
• Shipping less data over the network (IoT, microservices)
• Approximate nearest-neighbor search on compressed vectors

Not for:

• Financial trading (use exact data for execution)
• Scientific computation requiring full precision
• Data where every decimal matters

Honest benchmarks

On correlated data (prices, sensors):

• 4-bit: 10.6x compression, ~3% MAPE
• 3-bit: 12.7x compression, ~2% MAPE

On uncorrelated data (random):

• 4-bit: 10.6x compression, ~6% MAPE
• Use compress(data, use_delta=False) for uncorrelated data

API

# Full control
tq = TurboQuant(bits=4, block_size=32, seed=42)
compressed = tq.compress(data, use_delta=True)
restored = tq.decompress(compressed)
ratio = tq.ratio(data, compressed)
error = tq.error(data, restored)       # Mean absolute error
pct_error = tq.relative_error(data, restored)  # MAPE %

# One-liners
from turboquant import compress, decompress
compressed = compress(data, bits=4)
restored = decompress(compressed)

# Files
from turboquant import compress_file, decompress_file
compress_file("data.csv", "data.tq", bits=4)
decompress_file("data.tq", "data_restored.json")

License

MIT