ShiftEngine 🚀

ShiftEngine is a high-performance quantitative filtering tool designed to detect market regime shifts (Bull vs. Bear) in real-time. By leveraging recursive Bayesian updates in log-space, it acts as an event-driven Hidden Markov Model (HMM) capable of processing millions of data points with microsecond latency.

---

⚡ Performance Benchmarks

Designed for low-latency trading environments, ShiftEngine adheres to strict quantitative engineering standards:

• Mean Latency: ~1.8 - 2.2 microseconds ($\mu s$) per tick.
• Throughput: >450,000 updates per second.
• Memory Complexity: Strictly $O(1)$. The engine does not store history internally, allowing it to run indefinitely on streaming data without memory leaks.
• Numerical Stability: Operations are performed entirely in log-space using the Log-Sum-Exp trick to prevent floating-point underflow.

---

🛠️ Advanced Validation Suite

ShiftEngine goes beyond historical backtesting by utilizing a dual-layer robustness framework:

1. Grid Search Sensitivity Analysis

We systematically sweep across parameter sets (Transition Matrix stickiness and Priors) to find the "Stability Sweet Spot"—minimizing detection lag while maximizing signal clarity.

2. Monte Carlo Stress-Testing

The model is subjected to 1,000+ synthetic "possible futures." We randomize both the severity of the crash and the background market noise to ensure the engine generalizes well to novel market conditions, not just what it's configured to expect.

• Reliability: Achieves ~96.3% Detection Success Rate (within a strict 14-day window) across randomized synthetic paths, with an average detection lag of ~2.5 ticks.

---

📈 Visual Results

Historical Regime Detection (2020 COVID Crash)

The engine successfully detected the volatility shift on February 24, 2020, with 0 days of detection lag.

Historical Regime Detection (2008 Financial Crisis)

🚀 Quick Start

1. Setup Environment

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

2. Fetch Data & Run Backtests

python src/utils/fetch_data.py        # Downloads historical SPY data
python validation/backtest.py       # Runs simulation on historical crashes
python src/utils/visualize_results.py  # Generates regime charts in /images

3. Performance & Sensitivity

python validation/profile_engine.py      # Verifies latency and O(1) memory
python validation/sensitivity_analysis.py # Runs Grid Search & Monte Carlo

---

🗂️ Project Structure

• src/engine/: Core Bayesian logic (RegimeDetector, DataStream, math_utils).
• src/utils/: Data fetching and visualization utilities.
• validation/: Backtesting, sensitivity analysis, and performance profiling.
• docs/: Mathematical derivations and technical architecture guides.
• images/: Exported visualizations (Regime charts, Heatmaps).
• data/: Historical CSV datasets (ignored by Git).

---

📖 Documentation

• Mathematical Foundation — Bayes' Theorem, log-space derivations, HMM transition math.
• Technical Architecture & Usage — APIs, data pipeline, validation methodology.