Market Data Ingestion
Implemented componentThe entry point of the pipeline: historical market-data ingestion and real-time ingestion of one-minute bars, through the Alpaca provider.
02 / MARKET DATA & TECHNICAL ANALYSIS
AutoBroker.
From market events
to structured information.
A personal C# market-data project with a deliberate progression: ingest historical and real-time bars, persist them in TimescaleDB, aggregate higher timeframes, and calculate technical information. A distinct Analyzer is planned for higher-level market analysis.
DOMAINMarket data & analysis
FOUNDATIONC# / .NET
ARCHITECTUREDurable persistence + lightweight messaging
THE IDEA
Bring structure
to a stream of events.
01 — THE PIPELINE
Four stages.
Clear responsibilities.
The intended architecture separates collecting market data, building timeframes, deriving technical information, and interpreting that information. TimescaleDB is the durable persistence layer for market bars across all timeframes.
Market Data Aggregator
Partially implementedReads one-minute bars and aggregates higher timeframes such as 5m, 15m, 1h, and daily. The intended architecture persists all bar timeframes in TimescaleDB; aggregation logic exists, with persistence integration still partial.
Market Data Processor
Partially implementedA distinct stage after aggregation. It calculates technical indicators and supports derived technical structures, including support and resistance levels. Calculation components exist; complete stage coordination remains partial.
Market Data Analyzer
Planned · not yet implementedThe next architectural stage: chart-pattern matching, indicator-pattern matching, combinations and sequences of technical conditions, and other higher-level analysis over processed market data.
Status labels distinguish component implementation from completed stage integration. The architecture is defined; its implementation is progressing. The Analyzer is not yet implemented.
02 — THE TOOLKIT
A purposeful
persistence boundary.
C# / .NET
Independent ingestion, aggregation, and processing service components, with a planned Analyzer stage.
Alpaca
Historical market-data and real-time one-minute bar ingestion.
TimescaleDB / PostgreSQL
The intended primary store for time-series market bars across all timeframes, from one-minute input to higher-timeframe aggregates. PostgreSQL bar-write code exists; full TimescaleDB integration is partially implemented.
RabbitMQ
A slim messaging layer for service signaling and work distribution. Durable market state belongs in persistence; the intended message design keeps payloads relatively small.
SQL Server
Other relational data supported by current repository code, including asset metadata and derived key-level records. This role is separate from TimescaleDB’s time-series bar storage in the intended architecture.
ASP.NET Core / SignalR
Existing dashboard endpoints and subscription-based browser update components, complementary to the processing pipeline.
JavaScript / Canvas
A browser dashboard for candle visualization and symbol/timeframe selection.
03 — SYSTEM ARCHITECTURE
Persist. Coordinate. Process.
Historical data and real-time one-minute bars enter through ingestion. TimescaleDB persists input bars and the Aggregator’s higher timeframes. The Processor derives indicators and technical structures; the planned Analyzer will interpret that processed information. RabbitMQ coordinates work alongside the durable data path.
IMPLEMENTATION STATUS
Ingestion and technical calculation components are implemented. Aggregation, TimescaleDB integration, and Processor coordination are partially implemented. The Analyzer is planned and not yet implemented. Lightweight persisted-data signaling and recovery describe the architectural design, rather than a claim that every current handler already follows it.
04 — ENGINEERING DECISIONS
Durable state.
Slim messaging.
Persistence + lightweight messaging
As a personal project, AutoBroker was designed with infrastructure cost and operational complexity as architectural constraints. Persisting important market state and intermediate results in the database, alongside RabbitMQ for lightweight signaling and work distribution, keeps the system appropriate for its scale.
Durable state remains in the persistence layer, messages stay relatively small, and services can recover the data they need from persistent storage. RabbitMQ coordinates processing without becoming the source of truth. This intentional separation provides a clear recovery model while avoiding unnecessary infrastructure cost and complexity.
Processing before interpretation
The Processor derives indicators, support/resistance levels, and related technical information from aggregated bars. The planned Analyzer owns higher-level pattern matching and combinations or sequences of conditions. Distinct stages keep those responsibilities explicit.
Explicit time windows
Aggregation uses half-open time windows and distinguishes partial, finalized, and corrected bars. Existing aggregation components and focused tests establish those calculation boundaries.
EXPLORE THE OTHER PROJECT
VaadBot