Deployment & Scaling

System design guide for scaling DepthSight to handle thousands of concurrent users — horizontal scaling, Redis splitting, sharded bot processes, PgBouncer integration, and the stateless architecture.

⏱️ 6 min read📊 Level: Advanced

DepthSight is built with a stateless, decentralized architecture. While a solo deployment runs comfortably on a single server (4 CPU, 12 GB RAM handles ~50 concurrent users), a commercial SaaS handles scaling by partitioning users and resources across multiple independent clusters.


Horizontal Scaling Model

In a clustered deployment, services are distributed across independent host groups to isolate CPU-intensive operations from critical execution pipelines.

Rendering diagram...

Service Distribution

Service GroupScaling StrategyResource Profile
API Nodes (FastAPI)Stateless, add behind LBCPU-light, memory-light
Bot Runners (TradingController)Sharded by api_key_id % workersCPU-heavy (indicator calc), memory-heavy
MarketDataServiceSingle service, per-exchange consumersNetwork I/O heavy, CPU-light
Redis (System)Standalone instance (not a cluster)Memory-bound
Redis (Market)Standalone instance (not a cluster)Throughput-bound
PostgreSQLSingle primary (+ optional PgBouncer overlay)I/O-bound

Redis Splitting (State vs Data)

To support high-frequency trading (HFT) and keep order book telemetry separate from system operations, the infrastructure splits Redis into two isolated instances:

System Redis (Commands & Cache)

FunctionData StructureExamples
Rate LimitingString (TTL)Rate-limit counters, TTL guard strings
Command BusPub/Subdepthsight:commands channel
Quota TrackingString (TTL)usage:{user_id}:{feature}:{date} counters
Celery BackendString / ListTask results, worker queues
State StorageStringdepthsight:state:positions:{user_id}:{api_key_id}, depthsight:state:portfolio:{user_id}:{api_key_id}
Webhook DedupString (TTL, 30s)tv:webhook:dedupe:*
WebSocket Fan-OutPub/Subuser_logs:{user_id}, depthsight:events:positions:{user_id}

Telemetry Redis (Market Data Fan-Out)

FunctionData StructureExamples
Market EventsPub/Subdepthsight:market_data:events:*
Warm SnapshotsString (TTL, 1h)depthsight:market_data:snapshot:*
Order BooksStringLatest L2 snapshots

This separation ensures that even if a market data flood overwhelms the telemetry Redis, system operations (auth, billing, command dispatch) remain unaffected.


Sharded Bot Processes

bot_runner.py spawns additional OS-process shards only when BOT_RUNNER_PROCESSES > 1 (default 1, see bot_module/config.py). Each shard is a single asyncio event loop hosting many TradingController instances as concurrent tasks; controllers are partitioned across shards by API key, not by user.

User Sharding

Controllers are assigned to shards by the numeric id of the API key they trade with (every controller belongs to exactly one ApiKey row):

Shard assignment (one shard = one bot_runner process):
  shard_id = api_key_id % BOT_RUNNER_PROCESSES

The shard assignment is computed as:

Sources:

Stateless Execution

Individual user bots fetch runtime parameters from PostgreSQL and cache status in Redis:

Bot Worker startup:
  1. Load StrategyConfig from PostgreSQL
  2. Load AppConfig (risk limits, notifications)
  3. Restore active positions from Redis
  4. Connect to MarketDataService via Redis Pub/Sub
  5. Begin trading loop

Crash Recovery: If a bot runner server crashes:

  1. A health monitoring script detects the failure.
  2. A new instance is spawned on a healthy node.
  3. The controller pulls last active position states from Redis.
  4. Exchange reconciliation confirms position accuracy.
  5. Trading resumes within seconds.

Process Isolation

When sharded across multiple processes, each bot_runner worker runs as a separate OS process with:

  • Independent Python interpreter
  • Isolated memory space (no GIL contention between shards)
  • Individual Redis connections
  • Dedicated asyncio event loop (controllers on the same shard share that loop as tasks)

This prevents one user's strategy (e.g., memory leak in custom indicator) from affecting controllers scheduled onto other shards.


PgBouncer Integration (Optional)

With thousands of active bot workers running asynchronously, PostgreSQL's native connection limits (typically 100–500) can be exhausted on large SaaS deployments. DepthSight therefore ships PgBouncer as an optional overlay (docker-compose.pgbouncer.yml); it is not enabled by default.

Transaction Mode Pooling

When enabled, PgBouncer runs in Transaction Mode:

Bot Worker -> PgBouncer (Transaction Mode) -> PostgreSQL
     |                                       |
     | Connection returned to pool            |
     | after each SQL statement               |
Without PgBouncerWith PgBouncer
1000 workers = 1000 idle TCP connections1000 workers share 20-50 pool connections
PostgreSQL memory: ~10MB per connectionPostgreSQL memory: ~10MB total pool
Connection storms on restartControlled pool warm-up

Connection Lifecycle

  1. Worker needs to write trade log → requests connection from PgBouncer.
  2. PgBouncer assigns from pool (or creates new if pool not full).
  3. Worker executes INSERT/UPDATE.
  4. Connection returned to pool immediately (transaction committed).
  5. No idle connections held during trading loop.

Deployment Configuration Reference

EnvironmentTypical SpecUsers Supported
Solo (all-in-one)4 CPU, 12 GB RAM, 50 GB SSD1-50 users
Small Production8 CPU, 16 GB RAM, 100 GB SSD, 2 nodes50-500 users
Medium SaaS16 CPU, 32 GB RAM, 200 GB SSD, 4 nodes500-2000 users
Large Enterprise32+ CPU, 64+ GB RAM, 500 GB+ SSD, 10+ nodes2000+ users

Docker Compose Services

ServiceReplicasRestart Policy
FastAPI API2+always
WebSocket Server2+on-failure
bot_runnerSharded per node (BOT_RUNNER_PROCESSES)on-failure
MarketDataService1on-failure
Celery Worker2+on-failure
Celery Beat1on-failure
Slack Agent1on-failure
MCP Memory1always
Redis (system + market)1 eachalways
PostgreSQL1always
PgBouncer (optional overlay)1 per DBalways
Caddy (reverse proxy)1always