Trading Engine

Trading Controller Lifecycle

Detailed analysis of the initialization, state management, position lifecycle, event processing loop, and graceful shutdown inside the core TradingController.

⏱️ 12 min read📊 Level: Advanced

The TradingController (bot_module/controller.py, ~19,000 LOC) is the central nervous system of the DepthSight bot execution engine. It orchestrates incoming market data, runs strategy instances, validates every signal through the RiskManager, and dispatches orders to exchange executors — all within a single asynchronous event loop.


Critical Structural Components

When initialized, the constructor (init, lines 471–746) establishes these critical structures to coordinate asynchronous processing and ensure thread safety across hundreds of concurrently running strategies:

ComponentTypePurpose
event_queueasyncio.Queue(maxsize=1000)Central bounded event bus for incoming market data and order updates
_event_handler_semaphoreasyncio.SemaphoreLimits concurrent event handlers (config.MAX_PARALLEL_EVENT_HANDLERS, default 64) to prevent thread exhaustion
_positions_dict_lockasyncio.LockProtects structural integrity of the ActivePositionMap dictionary
_symbol_locksDict[str, asyncio.Lock]Per-symbol locks ensuring sequential order processing on the same coin
_active_positionsActivePositionMapThread-safe, market-aware runtime storage for active positions
running_strategy_instancesDict[str, Tuple[BaseStrategy, dict]]Maps running strategy config_id to its strategy instance and runtime config
executorsDict[str, Executor]References to "live" (CcxtExecutor) and "paper" (PaperTradingExecutor)
redis_clientredis.asyncio.RedisClient for streaming state changes, runtime variables, and logs

ActivePositionMap — Multi-Market Position Storage

The ActivePositionMap is a custom dict subclass that stores active positions using composite keys of the form market_type:SYMBOL (e.g., futures_usdtm:BTCUSDT). It preserves backward compatibility by allowing symbol-only queries when the lookup is unambiguous, and falls back to matching logic when multiple market types share the same ticker symbol. This design is critical for multi-market support (linear futures, spot, coin-margined) on the same asset.


Startup Orchestration ( start() )

The start() method (lines 1122–1386) executes a carefully ordered sequence of startup tasks that must complete before any trading activity can occur:

Rendering diagram...

Step 1 — Configuration Loading (lines 1122–1174)

Fetches the user's AppConfig from PostgreSQL and applies runtime settings:

Sources:

Key operations:

  • Risk Limits: Applies risk-per-trade, stop-distance, drawdown, daily-loss, consecutive-loss, max-concurrent, and R:R settings via RiskManager.apply_user_settings (risk_manager.py:448–665).
  • Notifications: Loads custom Telegram chat IDs for real-time trade and alert delivery.
  • Blacklist: Syncs the static blacklist from the database into the controller's runtime memory.

Step 2 — Symbol Selection Config (lines 1002–1041, called at line 1179)

Loads the SymbolSelectionConfig which dictates the bot's market scope:

ModeBehaviorUse Case
STATICTrades only a fixed whitelist of symbols from strategy configConservative, manually curated portfolios
DYNAMIC_NATRSelects symbols based on NATR (Normalized ATR) volatility rankingAdaptive market making across top movers
DYNAMIC_ORACLEAI-driven symbol selection using ML Oracle regime parametersExperimental / high-alpha strategies

Symbol selection is configuration-only at this stage; per-symbol data subscriptions are applied later by _update_monitored_symbols() (controller.py:8118), invoked from _check_and_update_symbols() during startup (line 1307).

Step 3 — Runtime State Recovery (_load_runtime_state, lines 4595–4826)

After a server restart or process migration, the controller must reconnect to its prior state without losing track of open trades:

Sources:

After deserialization, positions are verified against the exchange via live_executor.get_open_positions() (line 4693) and inserted into the active map under _positions_dict_lock (lines 4774–4776).

Redis keys used for persistence:

  • depthsight:state:positions:{user_id}:{api_key_id} — ephemeral snapshot of active positions
  • depthsight:state:portfolio:{user_id}:{api_key_id} — portfolio-level stats (daily PnL, balance)
  • depthsight:state:strategies:{user_id}:{api_key_id} — running strategy instances and their params
  • depthsight:controller:runtime_state:{user_id}:{api_key_id} — durable runtime-state key written by _save_runtime_state() (line 4543), the source used for recovery

The depthsight:state:* snapshots carry a short TTL (REDIS_STATE_TTL_SECONDS, default 30 s, lines 5138–5152) so a dead controller's snapshot expires instead of lingering.

Step 4 — Exchange Reconciliation (_reconcile_positions_with_exchange, line 3855)

Automatically cross-references local state with actual exchange positions via the executor's get_open_positions(). This critical step handles:

  • Orphaned Positions: If the exchange reports an open position that the controller has no record of (e.g., due to a prior crash), it imports the position with an adopted flag via _build_adopted_position (lines 3310–3500).
  • Stale Entries: If the controller has a position in _active_positions but the exchange reports it as closed, it routes the trade through _handle_final_exit (line 10851) with reason CLOSED_ON_EXCHANGE and writes the closing record to the database.
  • Quantity Discrepancies: If partial fills occurred during downtime, the controller adjusts internal quantity tracking to match exchange reality.

Step 5 — Data Consumer Registration (lines 1194–1268)

The controller starts the DataConsumer (line 1194) and registers exchange and Redis listeners. For each active symbol:

  1. Candle close events (type: "CANDLE_CLOSE", emitted by DataConsumer at data_consumer.py:4018) — triggers strategy evaluation on each new closed candle.
  2. Tick events (type: "TICK", emitted at data_consumer.py:4028) — used for intra-candle stop-loss monitoring and tape-reading strategies.
  3. Order updates — exchange user-data streams call _handle_order_update (line 1260, definition at 15654); private streams are lazy by default and only boot-opened when PRIVATE_WS_MODE=always.

In addition, the Redis command listener and HFT event listener tasks are spawned (lines 1262–1268) to accept external CLOSE_POSITION / EMERGENCY_STOP commands.


Event Processing Loop (_run_main_loop())

Once started, the controller enters its main asynchronous event loop (_run_main_loop, lines 5431–5459). This is the heart of all trading activity:

Rendering diagram...

Event Types and Dispatch Logic

The main loop dequeues each event and spawns a handler task via _handle_event_with_context (lines 5420–5429), which throttles concurrency through the semaphore before calling _handle_event (line 5988).

Event TypeSourceAction
CANDLE_CLOSEDataConsumer — kline streamPosition management (line 6019) plus entry evaluation for on_candle_close triggers
TICKDataConsumer — aggTrade streamSpot virtual-TP checks (lines 6011–6016); entry evaluation for on_tick / on_condition_met triggers (lines 7517–7522)
SCALE_IN_RECALCController (after DCA fill)Forced position-management recalculation (line 6019)
ORDER_UPDATEExchange user data stream_handle_order_update — fill confirmation, updates position state
SIGNALStrategy instance (internal)_process_signal (line 8567) dispatches to RiskManager for assessment and execution
MANUAL_EXITRedis command bus / user commandclose_position() via _redis_command_listener (line 1513)
REGIME_CHANGEScreener / ML Oraclecheck_on_screener_update (lines 5701–5705), close with reason REGIME_CHANGE_LOSS_CUT (line 5714)

Position management runs through strategy_instance.manage_position() (line 6206); exit signals returned by the strategy are routed to close_position() (line 6226), and stop-loss modifications go through _replace_stop_loss (line 6362). Entry signals are checked last via _check_signals_for_symbol_on_event (lines 6511–6512).

Concurrency Model

The controller uses a triple-lock strategy to prevent race conditions:

  1. _positions_dict_lock (asyncio.Lock): Guards structural mutations to _active_positions dictionary (adding/removing positions).
  2. _symbol_locks[symbol] (asyncio.Lock): Per-symbol ordering guarantee. Ensures events for BTCUSDT are processed sequentially, preventing duplicate orders or conflicting state updates on the same asset.
  3. _event_handler_semaphore (Semaphore, default 64): Global throttle capping concurrent event handler tasks, preventing memory exhaustion during volatile market conditions when thousands of events arrive per second.

Position Lifecycle (Open — Manage — Close)

Each position traverses a well-defined lifecycle within the controller:

Rendering diagram...

1. Position Entry (_process_signal → LivePosition, lines 8567–9456)

When a signal passes the RiskManager assessment (call at lines 9175–9182), the controller constructs the position object BEFORE placing the order to avoid races with early ORDER_UPDATE events:

Sources:

The position is inserted into the active map under _positions_dict_lock (lines 9454–9456); exchange entry order placement and fill tracking are handled by _handle_entry_fill (line 9982).

2. Position Management (strategy.manage_position + controller actions)

On each CANDLE_CLOSE or SCALE_IN_RECALC event, the controller iterates active positions and calls the strategy's management layer:

  • Stop Loss / Take Profit / Partial Targets: Checked inside the strategy's _execute_position_management (strategy.py:4417), which iterates configured management blocks — trailing stop (4455), move-to-breakeven (4458), conditional exit (4461).
  • Trailing Stop: Supports ATR, percentage, and MA modes; when mode == "exchange" the controller places a native TRAILING_STOP_MARKET order (strategy.py:3895–3984).
  • Breakeven: Activated by target_type / target_value with an optional offset_pips buffer (strategy.py:3995–4007).
  • Controller-side actions: Management actions returned by the strategy are applied through _execute_management_actions (lines 2943–2991); stop modifications go through _replace_stop_loss.

3. Position Exit (close_position, lines 17692–17708)

Sources:

The reliable closure procedure polls for exchange confirmation for up to 120 attempts at 0.5-second intervals (lines 17811–17812), then finalizes the trade record.


Order Submission Flow (Signal to Order)

The path from strategy signal to live order traverses four distinct layers:

Rendering diagram...

RiskManager Gate

The assess_signal() call (bot_module/risk_manager.py, lines 1630–2216) performs twelve sequential stages of validation before any capital is committed — see Dynamic Risk Management for the canonical stage-by-stage breakdown and rejection codes. Highlights:

StageCheckRejection Code
0Symbol blacklistSYMBOL_BLACKLISTED
1Balance fetchZERO_BALANCE / PAPER_ZERO_BALANCE
3Global risk limits (daily loss, drawdown, max trades)GLOBAL_RISK_LIMIT
5Dynamic strategy/symbol multiplierZERO_RISK
7Entry price & SL validationINVALID_PRICE, SL_WRONG_SIDE, ZERO_SL_DISTANCE
8Reward/Risk ratioLOW_RR
9Exchange lot filters (stepSize, minQty, minNotional)MIN_QTY_VIOLATION, MIN_NOTIONAL_VIOLATION
10Dollar-based R/R checkLOW_DOLLAR_RR

Exchange Dispatch

Once approved, the controller selects the correct executor and calls place_order(). The CCXTExecutor.place_order() method (bot_module/exchanges/ccxt_executor.py, lines 773–1183) handles:

  1. Symbol normalization: BTCUSDT to BTC/USDT:USDT (futures) or BTC/USDT (spot).
  2. Order type mapping: Binance STOP_MARKET mapped to CCXT "market" with stopLossPrice / takeProfitPrice params (lines 801–843; mapping in _map_order_type_to_ccxt, lines 2509–2516).
  3. Exchange-specific parameters: OKX uses posSide (lines 877–896), Gate.io uses settle: "usdt" (lines 901–903), Bitget sets params["hedged"] = not is_unilateral (lines 917–919).
  4. Rate-limit compliance: Delegated to CCXT's built-in enableRateLimit token bucket.

State Persistence & Redis Publishing

The controller continuously publishes its runtime state to Redis (via _publish_state_to_redis_impl, lines 4847–5203), enabling:

  1. Real-Time Frontend Updates: Position, strategy, and portfolio snapshots are written every ~8 seconds (state_publish_interval, line 5473) to per-controller keys.
  2. Crash Recovery: A new controller instance reconstructs its active position map from the durable runtime-state key in _load_runtime_state (lines 4595–4826), then re-verifies against the exchange.
  3. Inspection: Operators and multi-account views can read the latest snapshot; ephemeral keys expire automatically if the controller dies.
Sources:

Shutdown & Cleanup Sequence (stop())

When a stop command is received (via Redis command bus or API), the stop() method (lines 1388–1511) executes a graceful teardown:

Rendering diagram...

Shutdown Steps

  1. _running = False: The main loop terminates on its next iteration.
  2. Task Cancellation: Cancels the main loop, config reloader, market-info updater, Redis listeners, dynamic symbol selection, and equity recorder tasks (lines 1400–1443), waiting up to 5 seconds for them to finish.
  3. Stream & Subscription Teardown: Clears DataConsumer subscriptions (line 1448), releases shared private WebSocket streams and stops executor user-data streams (lines 1453–1468), stops the consumer (line 1470), and closes executors (lines 1471–1492).
  4. TradeLogger Stop: Stops the trade logger (lines 1493–1495).
  5. State Finalization: Persists runtime state via _save_runtime_state() (line 1499) and closes the Redis client (lines 1501–1509).

Note that stop() does not liquidate open positions or cancel open exchange orders — positions remain on the exchange and are recovered on the next startup by _load_runtime_state + reconciliation. On-demand exits are issued via close_position() (e.g., CLOSE_POSITION / EMERGENCY_STOP Redis commands handled by _redis_command_listener, line 1513).


Error Handling & Recovery

The controller implements multiple layers of error resilience:

Exchange API Failures

Stop-loss placement failures are retried with fixed backoff inside _replace_stop_loss — up to 3 attempts with 2/4/8-second delays (lines 12624–12659), aborting early if the position is no longer OPEN:

Sources:

Full position closure uses the reliable procedure in close_position (see Position Exit above) with its own bounded retry loop.

Position Reconciliation Timer

A background task (_run_periodic_tasks, line 5461) reconciles every 60 seconds (lines 5482–5485), with the first run staggered by a random delay to avoid thundering-herd REST calls after a mass restart. Reconciliation only runs while live strategy instances exist (lines 5543–5550) and:

  1. Fetches open positions from the exchange via the executor's get_open_positions().
  2. Cross-references with _active_positions.
  3. Repairs any discrepancies — adopting missing positions, closing stale entries, and correcting quantity mismatches (same paths as startup reconciliation).

Graceful Degradation

If a Redis state publish fails, the controller logs the error (lines 5199–5203) and continues trading on in-memory state — there is no separate degraded mode. Snapshots refresh on the next ~8-second interval once Redis is reachable again; if the controller dies, its ephemeral keys expire after their 30-second TTL.