Ingestion Operations Runbook

This is the operational checklist for the v1 market data ingestion service in services/market-backend.

For source-specific ingestion behavior, see:

• Hyperliquid Data Source
• Polymarket Data Source
• DefiLlama Data Source

Runtime Shape

Run the API server and ingestion scheduler as separate Node processes. Both use the same SQLite database file.

cd services/market-backend

# Terminal 1: API server
npm run api

# Terminal 2: full ingestion scheduler
npm run ingest

For local testing, use the capped scheduler in Terminal 2 instead:

npm run ingest:local

npm run dev is an alias for npm run ingest.

The worker starts:

• Hyperliquid allMids midpoint ingestion.
• Hyperliquid 1-minute OHLC candle ingestion.
• Hyperliquid feature ingestion.
• Hyperliquid rollup/prune maintenance.
• Polymarket discovery and summary ingestion.
• DefiLlama macro ingestion.

Market data is global. Clone strategies read from the shared SQLite market memory instead of creating per-clone source fetches.

The scheduler is intentionally in-process Node scheduling, not host cron. A process manager should keep this worker alive in production; the worker itself owns source cadences, skip-while-running guards, health records, and graceful shutdown.

Health Checks

CLI:

npm run ingestion:status
npm run ingestion:status -- --json
npm run ingestion:check

API:

GET /api/v1/ingestion/health

The API currently requires the development x-user-id header. The response shape is:

{
  "ok": false,
  "rows": [
    {
      "source": "hyperliquid",
      "channel": "candles",
      "status": "ok",
      "expectedCadenceSec": 60,
      "staleAfterSec": 360,
      "ageSec": 20,
      "rowsWritten": 512
    }
  ]
}

Statuses:

Status	Meaning
ok	Latest run completed within the stale threshold.
running	Latest run has started but not completed and is still within the stale threshold.
missing	No run has been recorded for the expected source/channel.
failed	Latest run failed.
stale	Latest run is older than the cadence-derived stale threshold.

npm run ingestion:check exits nonzero if any channel is missing, failed, or stale.

Manual Retry And Backfill

Use one-off commands to retry failed channels or fill recent gaps:

npm run hyperliquid:candles
npm run hyperliquid:features
npm run hyperliquid:maintain
npm run polymarket:discover
npm run defillama:ingest

Dry-run external probes:

npm run polymarket:discover -- --dry-run --max-search-terms=5
npm run defillama:ingest -- --dry-run

Hyperliquid price ingestion is streaming. allMids writes compact 5-second midpoint buckets, and 1-minute candle WebSocket subscriptions write real OHLC rows. npm run hyperliquid:candles -- --duration-ms=70000 runs a short local stream probe. Polymarket and DefiLlama use upsert-style writes for compact registry/summary rows.

Alerting Policy

Production should wire npm run ingestion:check or GET /api/v1/ingestion/health into the host monitor.

Initial alert threshold:

• Page or notify when ok is false for two consecutive checks.
• Treat failed as higher severity than stale.
• Treat missing after a fresh deploy as setup/configuration failure.

Retention

Market data retention is controlled by the source storage policies. Ingestion run records are compact and may be pruned later once host-level monitoring is stable.