Ingestion Operations Runbook

This is the operational checklist for the v1 market data ingestion service in backend.

For source-specific ingestion behavior, see:

• Hyperliquid Data Source
• Polymarket Data Source
• DefiLlama Data Source

Runtime Shape

Run the API server and ingestion scheduler as separate Node processes. Both use the same SQLite database file.

cd backend

# Terminal 1: API server
npm run api

# Terminal 2: full ingestion scheduler
npm run ingest

For local testing, use the capped scheduler in Terminal 2 instead:

npm run ingest:local

npm run dev is an alias for npm run ingest.

The worker starts:

• Hyperliquid 1-minute OHLCV candle ingestion.
• Hyperliquid funding and OHLC-derived technical feature ingestion.
• Hyperliquid rollup/prune maintenance.
• Polymarket discovery and summary ingestion.
• DefiLlama macro ingestion.

Market data is global. Clone strategies read from the shared SQLite market memory instead of creating per-clone source fetches.

The scheduler is intentionally in-process Node scheduling, not host cron. A process manager should keep this worker alive in production; the worker itself owns source cadences, skip-while-running guards, health records, and graceful shutdown.

Health Checks

CLI:

npm run ingestion:status
npm run ingestion:status -- --json
npm run ingestion:check

API:

GET /api/v1/ingestion/health

The API currently requires the development x-user-id header. The response shape is:

{
  "ok": false,
  "rows": [
    {
      "source": "hyperliquid",
      "channel": "candles",
      "status": "ok",
      "expectedCadenceSec": 60,
      "staleAfterSec": 360,
      "ageSec": 20,
      "rowsWritten": 512
    }
  ]
}

Statuses:

Status	Meaning
ok	Latest run completed within the stale threshold.
running	Latest run has started but not completed and is still within the stale threshold.
missing	No run has been recorded for the expected source/channel.
failed	Latest run failed.
stale	Latest run is older than the cadence-derived stale threshold.

npm run ingestion:check exits nonzero if any channel is missing, failed, or stale.

Dashboard

The production frontend includes an unlinked ingestion dashboard page:

/admin/ingestion

The page is rendered by the existing Next/Vercel app and server-fetches the backend. Set PUMP_PARTY_BACKEND_API_URL in the frontend deployment environment to the production backend API origin.

The dashboard currently remains public by design for temporary operational visibility. Add a frontend and backend admin check before linking it from navigation or exposing broader operational controls.

Dashboard API:

GET /api/v1/ingestion/dashboard

Unlike GET /api/v1/ingestion/health, the dashboard endpoint is currently unauthenticated. It returns source/channel health plus operational volume and lag signals:

• total rows, rows in the last hour, and rows in the last 24 hours;
• unique subjects in the last hour and 24 hours;
• expected subject coverage where the channel has a registry-backed expectation;
• latest observed data timestamp and data lag;
• latest run rows, run age, run duration, and recent failures;
• retry and throttle counters when ingestion run metadata records them;
• derived warnings such as stale, data_lag, zero_rows_last_run, low_coverage, recent_failures, and throttled.

Manual Retry And Backfill

Use one-off commands to retry failed channels or fill recent gaps:

npm run hyperliquid:ohlcv
npm run hyperliquid:features
npm run hyperliquid:maintain
npm run polymarket:discover
npm run defillama:ingest

Dry-run external probes:

npm run polymarket:discover -- --dry-run --max-search-terms=5
npm run defillama:ingest -- --dry-run

Hyperliquid price ingestion is streaming. 1-minute candle WebSocket subscriptions write real OHLCV rows, and the feature ingestor derives RSI, Bollinger Bands, Moving Averages, and MACD from those rows. npm run hyperliquid:ohlcv -- --duration-ms=70000 runs a short local stream probe. Polymarket and DefiLlama use upsert-style writes for compact registry/summary rows.

Alerting Policy

Production should wire npm run ingestion:check or GET /api/v1/ingestion/health into the host monitor.

Initial alert threshold:

• Page or notify when ok is false for two consecutive checks.
• Treat failed as higher severity than stale.
• Treat missing after a fresh deploy as setup/configuration failure.

Retention

Market data retention is controlled by the source storage policies. Ingestion run records are compact and may be pruned later once host-level monitoring is stable.