Skip to content

Data Source API And UX Spec

This is a reference contract for the product/API layer on top of the SQLite backend.

For review-level docs, start with:

The current target UX model is:

text
Data Stream Block -> Asset Block -> Trading Prompt Block

The backend stores the same graph shape for every mode:

text
data_stream -> asset_selection -> trading_prompt

Each Trading Prompt Block defines one strategy. The backend expands each strategy across its enabled assets at execution time.

The workflow canvas can additionally show a visual Trading Clone Block:

text
data_stream -> asset_selection -> trading_prompt -> trading_clone

trading_clone is not persisted as a strategy node. It is a UI terminal for clone-level model, trading on/off, and history. Its position and visible prompt-to-clone line are persisted as layout metadata.

Control Granularities

Basic

Basic uses one hidden generated pipeline:

text
Default Data Stream -> Default Asset Selection -> Default Trading Prompt

Controls:

  • Asset category selection.
  • Data depth: Fast, Balanced, or Deep.
  • Optional custom behavior prompt.
  • Clone model.
  • Trading enabled/paused.

No source/channel/granularity/lookback/max-point controls are exposed.

Custom

Custom uses the same graph shape, but exposes editable pieces:

text
Chosen Data Stream -> Individual Asset -> Chosen Trading Prompt

Controls:

  • Individual Asset Blocks selected from a searchable category/subcategory palette.
  • Hyperliquid Data Stream Blocks are dropped by source/channel, then configured with the associated asset inside the block.
  • DefiLlama Data Stream Blocks are dropped empty, then configured inside the block.
  • Polymarket Data Stream Blocks are dropped empty, then configured by keyword-searching and selecting a specific market.
  • Retrieval depth: Fast, Balanced, or Deep.
  • Stream prompt inclusion: Off, Auto, Standard, or Required.
  • Trading prompt enabled/paused state, cadence, custom behavior prompt, and execution behavior.
  • View Data action on each Data Stream Block links to the source page.

Custom users do not manually tune lookback, granularity, max points, or channel weights.

Power User

Power users can create multiple branches:

text
Deep Hyperliquid Stream -> BTC Asset -> Macro Trading Prompt
Fast Hyperliquid Stream -> DOGE Asset -> Momentum Trading Prompt
Polymarket Market -> BTC Asset -> Event Trading Prompt

Controls:

  • Create/edit data stream nodes.
  • Create/edit individual asset nodes.
  • Create/edit trading prompt nodes.
  • Connect streams, selections, and prompts.
  • Open advanced stream controls for prompt inclusion, lookback, and granularity.
  • Keep Asset Blocks limited to asset identity, enabled/paused state, and exchange links.
  • Use global prompts, global prompts with per-asset prompt configurations, or asset-specific-only prompts.
  • Configure strategy, risk, and execution behavior on Trading Prompt Blocks.
  • Override decision cadence per trading prompt within backend limits.

Power users still do not use an AI prompt for retrieval. Retrieval stays deterministic.

Asset Naming

Use asset everywhere in product and API language. Use symbol for the exchange-facing identifier.

Examples:

ts
supportedAssets
enabledAssets
assetSelection
assetOverrides
assetGroups
symbol

Avoid assetOrToken. External upstream fields can keep native names, such as Polymarket clobTokenIds.

Asset Selection

Asset selection is user/config bounded before the AI decision runs:

  1. User chooses high-level Hyperliquid categories.
  2. User chooses subcategories where supported.
  3. User can manually include or exclude specific symbols.
  4. The trading engine only considers enabled assets for trades.

Enabled assets receive asset-specific context. Broader market context, such as macro or related-market data, can still be used to evaluate those enabled assets.

Hyperliquid Category Taxonomy

The picker should follow Hyperliquid's visible market taxonomy:

text
High level:
  All
  Perps
  Spot
  Crypto
  Tradfi
  Trending

Crypto:
  All
  AI
  Defi
  Gaming
  Layer 1
  Layer 2
  Meme

Tradfi:
  All
  Stocks
  Indices
  Commodities
  FX
  Pre-IPO

Spot, if exposed:
  All
  USDC
  USDH
  USDT

Out of scope:

  • Generic HIP-3 category exposure.

The official Hyperliquid Info API gives the perp and spot market universes, but not durable category/subcategory tags. Category membership should live in SQLite as asset metadata, seeded from a checked-in taxonomy snapshot and refreshed intentionally.

Data Streams

Data stream nodes define what structured market memory can be retrieved.

Use Fast, Balanced, and Deep for everyone:

  • Basic gets a generated default.
  • Custom chooses profiles for source/channel blocks.
  • Power opens advanced stream controls.

Default retrieval tuning is global per data stream/profile. The current workflow prefers individual Asset Blocks rather than category blocks with hidden per-symbol overrides.

Profile definitions include:

  • Enabled source/channel pairs.
  • Lookback.
  • Granularity.
  • Derived max points.
  • Internally managed context prompt budget.
  • Prompt inclusion policy: off, auto, standard, or required.

Internal prompt priority can still be derived from the preset for context ordering, but it is not a user-facing control.

Override precedence:

text
per-asset configuration
> asset/group profile assignment
> data stream default profile
> built-in profile definition

Basic users do not see advanced stream controls. Custom users choose profiles and stream prompt inclusion. Power users can configure prompt inclusion, lookback, and granularity directly.

Trading Prompt Coverage

Trading prompt nodes support three coverage modes:

text
global
  One prompt applies to all enabled assets.

global_with_asset_overrides
  Global prompt applies by default.
  Specific assets can use their own prompt settings.

asset_specific_only
  No global fallback.
  Only assets explicitly listed in per-asset prompt configurations are eligible for this prompt node.

Compiler rule:

text
specific asset prompt > global prompt

If two asset-specific prompts claim the same asset inside one compiled branch, reject the graph as invalid in v1.

Source Coverage

Initial source/channel catalog defaults should mirror backend cadence:

  • Hyperliquid OHLCV: 60s source rows from candle WebSocket subscriptions; larger windows derived locally.
  • Hyperliquid RSI: Wilder RSI 14 derived from OHLCV, 300s buckets.
  • Hyperliquid Bollinger Bands: 20-period SMA plus/minus 2 standard deviations, derived from OHLCV, 300s buckets.
  • Hyperliquid Moving Averages: SMA 20 and EMA 20 derived from OHLCV, 300s buckets.
  • Hyperliquid MACD: 12/26/9 EMA-derived MACD, 300s buckets.
  • Hyperliquid funding rates: 300s buckets from perp context and predicted funding polls; unavailable for spot assets.
  • Polymarket market discovery: registry channel, 3600s cadence.
  • Polymarket immediate market: 300s cadence.
  • Polymarket registry/search should include live and upcoming markets only. Closed, archived, and ended markets are pruned from local SQLite.
  • Polymarket later market: 1800s cadence.
  • DefiLlama TVL/base capital, flow changes, and DEX volume/fees/revenue/open interest: 3600s cadence.

Subject Model

Use a subject model for context reads.

ts
type DataSubjectType =
  | 'asset'
  | 'market'
  | 'chain'
  | 'protocol'
  | 'stablecoin'
  | 'global';

interface DataSubject {
  type: DataSubjectType;
  id: string;
  displayName?: string;
  linkedAssetSymbol?: string;
}

Examples:

json
{ "type": "asset", "id": "BTC", "displayName": "Bitcoin" }
{ "type": "chain", "id": "solana", "displayName": "Solana", "linkedAssetSymbol": "SOL" }
{ "type": "global", "id": "defi", "displayName": "DeFi Market" }
{ "type": "stablecoin", "id": "USDC", "displayName": "USDC" }

Hyperliquid is asset-scoped. In the workflow UI, users drop a Hyperliquid source/channel block first, then choose the associated Hyperliquid asset inside that block before connecting it to the matching Asset Block. The picker should filter assets by the selected channel's hyperliquid_assets support flags, and the compiler must skip unsupported pairs with a warning. Example: spot assets can be valid for OHLCV-derived RSI but invalid for funding rates. Polymarket is market-scoped: users search Polymarket and choose a specific market/event. Legacy asset-linked Polymarket rows can still exist in the registry for ingestion matching. DefiLlama is macro-scoped: global, chain, protocol, and stablecoin.

Pipeline Graph API

http
GET /api/v1/clones/:cloneId/pipeline
PUT /api/v1/clones/:cloneId/pipeline
GET /api/v1/clone/clones/:cloneId/entitlement
GET /api/v1/polymarket/markets/search?q=:query

Shape:

ts
type PipelineNodeKind = 'data_stream' | 'asset_selection' | 'trading_prompt';
type PipelineEdgeKind = 'provides_context_to' | 'selects_assets_for';

interface ClonePipelineConfig {
  version: 1;
  cloneId: number;
  nodes: Array<{
    id: string;
    kind: PipelineNodeKind;
    name: string;
    config: Record<string, unknown>;
    position?: { x: number; y: number };
  }>;
  edges: Array<{
    id: string;
    fromNodeId: string;
    toNodeId: string;
    kind: PipelineEdgeKind;
    priority: number;
  }>;
  layout?: {
    strategyClone?: {
      position?: { x: number; y: number };
      connectedPromptNodeIds?: string[];
    };
  };
}

The backend validates v1 graphs:

text
data_stream -> asset_selection
asset_selection -> trading_prompt

The visual trading_prompt -> trading_clone route is not persisted in edges. It round-trips through layout.strategyClone.connectedPromptNodeIds, so the canvas can keep the line visible without changing the executable graph.

Save, dry-run, backtest, and runtime paths compile the graph into enabled assets, data read plans, estimated prompt tokens, warnings, and the trading prompt payload. Execution expands those candidates into one prompt run per due asset.

Pipeline saves enforce active asset capacity against GET /api/v1/clone/clones/:cloneId/entitlement. Clones without a purchase-backed entitlement row receive a mock Starter fallback for local/dev and legacy paths.

During development these endpoints accept x-user-id as the backend-owned user id. Production auth should map external providers into users.id before route handlers run.

Latest Data API

http
POST /api/v1/clones/:cloneId/assets/:symbol/latest-data

The Latest Data endpoint is a read-only review endpoint for internal stats, debugging, or later data-inspection views. It accepts an optional pipeline draft and returns the effective memory reads for one asset using the same context resolver path as trading decisions. The current strategy canvas View Data action links out to the original source site instead of opening this raw-memory response.

For legacy DefiLlama reads without an explicit scope, the resolver filters relevant subjects before applying maxPoints: global defi, matching chain rows by metrics.tokenSymbol, and matching stablecoin subjects by symbol. New DefiLlama blocks can explicitly target one scope: global, chain, protocol, or stablecoin.

Shape:

ts
interface AssetLatestDataResponse {
  symbol: string;
  asOfMs: number;
  branches: Array<{
    id: string;
    dataStreamNodeId: string;
    assetSelectionNodeId: string;
    tradingPromptNodeId: string;
    candidateSymbols: string[];
    memoryReads: Array<{
      key: string;
      symbol: string;
      source: 'hyperliquid' | 'polymarket' | 'defillama';
      channel: string;
      mode: 'latest' | 'timeseries' | 'summary' | 'signal';
      granularitySec: number;
      lookbackSec: number;
      maxPoints: number;
      required: boolean;
      recordCount: number;
      records: Array<Record<string, unknown>>;
    }>;
    warnings?: string[];
  }>;
  warnings?: string[];
}

Ingestion Health API

http
GET /api/v1/ingestion/health

Returns source/channel health rows from ingestion_runs for admin UI and host-level monitoring. ok is false if any expected channel is missing, failed, or stale.

Ingestion Dashboard API

http
GET /api/v1/ingestion/dashboard

Returns the public temporary ingestion dashboard payload used by the unlinked frontend page at /admin/ingestion. The endpoint is intentionally unauthenticated for now and should receive an admin gate before it is linked from product navigation or expanded with operational controls.

The response combines latest health rows with source table statistics:

ts
interface IngestionDashboardResponse {
  ok: boolean;
  generatedAt: number;
  windows: {
    lastHourSec: 3600;
    last24hSec: 86400;
  };
  summary: {
    totalRows: number;
    rowsLastHour: number;
    rowsLast24h: number;
    okChannels: number;
    runningChannels: number;
    missingChannels: number;
    staleChannels: number;
    failedChannels: number;
    retryCountLast24h: number;
    throttleCountLast24h: number;
  };
  channels: Array<{
    source: string;
    channel: string;
    table: string;
    mode: 'stream' | 'poll' | 'rollup';
    origin: 'source' | 'derived';
    cadenceSec: number;
    status: 'ok' | 'running' | 'missing' | 'failed' | 'stale';
    staleAfterSec: number;
    ageSec: number | null;
    lastStartedAt: number | null;
    lastCompletedAt: number | null;
    lastRowsWritten: number | null;
    errorMessage: string | null;
    totalRows: number;
    rowsLastHour: number;
    rowsLast24h: number;
    uniqueSubjectsLastHour: number;
    uniqueSubjectsLast24h: number;
    expectedSubjects: number | null;
    coveragePctLast24h: number | null;
    firstObservedAt: number | null;
    latestObservedAt: number | null;
    dataLagSec: number | null;
    runsLast24h: number;
    completedRunsLast24h: number;
    failedRunsLast24h: number;
    runningRunsLast24h: number;
    avgDurationSec: number | null;
    maxDurationSec: number | null;
    retryCountLast24h: number;
    throttleCountLast24h: number;
    warnings: string[];
  }>;
}

Asset Selection Projection

The pipeline graph is the canonical source of truth. If Basic/Custom flows need a simpler asset-selection API later, it should be a projection that reads and writes generated asset_selection graph nodes.

Potential later endpoints:

http
GET/PUT /api/v1/clones/:cloneId/asset-selection
GET     /api/v1/clones/:cloneId/trade-universe

Shape:

ts
interface AssetSelectionConfig {
  version: 1;
  cloneId: number;
  source: 'hyperliquid';
  highLevelCategories: Array<'all' | 'perps' | 'spot' | 'crypto' | 'tradfi' | 'trending'>;
  subcategories: {
    crypto?: string[];
    tradfi?: string[];
    spot?: string[];
  };
  explicitlyEnabledSymbols: string[];
  explicitlyDisabledSymbols: string[];
}

The workflow canvas currently uses the pipeline graph plus GET /api/v1/assets for the builder palette.

Execution Config Projection

The pipeline graph and clone row are the canonical sources of truth. If simple controls need an execution-config API later, it should be a projection over:

  • the clone row for model and active/paused state;
  • the active Trading Prompt Block for enabled/paused state, cadence, custom behavior prompt, strategy, risk, and execution behavior;
  • the relevant Asset Blocks for enabled asset identity.

Shape:

ts
interface CloneExecutionConfig {
  version: 1;
  cloneId: number;
  model: string;
  status: 'active' | 'paused' | 'inactive' | 'archived';
  tradingPromptEnabled: boolean;
  customBehaviorPrompt: string;
  decisionCadenceSec?: number;
  candidatePolicy: {
    includeOpenPositions: boolean;
    includeUserUniverse: boolean;
    deterministicPrefilter: boolean;
  };
}

Legacy migration fields such as trader archetype, market momentum sensitivity, and analysis timeframe can still be carried in compatibility paths, but they should not be treated as primary workflow controls.

Default scheduled execution is every 5 minutes per enabled clone/branch/asset.

Recommended v1 cadence:

text
Default: every 5 minutes
Optional slower prompt cadences: 15 minutes, 30 minutes, 2 hours, 6 hours

Candidate Asset Policy

The platform can support every Hyperliquid asset, but the LLM should not evaluate all assets on every run by default.

Separate three concepts:

  • Supported assets: everything the backend knows about.
  • Trade universe: assets the user has allowed this clone to trade.
  • Run candidates: assets included in this specific decision call.

Default candidates:

  • Assets with open positions.
  • Assets in the user-enabled trade universe.
  • Assets passing deterministic prefilters, capped to a safe maximum.

Recommended branch candidate cap before per-asset execution:

text
Max candidate assets per strategy branch: 10 default, 25 advanced maximum

For a power user who wants to watch hundreds of assets, the system should scan structured data deterministically first, then send only the strongest candidates to the model.

Decision Worker

The worker operates on compiled pipeline branches:

text
active clone
  -> effective context
  -> cadence-due asset
  -> per-asset decision run row
  -> decision engine
  -> validated action rows

The local worker command defaults to a no-op hold engine so the storage and cadence path can be tested without placing trades or requiring model credentials.

bash
npm run decisions:run-once -- --clone-id=42 --force
npm run decisions:worker

Set DECISION_MODEL_PROVIDER=auto plus the relevant provider API key to use the model-backed engine selected by the clone's product model. Provider-specific modes can still be used for local testing. The model engine expects strict JSON actions; Prop Trading performs account-aware safety gates and mutates prop-account state for product execution.

The clone-scoped dry-run execution layer still records workflow-debug actions as clone_execution_orders with status dry_run and writes validation/rejection events to clone_execution_ledger. New audition and race runtime work should use prop-account decision actions, paper fills, positions, balances, and ledger rows from Prop Trading.

SQLite Configuration Tables

Target state: SQLite owns backend state end to end.

Core tables:

  • clones
  • clone_asset_selection_rules
  • clone_trade_universe
  • clone_pipeline_nodes
  • clone_pipeline_edges
  • clone_pipeline_layouts
  • clone_pipeline_versions
  • clone_execution_configs
  • clone_decision_runs
  • clone_decision_actions
  • orders, fills, positions, and ledger entries

clone_asset_selection_rules and clone_execution_configs are helper/projection tables. They should not become competing configuration systems. The canonical editable graph remains clone_pipeline_nodes plus clone_pipeline_edges.

clone_asset_selection_rules

Preserves user category/subcategory intent:

sql
CREATE TABLE IF NOT EXISTS clone_asset_selection_rules (
  clone_id INTEGER PRIMARY KEY,
  source TEXT NOT NULL DEFAULT 'hyperliquid',
  high_level_categories_json TEXT NOT NULL DEFAULT '["crypto"]',
  subcategories_json TEXT NOT NULL DEFAULT '{"crypto":["all"]}',
  explicitly_enabled_symbols_json TEXT NOT NULL DEFAULT '[]',
  explicitly_disabled_symbols_json TEXT NOT NULL DEFAULT '[]',
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

clone_trade_universe

Stores expanded effective assets after selection rules and explicit symbol selections:

sql
CREATE TABLE IF NOT EXISTS clone_trade_universe (
  clone_id INTEGER NOT NULL,
  symbol TEXT NOT NULL,
  display_name TEXT,
  enabled INTEGER NOT NULL DEFAULT 1,
  source TEXT NOT NULL DEFAULT 'hyperliquid',
  selection_source TEXT NOT NULL DEFAULT 'category_rule',
  sort_order INTEGER,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  PRIMARY KEY (clone_id, symbol),
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

clone_pipeline_nodes / clone_pipeline_edges

These tables store the graph. Node-specific config stays JSON because node types have different shapes.

sql
CREATE TABLE IF NOT EXISTS clone_pipeline_nodes (
  id TEXT PRIMARY KEY,
  clone_id INTEGER NOT NULL,
  kind TEXT NOT NULL CHECK (kind IN ('data_stream', 'asset_selection', 'trading_prompt')),
  name TEXT NOT NULL,
  config_json TEXT NOT NULL,
  position_json TEXT,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

CREATE TABLE IF NOT EXISTS clone_pipeline_edges (
  id TEXT PRIMARY KEY,
  clone_id INTEGER NOT NULL,
  from_node_id TEXT NOT NULL,
  to_node_id TEXT NOT NULL,
  kind TEXT NOT NULL CHECK (kind IN ('provides_context_to', 'selects_assets_for')),
  priority INTEGER NOT NULL DEFAULT 0,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE,
  FOREIGN KEY (from_node_id) REFERENCES clone_pipeline_nodes(id) ON DELETE CASCADE,
  FOREIGN KEY (to_node_id) REFERENCES clone_pipeline_nodes(id) ON DELETE CASCADE
);

clone_pipeline_layouts

Stores canvas-only layout metadata that should round-trip but should not affect graph compilation.

sql
CREATE TABLE IF NOT EXISTS clone_pipeline_layouts (
  clone_id INTEGER PRIMARY KEY,
  layout_json TEXT NOT NULL,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

clone_pipeline_versions also stores layout_json so graph snapshots preserve the visible canvas shape.

Trading Prompt Config

Trading prompt config is stored inside clone_pipeline_nodes.config_json for trading_prompt nodes.

Current workflow fields:

ts
interface TradingPromptConfig {
  enabled: boolean;
  customBehaviorPrompt: string;
  decisionCadenceSec: number | null;
  candidatePolicy: {
    includeOpenPositions: boolean;
    includeUserUniverse: boolean;
    deterministicPrefilter: boolean;
  };
  executionControls: {
    maxNotionalUsdc: number | null;
    cooldownSec: number | null;
    maxTradesPerDay: number | null;
    minConfidencePct: number | null;
  };
  coverageMode: 'global' | 'global_with_asset_overrides' | 'asset_specific_only';
  assetOverrides: Array<{
    symbol: string;
    prompt: Partial<TradingPromptConfig>;
  }>;
}

maxAssetsPerRun can still exist in compatibility config, but v1 does not expose or enforce it. A prompt branch runs once per due enabled asset.

Legacy config fields such as trader_type, analysis_timeframe, and market_momentum_sensitivity may be read during migration, but they should not be the primary v1 workflow controls.

clone_decision_runs / clone_decision_actions

Decision storage is compact. Store status, candidate assets, action indexes, one-line action summaries, and R2 keys in SQLite. Store full prompt/context/response/reason payloads in R2. Decision timestamps are epoch milliseconds.

sql
CREATE TABLE IF NOT EXISTS clone_decision_runs (
  id TEXT PRIMARY KEY,
  clone_id INTEGER NOT NULL,
  branch_id TEXT NOT NULL,
  symbol TEXT NOT NULL,
  status TEXT NOT NULL,
  trigger TEXT NOT NULL,
  scheduled_for INTEGER NOT NULL,
  started_at INTEGER,
  completed_at INTEGER,
  candidate_symbols_json TEXT NOT NULL DEFAULT '[]',
  model TEXT NOT NULL,
  prompt_r2_key TEXT,
  response_r2_key TEXT,
  context_r2_key TEXT,
  error_message TEXT,
  metadata_json TEXT NOT NULL DEFAULT '{}',
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

CREATE TABLE IF NOT EXISTS clone_decision_actions (
  id TEXT PRIMARY KEY,
  run_id TEXT NOT NULL,
  clone_id INTEGER NOT NULL,
  symbol TEXT NOT NULL,
  action TEXT NOT NULL,
  status TEXT NOT NULL,
  confidence REAL NOT NULL,
  quantity REAL,
  notional_usd REAL,
  limit_price REAL,
  reason_summary TEXT,
  reason_r2_key TEXT,
  order_id TEXT,
  error_message TEXT,
  metadata_json TEXT NOT NULL DEFAULT '{}',
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (run_id) REFERENCES clone_decision_runs(id) ON DELETE CASCADE,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

Migration Path

Current state:

  1. Pipeline graph, default generation, compiler, effective context, and latest-data review endpoints are in place.
  2. Decision rows, worker cadence, dry-run debug, prop-account decision runner, and prop-account paper execution are in place.
  3. Frontend Controls still need cleanup against the backend graph/API.
  4. Legacy source-array, active-asset array, and clone-scoped execution paths can be retired after saved workflows have migrated.