Skip to content

Data Source API And UX Spec

This is a reference contract for the product/API layer on top of the SQLite market backend.

For review-level docs, start with:

The current target UX model is:

text
Data Stream Block -> Asset Block -> Trading Prompt Block

The backend stores the same graph shape for every mode:

text
data_stream -> asset_selection -> trading_prompt

Each Trading Prompt Block defines one strategy. The backend expands each strategy across its enabled assets at execution time.

The workflow canvas can additionally show a visual Trading Clone Block:

text
data_stream -> asset_selection -> trading_prompt -> trading_clone

trading_clone is not persisted as a strategy node. It is a UI terminal for clone-level model, trading on/off, and history. Its position and visible prompt-to-clone line are persisted as layout metadata.

Control Granularities

Basic

Basic uses one hidden generated pipeline:

text
Default Data Stream -> Default Asset Selection -> Default Trading Prompt

Controls:

  • Asset category selection.
  • Data depth: Fast, Balanced, or Deep.
  • Optional custom behavior prompt.
  • Clone model.
  • Trading enabled/paused.

No source/channel/granularity/lookback/max-point controls are exposed.

Custom

Custom uses the same graph shape, but exposes editable pieces:

text
Chosen Data Stream -> Individual Asset -> Chosen Trading Prompt

Controls:

  • Individual Asset Blocks selected from a searchable category/subcategory palette.
  • One source-family Data Stream Block per connected asset/source.
  • Retrieval depth: Fast, Balanced, or Deep.
  • Source-level prompt inclusion: Off, Auto, Standard, or Required.
  • Trading prompt cadence, max branch assets, custom behavior prompt, and execution behavior.
  • Latest Data viewer for the selected asset's effective memory reads.

Custom users do not manually tune lookback, granularity, max points, or channel weights.

Power User

Power users can create multiple branches:

text
Deep Hyperliquid Stream -> BTC Asset -> Macro Trading Prompt
Fast Hyperliquid Stream -> DOGE Asset -> Momentum Trading Prompt
Polymarket Stream -> BTC Asset -> Event Trading Prompt

Controls:

  • Create/edit data stream nodes.
  • Create/edit individual asset nodes.
  • Create/edit trading prompt nodes.
  • Connect streams, selections, and prompts.
  • Open advanced channel controls for prompt inclusion, switches, lookback, and granularity.
  • Keep Asset Blocks limited to asset identity, enabled/paused state, latest-data review, and exchange links.
  • Use global prompts, global prompts with per-asset prompt configurations, or asset-specific-only prompts.
  • Configure strategy, risk, and execution behavior on Trading Prompt Blocks.
  • Override decision cadence per trading prompt within backend limits.

Power users still do not use an AI prompt for retrieval. Retrieval stays deterministic.

Asset Naming

Use asset everywhere in product and API language. Use symbol for the exchange-facing identifier.

Examples:

ts
supportedAssets
enabledAssets
assetSelection
assetOverrides
assetGroups
symbol

Avoid assetOrToken. External upstream fields can keep native names, such as Polymarket clobTokenIds.

Asset Selection

Asset selection is user/config bounded before the AI decision runs:

  1. User chooses high-level Hyperliquid categories.
  2. User chooses subcategories where supported.
  3. User can manually include or exclude specific symbols.
  4. The trading engine only considers enabled assets for trades.

Enabled assets receive asset-specific context. Broader market context, such as macro or related-market data, can still be used to evaluate those enabled assets.

Hyperliquid Category Taxonomy

The picker should follow Hyperliquid's visible market taxonomy:

text
High level:
  All
  Perps
  Spot
  Crypto
  Tradfi
  Trending

Crypto:
  All
  AI
  Defi
  Gaming
  Layer 1
  Layer 2
  Meme

Tradfi:
  All
  Stocks
  Indices
  Commodities
  FX
  Pre-IPO

Spot, if exposed:
  All
  USDC
  USDH
  USDT

Out of scope:

  • Generic HIP-3 category exposure.

The official Hyperliquid Info API gives the perp and spot market universes, but not durable category/subcategory tags. Category membership should live in SQLite as asset metadata, seeded from a checked-in taxonomy snapshot and refreshed intentionally.

Data Streams

Data stream nodes define what structured market memory can be retrieved.

Use Fast, Balanced, and Deep for everyone:

  • Basic gets a generated default.
  • Custom chooses profiles for source-family blocks.
  • Power opens advanced channel controls.

Default retrieval tuning is global per data stream/profile. The current workflow prefers individual Asset Blocks rather than category blocks with hidden per-symbol overrides.

Profile definitions include:

  • Enabled source/channel families.
  • Lookback.
  • Granularity.
  • Derived max points.
  • Internally managed context prompt budget.
  • Prompt inclusion policy: off, auto, standard, or required.

Internal prompt priority can still be derived from the preset for context ordering, but it is not a user-facing control.

Override precedence:

text
per-asset configuration
> asset/group profile assignment
> data stream default profile
> built-in profile definition

Basic users do not see advanced channel controls. Custom users choose profiles and source-level prompt inclusion. Power users can configure channel switches, prompt inclusion, lookback, and granularity directly.

Trading Prompt Coverage

Trading prompt nodes support three coverage modes:

text
global
  One prompt applies to all enabled assets.

global_with_asset_overrides
  Global prompt applies by default.
  Specific assets can use their own prompt settings.

asset_specific_only
  No global fallback.
  Only assets explicitly listed in per-asset prompt configurations are eligible for this prompt node.

Compiler rule:

text
specific asset prompt > global prompt

If two asset-specific prompts claim the same asset inside one compiled branch, reject the graph as invalid in v1.

Source Coverage

Initial source/channel catalog defaults should mirror backend cadence:

  • Hyperliquid mid prices: 5s allMids buckets for short-horizon context.
  • Hyperliquid OHLC candles: 60s source rows from candle WebSocket subscriptions; larger windows derived locally.
  • Hyperliquid liquidity profile: 300s buckets.
  • Hyperliquid funding pressure: 300s buckets.
  • Hyperliquid volatility profile: derived, 300s buckets.
  • Hyperliquid positioning pressure: derived, 300s buckets.
  • Polymarket market discovery: registry channel, 3600s cadence.
  • Polymarket immediate market: 300s cadence.
  • Polymarket recently closed markets: 900s cadence.
  • Polymarket later market: 1800s cadence.
  • DefiLlama capital base, capital flows, economic throughput: 3600s cadence.

Subject Model

Use a subject model for context reads.

ts
type DataSubjectType =
  | 'asset'
  | 'market'
  | 'chain'
  | 'protocol'
  | 'stablecoin'
  | 'global';

interface DataSubject {
  type: DataSubjectType;
  id: string;
  displayName?: string;
  linkedAssetSymbol?: string;
}

Examples:

json
{ "type": "asset", "id": "BTC", "displayName": "Bitcoin" }
{ "type": "chain", "id": "solana", "displayName": "Solana", "linkedAssetSymbol": "SOL" }
{ "type": "global", "id": "defi", "displayName": "DeFi Market" }
{ "type": "stablecoin", "id": "USDC", "displayName": "USDC" }

Hyperliquid is mostly asset-scoped. Polymarket can be asset-linked but should also support market/event subjects later. DefiLlama should be macro-scoped: global, chain, protocol, and stablecoin.

Pipeline Graph API

http
GET /api/v1/clones/:cloneId/pipeline
PUT /api/v1/clones/:cloneId/pipeline
POST /api/v1/clones/:cloneId/pipeline/preview

Shape:

ts
type PipelineNodeKind = 'data_stream' | 'asset_selection' | 'trading_prompt';
type PipelineEdgeKind = 'provides_context_to' | 'selects_assets_for';

interface ClonePipelineConfig {
  version: 1;
  cloneId: number;
  nodes: Array<{
    id: string;
    kind: PipelineNodeKind;
    name: string;
    config: Record<string, unknown>;
    position?: { x: number; y: number };
  }>;
  edges: Array<{
    id: string;
    fromNodeId: string;
    toNodeId: string;
    kind: PipelineEdgeKind;
    priority: number;
  }>;
  layout?: {
    strategyClone?: {
      position?: { x: number; y: number };
      connectedPromptNodeIds?: string[];
    };
  };
}

The backend validates v1 graphs:

text
data_stream -> asset_selection
asset_selection -> trading_prompt

The visual trading_prompt -> trading_clone route is not persisted in edges. It round-trips through layout.strategyClone.connectedPromptNodeIds, so the canvas can keep the line visible without changing the executable graph.

Preview compiles the graph into enabled assets, data read plans, estimated prompt tokens, warnings, and the trading prompt payload without creating a decision run. It also returns an effectiveContext preview packet with branch candidate assets and structured SQLite memory reads. Execution expands those candidates into one prompt run per due asset.

During development these endpoints accept x-user-id as the backend-owned user id. Production auth should map external providers into users.id before route handlers run.

Asset Latest Data API

http
POST /api/v1/clones/:cloneId/assets/:symbol/latest-data

The Latest Data endpoint is a read-only review endpoint for the workflow canvas. It accepts an optional pipeline draft and returns the effective memory reads for one asset using the same context resolver path as trading decisions.

For DefiLlama reads, the resolver filters relevant subjects before applying maxPoints: global defi, matching chain rows by metrics.tokenSymbol, and matching stablecoin subjects by symbol. This means BTC can show multiple global/Bitcoin rows across the lookback window even though DefiLlama is macro-scoped rather than a direct per-token feed.

Shape:

ts
interface AssetLatestDataResponse {
  symbol: string;
  asOfMs: number;
  branches: Array<{
    id: string;
    dataStreamNodeId: string;
    assetSelectionNodeId: string;
    tradingPromptNodeId: string;
    candidateSymbols: string[];
    memoryReads: Array<{
      key: string;
      symbol: string;
      source: 'hyperliquid' | 'polymarket' | 'defillama';
      channel: string;
      mode: 'latest' | 'timeseries' | 'summary' | 'signal';
      granularitySec: number;
      lookbackSec: number;
      maxPoints: number;
      required: boolean;
      recordCount: number;
      records: Array<Record<string, unknown>>;
    }>;
    warnings?: string[];
  }>;
  warnings?: string[];
}

Ingestion Health API

http
GET /api/v1/ingestion/health

Returns source/channel health rows from ingestion_runs for admin UI and host-level monitoring. ok is false if any expected channel is missing, failed, or stale.

Asset Selection Projection

The pipeline graph is the canonical source of truth. If Basic/Custom flows need a simpler asset-selection API later, it should be a projection that reads and writes generated asset_selection graph nodes.

Potential later endpoints:

http
GET/PUT /api/v1/clones/:cloneId/asset-selection
GET     /api/v1/clones/:cloneId/trade-universe

Shape:

ts
interface AssetSelectionConfig {
  version: 1;
  cloneId: number;
  source: 'hyperliquid';
  highLevelCategories: Array<'all' | 'perps' | 'spot' | 'crypto' | 'tradfi' | 'trending'>;
  subcategories: {
    crypto?: string[];
    tradfi?: string[];
    spot?: string[];
  };
  explicitlyEnabledSymbols: string[];
  explicitlyDisabledSymbols: string[];
}

The workflow canvas currently uses the pipeline graph plus GET /api/v1/assets for the builder palette.

Execution Config Projection

The pipeline graph and clone row are the canonical sources of truth. If simple controls need an execution-config API later, it should be a projection over:

  • the clone row for model and active/paused state;
  • the active Trading Prompt Block for cadence, max branch assets, custom behavior prompt, strategy, risk, and execution behavior;
  • the relevant Asset Blocks for enabled asset identity.

Shape:

ts
interface CloneExecutionConfig {
  version: 1;
  cloneId: number;
  model: string;
  status: 'active' | 'paused' | 'inactive' | 'archived';
  customBehaviorPrompt: string;
  decisionCadenceSec?: number;
  maxAssetsPerRun: number;
  candidatePolicy: {
    includeOpenPositions: boolean;
    includeUserUniverse: boolean;
    deterministicPrefilter: boolean;
  };
}

Legacy migration fields such as trader archetype, market momentum sensitivity, and analysis timeframe can still be carried in compatibility paths, but they should not be treated as primary workflow controls.

Default scheduled execution is every 5 minutes per enabled clone/branch/asset.

Recommended v1 cadence:

text
Default: every 5 minutes
Optional slower prompt cadences: 15 minutes, 30 minutes, 2 hours, 6 hours

Candidate Asset Policy

The platform can support every Hyperliquid asset, but the LLM should not evaluate all assets on every run by default.

Separate three concepts:

  • Supported assets: everything the market backend knows about.
  • Trade universe: assets the user has allowed this clone to trade.
  • Run candidates: assets included in this specific decision call.

Default candidates:

  • Assets with open positions.
  • Assets in the user-enabled trade universe.
  • Assets passing deterministic prefilters, capped to a safe maximum.

Recommended branch candidate cap before per-asset execution:

text
Max candidate assets per strategy branch: 10 default, 25 advanced maximum

For a power user who wants to watch hundreds of assets, the system should scan structured data deterministically first, then send only the strongest candidates to the model.

Decision Worker

The worker operates on compiled pipeline branches:

text
active clone
  -> effective context
  -> cadence-due asset
  -> per-asset decision run row
  -> decision engine
  -> validated action rows

The local worker command defaults to a no-op hold engine so the storage and cadence path can be tested without placing trades or requiring model credentials.

bash
npm run decisions:run-once -- --clone-id=42 --force
npm run decisions:worker

Set DECISION_MODEL_PROVIDER=anthropic plus ANTHROPIC_API_KEY to use the model-backed dry-run engine. It calls Claude, expects strict JSON actions, validates them against the candidate asset and configured execution controls, and stores the action rows without placing trades. Order execution is a later handoff.

The dry-run execution layer now records validated executable actions as clone_execution_orders with status dry_run and writes validation/rejection events to clone_execution_ledger. Fills and positions are schema-ready but should stay untouched until paper execution is implemented.

SQLite Configuration Tables

Target state: SQLite owns backend state end to end.

Core tables:

  • clones
  • clone_asset_selection_rules
  • clone_trade_universe
  • clone_pipeline_nodes
  • clone_pipeline_edges
  • clone_pipeline_layouts
  • clone_pipeline_versions
  • clone_execution_configs
  • clone_decision_runs
  • clone_decision_actions
  • orders, fills, positions, and ledger entries

clone_asset_selection_rules and clone_execution_configs are helper/projection tables. They should not become competing configuration systems. The canonical editable graph remains clone_pipeline_nodes plus clone_pipeline_edges.

clone_asset_selection_rules

Preserves user category/subcategory intent:

sql
CREATE TABLE IF NOT EXISTS clone_asset_selection_rules (
  clone_id INTEGER PRIMARY KEY,
  source TEXT NOT NULL DEFAULT 'hyperliquid',
  high_level_categories_json TEXT NOT NULL DEFAULT '["crypto"]',
  subcategories_json TEXT NOT NULL DEFAULT '{"crypto":["all"]}',
  explicitly_enabled_symbols_json TEXT NOT NULL DEFAULT '[]',
  explicitly_disabled_symbols_json TEXT NOT NULL DEFAULT '[]',
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

clone_trade_universe

Stores expanded effective assets after selection rules and explicit symbol selections:

sql
CREATE TABLE IF NOT EXISTS clone_trade_universe (
  clone_id INTEGER NOT NULL,
  symbol TEXT NOT NULL,
  display_name TEXT,
  enabled INTEGER NOT NULL DEFAULT 1,
  source TEXT NOT NULL DEFAULT 'hyperliquid',
  selection_source TEXT NOT NULL DEFAULT 'category_rule',
  sort_order INTEGER,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  PRIMARY KEY (clone_id, symbol),
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

clone_pipeline_nodes / clone_pipeline_edges

These tables store the graph. Node-specific config stays JSON because node types have different shapes.

sql
CREATE TABLE IF NOT EXISTS clone_pipeline_nodes (
  id TEXT PRIMARY KEY,
  clone_id INTEGER NOT NULL,
  kind TEXT NOT NULL CHECK (kind IN ('data_stream', 'asset_selection', 'trading_prompt')),
  name TEXT NOT NULL,
  config_json TEXT NOT NULL,
  position_json TEXT,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

CREATE TABLE IF NOT EXISTS clone_pipeline_edges (
  id TEXT PRIMARY KEY,
  clone_id INTEGER NOT NULL,
  from_node_id TEXT NOT NULL,
  to_node_id TEXT NOT NULL,
  kind TEXT NOT NULL CHECK (kind IN ('provides_context_to', 'selects_assets_for')),
  priority INTEGER NOT NULL DEFAULT 0,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE,
  FOREIGN KEY (from_node_id) REFERENCES clone_pipeline_nodes(id) ON DELETE CASCADE,
  FOREIGN KEY (to_node_id) REFERENCES clone_pipeline_nodes(id) ON DELETE CASCADE
);

clone_pipeline_layouts

Stores canvas-only layout metadata that should round-trip but should not affect graph compilation.

sql
CREATE TABLE IF NOT EXISTS clone_pipeline_layouts (
  clone_id INTEGER PRIMARY KEY,
  layout_json TEXT NOT NULL,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

clone_pipeline_versions also stores layout_json so graph snapshots preserve the visible canvas shape.

Trading Prompt Config

Trading prompt config is stored inside clone_pipeline_nodes.config_json for trading_prompt nodes.

Current workflow fields:

ts
interface TradingPromptConfig {
  customBehaviorPrompt: string;
  decisionCadenceSec: number | null;
  maxAssetsPerRun: number;
  candidatePolicy: {
    includeOpenPositions: boolean;
    includeUserUniverse: boolean;
    deterministicPrefilter: boolean;
  };
  coverageMode: 'global' | 'global_with_asset_overrides' | 'asset_specific_only';
  assetOverrides: Array<{
    symbol: string;
    prompt: Partial<TradingPromptConfig>;
  }>;
}

Legacy config fields such as trader_type, analysis_timeframe, and market_momentum_sensitivity may be read during migration, but they should not be the primary v1 workflow controls.

clone_decision_runs / clone_decision_actions

Decision storage is compact. Store status, candidate assets, action indexes, one-line action summaries, and R2 keys in SQLite. Store full prompt/context/response/reason payloads in R2. Decision timestamps are epoch milliseconds.

sql
CREATE TABLE IF NOT EXISTS clone_decision_runs (
  id TEXT PRIMARY KEY,
  clone_id INTEGER NOT NULL,
  branch_id TEXT NOT NULL,
  symbol TEXT NOT NULL,
  status TEXT NOT NULL,
  trigger TEXT NOT NULL,
  scheduled_for INTEGER NOT NULL,
  started_at INTEGER,
  completed_at INTEGER,
  candidate_symbols_json TEXT NOT NULL DEFAULT '[]',
  model TEXT NOT NULL,
  prompt_r2_key TEXT,
  response_r2_key TEXT,
  context_r2_key TEXT,
  error_message TEXT,
  metadata_json TEXT NOT NULL DEFAULT '{}',
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

CREATE TABLE IF NOT EXISTS clone_decision_actions (
  id TEXT PRIMARY KEY,
  run_id TEXT NOT NULL,
  clone_id INTEGER NOT NULL,
  symbol TEXT NOT NULL,
  action TEXT NOT NULL,
  status TEXT NOT NULL,
  confidence REAL NOT NULL,
  quantity REAL,
  notional_usd REAL,
  limit_price REAL,
  reason_summary TEXT,
  reason_r2_key TEXT,
  order_id TEXT,
  error_message TEXT,
  metadata_json TEXT NOT NULL DEFAULT '{}',
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (run_id) REFERENCES clone_decision_runs(id) ON DELETE CASCADE,
  FOREIGN KEY (clone_id) REFERENCES clones(id) ON DELETE CASCADE
);

Migration Path

  1. Add pipeline node/edge repository and graph validator.
  2. Add default pipeline generation for Basic mode.
  3. Add pipeline preview/compiler.
  4. Add effective context resolver and preview context packets.
  5. Add Latest Data review endpoint.
  6. Add execution decision runs/actions.
  7. Add scheduler/worker loop that creates runs, calls a pluggable decision engine, validates actions, and records outcomes.
  8. Add dry-run order/ledger storage.
  9. Add paper fill simulation and account/position mutation.
  10. Migrate frontend Controls to the backend API.
  11. Retire legacy source-array and active-asset array paths after saved workflows have migrated.