192 lines
13 KiB
Markdown
192 lines
13 KiB
Markdown
# Simulation Implementation Phases
|
|
|
|
Status: implementation plan for `specs/simulation/simulation.md`.
|
|
|
|
The full simulation architecture is intentionally broad. This document breaks it into phases that can be implemented and reviewed incrementally.
|
|
|
|
## Phase 1: Control Surface And Observability
|
|
|
|
Goal: start the normal app in simulation mode and inspect/drive the TUI through an external WebSocket driver.
|
|
|
|
This phase proves the core shape without swapping every foundational layer yet.
|
|
|
|
Implementation checklist:
|
|
|
|
- [x] Add `OPENCODE_DRIVE=<name>` activation in V1/full-TUI startup.
|
|
- [x] Add simulation trace service with in-memory append-only records.
|
|
- [x] Add OpenTUI UI state extraction for screen, focus, elements, and generated actions.
|
|
- [x] Add OpenTUI UI action execution for typing, keys, enter, arrows, focus, and click.
|
|
- [x] Add reusable JSON-RPC WebSocket server at the manifest's UI endpoint.
|
|
- [x] Expose `ui.state`, `ui.action`, `ui.render`.
|
|
- [x] Expose `trace.list`, `trace.clear`, `trace.export`.
|
|
- [x] Wire visible V1/full-TUI renderer path through the same action protocol.
|
|
- [ ] Verify a local driver can inspect state and execute a real TUI input.
|
|
|
|
Scope:
|
|
|
|
- Add `OPENCODE_DRIVE=<name>` activation.
|
|
- Start a TUI-owned JSON-RPC WebSocket server at the manifest's UI endpoint.
|
|
- Expose `ui.state`, `ui.action`, `ui.render`.
|
|
- Use the old simulation action model: type text, press keys, press enter, arrows, focus, click.
|
|
- Support fake OpenTUI renderer and visible renderer through the same action protocol.
|
|
- Add in-memory append-only trace with `trace.list`, `trace.clear`, `trace.export`.
|
|
- Record UI observations, generated actions, executed actions, errors, and render/stabilization events.
|
|
|
|
Done when:
|
|
|
|
- `OPENCODE_DRIVE=<name> bun run dev` starts the normal app and UI drive server.
|
|
- A local driver can connect to the WebSocket.
|
|
- The driver can inspect current screen/elements/actions.
|
|
- The driver can execute real TUI inputs.
|
|
- The trace shows observations and actions.
|
|
|
|
Out of scope:
|
|
|
|
- Backend layer replacement.
|
|
- Model-based runner.
|
|
- Generated plugin config.
|
|
- Deterministic replay tests.
|
|
|
|
## Phase 2: Foundational Simulation Layers
|
|
|
|
Goal: make the app safe and controlled by swapping the lowest layers, not app logic.
|
|
|
|
Implementation checklist:
|
|
|
|
- [x] Add `packages/simulation/src/backend` as the home for backend simulation layer replacements, exported from `backend/index.ts` as `simulationReplacements`; `@opencode-ai/simulation` is private/non-published and depends on logic/framework packages (`core`, `llm`, `effect`, OpenTUI), while `server` and `tui` consume it.
|
|
- [x] Wire simulation replacements through the server's `makeRoutes` via `Layer.unwrap` + dynamic `import("@opencode-ai/simulation/backend")` gated on `OPENCODE_SIMULATE`, so the simulation module is never loaded eagerly and `makeRoutes` stays synchronous.
|
|
- [x] Implement in-memory `FileSystem.FileSystem` (`simulation/filesystem.ts`) replacing the `NodeFileSystem` platform node. Backed by a flat path map; implements the operations the app uses (stat, access, chmod, realPath, read/write file, make/read directory, remove, rename, copy, copyFile, temp dirs, read-only open handles); unused operations die with a clear defect; `watch` fails as unsupported.
|
|
- [x] Root the fake filesystem at `process.cwd()` at layer-build time. The anchor is a real, empty host directory the runner creates and cds into.
|
|
- [x] Deny host filesystem escapes loudly: content/mutation operations outside the root fail with `PermissionDenied` simulation errors. Probe operations (`stat`/`access`/`exists`) report `NotFound` outside the root so walk-up loops (project discovery, `findUp`, `globUp`) terminate naturally.
|
|
- [x] Add `SimulationFSUtil` replacement (`simulation/fs-util.ts`): wraps the real `FSUtil` layer and reroutes `readDirectoryEntries`, `glob`, and `globUp` — which bypass the injected `FileSystem` via node `fs/promises` and the `glob` package — through the simulated filesystem.
|
|
- [x] Fix `LayerNode.hoist` conflict detection to compare node implementations instead of object identity; replacement rewriting produces dependency-rewritten copies of the same node, which previously false-positived as "conflicting implementations".
|
|
- [x] Add snapshot seeding from `OPENCODE_SIMULATE_STATE`: `files/` contents of the snapshot directory are read from the host once at layer-build time and seeded into the in-memory tree joined onto the anchor root.
|
|
- [x] Verify end to end: `opencode serve` boots with `OPENCODE_SIMULATE=1` + `OPENCODE_SIMULATE_STATE` + path/DB env seams (`OPENCODE_CONFIG_DIR`, `OPENCODE_TEST_HOME`, `OPENCODE_DB=:memory:`); `fs.list`/`fs.read` observe only seeded in-memory files; the anchor directory on the host remains empty after the run.
|
|
- [ ] Create the anchor directory + `chdir` + env seam setup automatically in CLI startup when simulation mode is enabled (currently set manually by the runner; a full run needs `OPENCODE_SIMULATE_STATE`, `OPENCODE_CONFIG_DIR`, `OPENCODE_TEST_HOME`, `OPENCODE_DB=:memory:`, and `XDG_*_HOME` pointed into the anchor, plus Bun's `--preload=@opentui/solid/preload` when launched outside `packages/cli`).
|
|
- [ ] Assert the anchor directory is still empty at the end of the run (KV/log/flock still write through real XDG paths; they are contained in the anchor by the env seams but not yet in-memory).
|
|
- [x] Add simulated network registry (`packages/simulation/src/backend/network.ts`): replaces the `httpClient` platform node, resolves all outbound HTTP against an in-memory route table, denies unknown destinations loudly, and keeps a bounded request log (design: `simulated-network-llm.md`).
|
|
- [x] Add driver-answered LLM as an OpenAI route in the simulated network (`openai.ts` + `llm-exchange.ts`): provider requests open exchanges; the driver streams chunks back which are encoded as real OpenAI Chat SSE (schema-checked against `OpenAIChatEvent`) and consumed by the real protocol pipeline. No enqueue store — the driver is the model.
|
|
- [x] Add backend-hosted drive control WebSocket (`control.ts`): JSON-RPC at the named manifest's backend endpoint, started when `OPENCODE_DRIVE` is set. Drivers connect directly (standalone topology — no frontend proxy): `llm.attach` (replays pending exchanges), `llm.chunk`, `llm.finish`, `llm.pending`, `network.log`; `llm.request` notifications push opened exchanges. This is also the headless-simulation interface. Drivers manage the manifest's UI endpoint for UI control and backend endpoint for LLM/network control.
|
|
- [x] Answer `https://models.dev/api.json` with an empty catalog in the simulated network; providers come from seeded config (`opencode.json` in the snapshot defines an openai-compatible provider with a dummy `apiKey`, which passes the catalog availability gate and resolves onto the real openai-chat route).
|
|
- [x] Fix `buildLocationServiceMap` to apply replacements when compiling hoisted global nodes; platform-node replacements (filesystem, httpClient) were silently ignored inside hoisted globals.
|
|
- [x] Verify end to end headless (real route stack in-process + backend control WS: prompt -> `llm.request` -> driver chunks -> assistant message contains driver text; script: `packages/server/script/e2e-sim.ts`) and through the TUI (fake renderer, both sockets: type + submit via TUI WS, answer `llm.request` via backend WS, assistant reply rendered on screen; script: `packages/tui/script/sim-llm-driver.ts`).
|
|
- [ ] Add simulated process registry (shell via `just-bash`, minimal fake `git`, deny unsupported spawns).
|
|
- [ ] Trace filesystem, process, and LLM exchange activity (network requests are traced in the backend network log ring buffer; LLM exchange trace records moved out with the frontend proxy and need re-adding on the backend control server).
|
|
|
|
Scope:
|
|
|
|
- Wire simulation replacements through `AppNodeBuilder.build(...)` and `AppNodeBuilderV1.build(...)`.
|
|
- Create a real, empty anchor directory (`mkdtemp`) and `process.chdir` into it before any command resolves its working directory; skip creation when the runner already spawned the app inside an anchor.
|
|
- Root the in-memory filesystem at `process.cwd()` (the anchor). No cwd monkey-patching: cwd, `$PWD`, and `path.resolve()` stay truthful.
|
|
- Add snapshot loading from `OPENCODE_SIMULATE_STATE`: read the snapshot directory once at startup and seed the in-memory filesystem (snapshot `files/` paths joined onto the anchor root), config, env, and optional LLM/network state from it.
|
|
- Route config/data/state/cache/temp paths into the simulated space using existing env seams (`OPENCODE_CONFIG_DIR`, `OPENCODE_TEST_HOME`, `OPENCODE_DB=:memory:`), set before `packages/core/src/global.ts` import-time path setup runs.
|
|
- Deny host filesystem escapes loudly (paths outside the anchor root fail with typed simulation errors).
|
|
- Assert the anchor directory on the host is still empty at the end of the run; anything written there means a code path bypassed the simulated filesystem.
|
|
- Add simulated network registry and deny unknown external network by default.
|
|
- Add scriptable LLM boundary.
|
|
- Add simulated process registry:
|
|
- shell through `just-bash` against the simulated filesystem.
|
|
- minimal fake `git` support for discovery/status paths.
|
|
- deny unsupported process spawns.
|
|
- Add simulation-gated backend control routes, proxied only through the frontend WebSocket.
|
|
- Expose backend methods through the frontend server: filesystem seed/write, network register, LLM enqueue, backend snapshot.
|
|
- Trace filesystem, network, LLM, process, and backend control activity.
|
|
|
|
Done when:
|
|
|
|
- Unknown network fails with a simulation error.
|
|
- Host filesystem escape fails with a simulation error.
|
|
- The anchor directory on the host is empty after a run.
|
|
- The app boots from a snapshot directory via `OPENCODE_SIMULATE_STATE` and observes the seeded project files, config, and env through normal app paths.
|
|
- A driver can seed a project filesystem.
|
|
- A driver can enqueue an LLM script and submit a prompt through the TUI.
|
|
- The real session/tool path consumes the scripted LLM behavior.
|
|
- Shell commands use `just-bash`; unsupported process spawns fail.
|
|
- Trace contains backend activity and snapshots.
|
|
|
|
Out of scope:
|
|
|
|
- Model-based generation.
|
|
- Generated plugin config state.
|
|
- Shrinking.
|
|
|
|
## Phase 3: Generated Config And Model-Based Runner
|
|
|
|
Goal: explore different app states using generated commands and plugin-provided config state.
|
|
|
|
Scope:
|
|
|
|
- Add generated simulation plugins as the primary config-state generation mechanism.
|
|
- Support generated plugin domains for:
|
|
- agents and defaults.
|
|
- provider/model availability.
|
|
- tool definitions and scripted tool behavior.
|
|
- MCP-like capabilities or endpoints.
|
|
- permission policies.
|
|
- instructions/system-context-like inputs where supported.
|
|
- workspace/project adapters where supported.
|
|
- Add runner commands to generate, enable, disable, and inspect generated plugin state.
|
|
- Build a custom external model-based runner, not `fast-check` yet.
|
|
- Runner command shape: precondition, execute, model update, postcondition.
|
|
- Runner model tracks only high-level observational state: screen category, prompt availability, sessions, files, queued LLM scripts, generated plugins, backend status, idle expectation.
|
|
- Generate valid command sequences from model state and current `ui.state.actions`.
|
|
- Record seed, command distribution, precondition rejections, generated plugin/config domain coverage, UI action coverage, and backend event coverage.
|
|
|
|
Done when:
|
|
|
|
- A seeded runner can generate a short valid exploration.
|
|
- The runner can generate plugin-provided config state without generating large arbitrary config files.
|
|
- The app loads and observes generated plugin state through normal plugin/config paths.
|
|
- The runner can type and submit prompts through the TUI using generated actions.
|
|
- Basic properties run after commands: no crash, no unknown network, no host FS escape, coherent stabilized state.
|
|
- Trace export includes enough state to replay the generated run later.
|
|
|
|
Out of scope:
|
|
|
|
- Shrinking.
|
|
- Coverage-guided mutation corpus.
|
|
- Differential testing.
|
|
- CI randomized runs.
|
|
|
|
## Phase 4: Replay, Promotion, And Campaigns
|
|
|
|
Goal: turn exploratory simulation into durable tests and prepare for larger campaigns.
|
|
|
|
Scope:
|
|
|
|
- Add replay from exported trace.
|
|
- Add deterministic replay test generation from successful or failing traces.
|
|
- Add stronger trace schema validation.
|
|
- Add property families beyond no-crash:
|
|
- durable prompt admission is not lost.
|
|
- no duplicated visible message IDs.
|
|
- no orphan tool results.
|
|
- queue/steer semantics hold at stabilization boundaries.
|
|
- interrupt/resume does not duplicate promoted inputs.
|
|
- Add corpus storage for interesting traces.
|
|
- Add simple coverage/novelty scoring over UI states, backend event types, tool outcomes, generated config domains, and errors.
|
|
- Add long-running campaign mode outside normal CI.
|
|
|
|
Done when:
|
|
|
|
- A trace from Phase 3 can be replayed deterministically.
|
|
- A trace can be promoted to a normal test fixture.
|
|
- Campaign runs can collect interesting traces without committing randomized tests to CI.
|
|
- Failures produce a compact reproduction command and trace export.
|
|
|
|
Out of scope:
|
|
|
|
- Full shrinking.
|
|
- Deterministic scheduler/clock control.
|
|
- Parallel campaigns.
|
|
- Differential testing across app versions.
|
|
|
|
## Later Work
|
|
|
|
- Shrinking failed traces.
|
|
- Coverage-guided mutation of structured traces.
|
|
- `fast-check` integration if the custom runner becomes too limited.
|
|
- Differential testing across versions, renderers, storage modes, or scheduler policies.
|
|
- Deterministic clock/random/scheduler control.
|
|
- Parallel isolated workers.
|
|
- Model-generated properties with validity/soundness/coverage scoring.
|