152 lines
9.3 KiB
Markdown
152 lines
9.3 KiB
Markdown
|
# Simulated Network And Driver-Scripted LLM
|
||
|
|
|
||
|
|
Status: design for the Phase 2 network and LLM items in `simulation-phases.md`.
|
||
|
|
|
||
|
|
## Summary
|
||
|
|
|
||
|
|
Simulation replaces the `HttpClient.HttpClient` platform node with a simulated network. The LLM is not a separate fake: it is one registered route in that network (`api.openai.com`), answered by the **external driver** over the existing control WebSocket. When the app issues a provider request, the backend forwards it to the driver and the driver streams response chunks back. There is no enqueueing and no scripted-response store; the driver is the model.
|
||
|
|
|
||
|
|
Everything above the HTTP boundary runs real: catalog and auth resolution, `LLMClient`, request body construction, SSE framing, the OpenAI protocol event schema, the `step` state machine, `Lifecycle` grammar, tool-argument accumulation, the session runner, tools, and permissions.
|
||
|
|
|
||
|
|
## Why the network seam
|
||
|
|
|
||
|
|
`LLMClient.stream` sits on a stack that ends in one platform node:
|
||
|
|
|
||
|
|
```
|
||
|
|
LLMClient.stream(request)
|
||
|
|
route.body.from LLMRequest -> OpenAI JSON body (real)
|
||
|
|
transport.prepare body + endpoint + auth -> HttpRequest (real)
|
||
|
|
RequestExecutor.execute status/error taxonomy (real)
|
||
|
|
HttpClient.HttpClient <- replaced by the simulated network
|
||
|
|
Framing.sse bytes -> frames (real)
|
||
|
|
protocol.stream.event frame -> OpenAIChatEvent, validated (real)
|
||
|
|
protocol.stream.step state machine -> LLMEvents (real)
|
||
|
|
```
|
||
|
|
|
||
|
|
Replacing `httpClient` (already a `LayerNode` in `app-node-platform.ts`, already used by `simulationReplacements` mechanics) keeps the entire pipeline under test and gives wire-fidelity observation of what would have been sent to the provider. Failure injection (429s, malformed SSE, truncated streams) exercises real error paths that a typed `LLMClient` fake cannot reach.
|
||
|
|
|
||
|
|
## Components
|
||
|
|
|
||
|
|
### 1. Simulated network (`packages/server/src/simulation/network.ts`)
|
||
|
|
|
||
|
|
Replaces `httpClient` in `simulationReplacements`. An in-memory route table:
|
||
|
|
|
||
|
|
- `register(matcher, responder)` where matcher is method + URL pattern and responder is `(HttpClientRequest) => Effect<HttpClientResponse>`.
|
||
|
|
- Unknown requests fail loudly with a typed simulation error (spec: deny unknown external network by default).
|
||
|
|
- Optional loopback allowance for the app's own server is not required server-side (the server does not call itself over HTTP); revisit if a consumer needs it.
|
||
|
|
- Every request/response summary is traced.
|
||
|
|
|
||
|
|
### 2. OpenAI endpoint route (`packages/server/src/simulation/openai.ts`)
|
||
|
|
|
||
|
|
Registered in the network at startup for `POST {DEFAULT_BASE_URL}{PATH}` from `protocols/openai-chat.ts` (`https://api.openai.com/v1/chat/completions`).
|
||
|
|
|
||
|
|
On request:
|
||
|
|
|
||
|
|
1. Allocate an exchange id. Parse the real OpenAI request body (available to the driver for assertions).
|
||
|
|
2. Publish a `request` record to the LLM exchange service (below) and create a chunk `Queue`.
|
||
|
|
3. Return `HttpClientResponse` with `content-type: text/event-stream` whose body stream reads from the queue, encoding each item as an SSE `data:` frame, terminated by `[DONE]`.
|
||
|
|
|
||
|
|
Chunks are constructed through the `OpenAIChatEvent` schema so drift in the protocol schema breaks the build, not the runtime.
|
||
|
|
|
||
|
|
The response stream is interruptible like a real HTTP response: if the runner cancels (user interrupt), the exchange closes and the driver is notified.
|
||
|
|
|
||
|
|
### 3. LLM exchange service (`packages/server/src/simulation/llm-exchange.ts`)
|
||
|
|
|
||
|
|
Process-global simulation service owning pending exchanges:
|
||
|
|
|
||
|
|
```
|
||
|
|
Exchange = { id, body, queue: Queue<Item | Error | Done>, deferred lifecycle }
|
||
|
|
```
|
||
|
|
|
||
|
|
- `requests()` — stream of newly opened exchanges (consumed by the control route).
|
||
|
|
- `push(id, item)` — append one response item to an open exchange.
|
||
|
|
- `finish(id, reason)` / `fail(id, failure)` — terminate the exchange.
|
||
|
|
- Exchanges that receive no driver within a configurable timeout fail the provider request with a simulation error (surfaces in the real provider-error path).
|
||
|
|
|
||
|
|
### 4. Backend control routes (simulation-gated, private)
|
||
|
|
|
||
|
|
Mounted only when `OPENCODE_SIMULATION` is set. Not for external use; the frontend simulation server proxies them (spec: external drivers connect only to the frontend WebSocket).
|
||
|
|
|
||
|
|
- `GET /experimental/simulation/llm/requests` — SSE stream of opened exchanges `{id, body}`.
|
||
|
|
- `POST /experimental/simulation/llm/:id/chunk` — append items.
|
||
|
|
- `POST /experimental/simulation/llm/:id/finish` — `{reason}`.
|
||
|
|
- `POST /experimental/simulation/llm/:id/fail` — `{status, body}` for failure injection (HTTP-level: the exchange responds with that status instead of SSE).
|
||
|
|
- `GET /experimental/simulation/network/log` — traced network activity.
|
||
|
|
|
||
|
|
### 5. Frontend WebSocket protocol (TUI control server)
|
||
|
|
|
||
|
|
The existing JSON-RPC server in `packages/tui/src/simulation/server.ts` gains LLM proxying. The TUI simulation module subscribes to the backend `llm/requests` SSE using its normal server connection and forwards over the WebSocket.
|
||
|
|
|
||
|
|
New server -> driver notification:
|
||
|
|
|
||
|
|
```
|
||
|
|
{ "jsonrpc": "2.0", "method": "llm.request",
|
||
|
|
"params": { "id": "ex_1", "model": "gpt-...", "body": { ...openai request body... } } }
|
||
|
|
```
|
||
|
|
|
||
|
|
New driver -> server methods (proxied to the backend routes):
|
||
|
|
|
||
|
|
```
|
||
|
|
llm.chunk { id, items: Item[] }
|
||
|
|
llm.finish { id, reason: "stop" | "tool-calls" | "length" | ... }
|
||
|
|
llm.fail { id, status, body? }
|
||
|
|
```
|
||
|
|
|
||
|
|
`Item` is the response vocabulary the driver speaks:
|
||
|
|
|
||
|
|
```
|
||
|
|
{ type: "textDelta", id, text }
|
||
|
|
{ type: "reasoningDelta", id, text }
|
||
|
|
{ type: "toolCall", id, name, input }
|
||
|
|
{ type: "raw", chunk } // escape hatch: raw OpenAIChatEvent JSON
|
||
|
|
```
|
||
|
|
|
||
|
|
The backend compiles items to OpenAI chunks (`delta.content`, `delta.tool_calls[].function.arguments`, `finish_reason`); `raw` passes through schema validation only. Streaming granularity is the driver's choice: many small `llm.chunk` calls stream word by word; one call with many items plus `llm.finish` responds at once.
|
||
|
|
|
||
|
|
Driver connection lifecycle: `llm.request` notifications are sent to control connections that have called `llm.attach`. If no driver is attached, exchanges fail after the timeout. Multiple drivers are out of scope; last attach wins.
|
||
|
|
|
||
|
|
### 6. Pacing and the clock
|
||
|
|
|
||
|
|
No server-side pacing by default: the driver controls timing by when it sends chunks, which is the point of driver-in-the-loop. A convenience `llm.chunk` option `{ delayMs }` may sleep via `Effect.sleep` between items server-side; because that uses the fiber `Clock`, scoping a controllable clock to the exchange stream (`Stream.provideService(Clock.Clock, simClock)`) remains available for deterministic replay without touching app time. Defer until replay work needs it.
|
||
|
|
|
||
|
|
### 7. Catalog and auth seeding
|
||
|
|
|
||
|
|
The driver-facing model must be selectable in the TUI. Simulation seeds config (via the snapshot filesystem) defining a provider on the openai-chat route with `baseURL` left at the OpenAI default and a dummy `apiKey` (satisfies `Catalog.available()`). No catalog code changes.
|
||
|
|
|
||
|
|
## End-to-end flow
|
||
|
|
|
||
|
|
```
|
||
|
|
driver TUI sim server backend
|
||
|
|
| | |
|
||
|
|
|-- ui.action (submit) ----->| |
|
||
|
|
| |-- (normal app HTTP) ---->| session runner starts
|
||
|
|
| | | llm.stream -> HttpClient
|
||
|
|
| | | simulated network matches openai route
|
||
|
|
| |<== SSE llm/requests ====| exchange ex_1 opened
|
||
|
|
|<== llm.request {ex_1} =====| |
|
||
|
|
|-- llm.chunk {ex_1,[...]}-->|-- POST .../chunk ------->| SSE frames flow into the real
|
||
|
|
|-- llm.chunk {ex_1,[...]}-->|-- POST .../chunk ------->| decode -> step -> LLMEvents ->
|
||
|
|
|-- llm.finish {ex_1} ------>|-- POST .../finish ------>| runner publishes, TUI renders
|
||
|
|
| | |
|
||
|
|
| (if toolCall was sent: runner executes the real tool against the
|
||
|
|
| fake filesystem, then issues the next provider turn -> new exchange
|
||
|
|
| ex_2 -> driver decides the next response)
|
||
|
|
```
|
||
|
|
|
||
|
|
The driver observes the TUI through `ui.state` while chunks stream, so mid-stream UI assertions need no clock control at all: the driver simply has not sent the rest yet.
|
||
|
|
|
||
|
|
## Implementation order
|
||
|
|
|
||
|
|
1. `network.ts`: simulated `HttpClient` + route table + deny-unknown + trace. Replace `httpClient` in `simulationReplacements`.
|
||
|
|
2. `llm-exchange.ts` + `openai.ts`: exchange service and the OpenAI SSE route (schema-constructed chunks, `[DONE]`, interruption).
|
||
|
|
3. Backend control routes (simulation-gated) exposing requests/chunk/finish/fail.
|
||
|
|
4. TUI sim server: `llm.attach`, `llm.request` forwarding, `llm.chunk|finish|fail` proxying.
|
||
|
|
5. Config seeding for the sim provider; end-to-end TUI run driven by `simulation-drive.ts` extended with an LLM auto-responder.
|
||
|
|
6. Trace records for network and LLM exchange activity.
|
||
|
|
|
||
|
|
## Consequences
|
||
|
|
|
||
|
|
- No enqueue/script store to keep consistent; the driver is the single source of model behavior.
|
||
|
|
- Deterministic tests write drivers (respond to `llm.request` programmatically) instead of pre-baked scripts; replay (Phase 4) records exchanges and replays them as an automatic driver.
|
||
|
|
- Provider-coupling is confined to `openai.ts` (one wire encoder against a schema that lives in the repo); a second simulated provider (e.g. Anthropic) is another route file if ever needed.
|