From 42560c6cd002b9e9b0d99441b35ef6980eaeb2b4 Mon Sep 17 00:00:00 2001 From: Paulus Schoutsen Date: Wed, 3 Jun 2026 09:10:36 -0400 Subject: [PATCH] sandbox_v2: transport cleanup + docs (transport T5) Bring the wire-protocol docs to the current protobuf + pluggable-transport reality and tick the transport trackers. * OVERVIEW.md / architecture.html: rewrite the transport row, the spawn prose, and the channel section to describe the three-layer Channel/Codec/Transport split, ProtobufCodec as the production wire, the Ready-frame handshake (no stdout text marker), length-prefixed framing, and stdio + unix transports (websocket reserved/future). Drop the stale --url ws:// example and JSON-line wording. * channel.py docstrings (both mirrors): ProtobufCodec is the production codec; JsonCodec is the registry-free channel-core test/debug wire. * protocol.py docstring: messages are typed protobuf (REGISTRY + sandbox_v2.proto); the payload shapes listed are the logical contract. * sandbox.py: SandboxRuntime docstrings note the --url-selected transport (stdio default, unix opt-in, ws reserved). * whats-changed.md: tick the protobuf-wire + typed-handlers boxes (T2 360e4543300) and pluggable-transports box (T3 1eaa79d261e). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../components/sandbox_v2/channel.py | 15 +-- .../components/sandbox_v2/protocol.py | 22 +++-- sandbox_v2/OVERVIEW.md | 28 ++++-- sandbox_v2/architecture.html | 91 ++++++++++++++----- sandbox_v2/hass_client/hass_client/channel.py | 8 +- sandbox_v2/hass_client/hass_client/sandbox.py | 23 +++-- sandbox_v2/plans/whats-changed.md | 13 +-- 7 files changed, 134 insertions(+), 66 deletions(-) diff --git a/homeassistant/components/sandbox_v2/channel.py b/homeassistant/components/sandbox_v2/channel.py index 0562c8689b2..9ea5f2af249 100644 --- a/homeassistant/components/sandbox_v2/channel.py +++ b/homeassistant/components/sandbox_v2/channel.py @@ -8,9 +8,10 @@ dispatch core: semaphore, ``register`` / ``call`` / ``push`` / ``close``. It speaks in :class:`Frame` objects and never touches raw bytes. * :class:`Codec` — turns a :class:`Frame` into bytes and back. - :class:`JsonCodec` is the line-compatible default (one JSON object per - frame); :mod:`.codec_protobuf` adds the protobuf wire on top of the same - seam. + :class:`~.codec_protobuf.ProtobufCodec` is the production wire (a typed + protobuf ``Frame`` envelope; the codec owns the ``type → message`` registry + so this dispatch core stays codec-agnostic). :class:`JsonCodec` (one JSON + object per frame) is retained only as the channel-core test/debug wire. * :class:`Transport` — moves whole frame blobs over some byte channel. :class:`StreamTransport` length-prefixes each frame (4-byte big-endian length + body) over an :class:`asyncio.StreamReader` / @@ -172,9 +173,11 @@ class Codec(Protocol): class JsonCodec: """One-JSON-object-per-frame codec. - Line-compatible with the original wire shape (sans the trailing - newline, which the length prefix replaces). Kept as the default for - tests and debugging; production rides :class:`ProtobufCodec`. + The registry-free test/debug wire: it passes frame payloads through as + plain JSON (no ``type``-to-proto lookup), so the concurrency-critical + channel core can be exercised with synthetic message types and arbitrary + dict/int payloads. Production rides :class:`ProtobufCodec`; this stays + for the channel-core tests only. """ def encode(self, frame: Frame) -> bytes: diff --git a/homeassistant/components/sandbox_v2/protocol.py b/homeassistant/components/sandbox_v2/protocol.py index 5c65cc1db5c..5851fa0e1ea 100644 --- a/homeassistant/components/sandbox_v2/protocol.py +++ b/homeassistant/components/sandbox_v2/protocol.py @@ -1,10 +1,20 @@ -"""Phase 5 wire-protocol constants and payload helpers. +"""Wire-protocol message-type constants. -The integration and the sandbox runtime exchange JSON-line messages over -the :class:`Channel` set up in Phase 4. Each message type is namespaced -``sandbox_v2/…``. Both sides share the same names — kept here on the HA -side and mirrored verbatim in :mod:`hass_client.protocol` so neither has -to import the other. +The integration and the sandbox runtime exchange typed protobuf messages +over the :class:`Channel`. Each message type is namespaced ``sandbox_v2/…``; +this module holds the type-string constants. Both sides share the same +names — kept here on the HA side and mirrored verbatim in +:mod:`hass_client.protocol` so neither has to import the other. + +The wire is protobuf (default codec :class:`~.codec_protobuf.ProtobufCodec`): +each ``type`` maps to a request/result proto message pair in +:mod:`.messages` (the `REGISTRY`), generated from +``sandbox_v2/proto/sandbox_v2.proto``. The payload shapes described below +are the *logical* contract for each call — they are carried as those typed +proto messages, not free-form dicts (only genuinely dynamic fields, e.g. +``service_data`` / state attributes / serialized voluptuous schemas, cross +as ``Struct`` / ``ListValue``). The line-oriented :class:`~.channel.JsonCodec` +is retained only as the channel-core test/debug wire. Main → Sandbox calls: diff --git a/sandbox_v2/OVERVIEW.md b/sandbox_v2/OVERVIEW.md index 82527950b7e..72f51eb0d29 100644 --- a/sandbox_v2/OVERVIEW.md +++ b/sandbox_v2/OVERVIEW.md @@ -41,7 +41,7 @@ inside the sandbox. | | v1 (`sandbox/`) | v2 (`sandbox_v2/`) | |---|---|---| | Routing | `entry.options["sandbox"]` set by hand | Computed at runtime from manifest + platform inspection ([`classifier.py`](../homeassistant/components/sandbox_v2/classifier.py)) | -| Transport | Live websocket connection back to main | JSON-line `Channel` over the subprocess's stdin/stdout | +| Transport | Live websocket connection back to main | Protobuf `Channel` over a pluggable transport (stdio by default, unix socket opt-in; websocket later) | | Entity bridge | Bespoke `sandbox/update_state` + `sandbox/entity_command_result` (Option A) | Shared `sandbox_v2/call_service` (Option B) — see [`docs/entity-bridge-decision.md`](docs/entity-bridge-decision.md) | | Config flow | Forwarded through host integration | Runs inside the sandbox; main owns the canonical `ConfigEntry` store | | Auth | System-user token, full HA scope | System-user token (scope enforcement deferred until the sandbox→main connection lands) | @@ -76,8 +76,8 @@ The design choices and the failure modes of v1 they fix are recorded in └─────────────────────────┬─────────────────────────────────────────┼───────────────────┘ │ │ │ subprocess.Popen │ Channel - │ python -m hass_client.sandbox_v2 │ (JSON lines over - │ --name … --url … --token … │ stdin/stdout) + │ python -m hass_client.sandbox_v2 │ (protobuf frames over + │ --name … --url … --token … │ stdio / unix socket) ▼ │ ┌──────────────────────────── Sandbox subprocess ──────────────────────────────────────┐ │ sandbox_v2/hass_client/hass_client/sandbox.py │ @@ -147,15 +147,23 @@ is: ``` python -m hass_client.sandbox_v2 \ --name \ - --url ws://localhost:8123/api/websocket \ + --url stdio:// \ --token ``` -The runtime prints `sandbox_v2:ready` on stdout once its -`HomeAssistant` instance is up; the manager parses that marker, then -opens a `Channel` over the subprocess's remaining stdin/stdout traffic. -Everything after the marker is JSON-line framed (one message per line, -length-delimited at the newline). +`--url` selects the control-channel transport: `stdio://` (the default — +frames ride the subprocess's stdin/stdout) or `unix://` (the +manager opens a unix socket and the runtime dials back). `ws://` / `wss://` +are reserved for the deferred websocket transport and rejected for now. +The runtime opens the channel and sends a `Ready` frame +(`sandbox_v2/ready`) as its first message; the manager treats its arrival +as "running" (there is no stdout text marker — stdout carries nothing but +channel frames). Frames are protobuf (a `Frame` envelope carrying one +typed message per `type`; `JsonCodec` is kept only for channel-core tests) +and length-prefixed (4-byte big-endian length + body) on the stream +transports. The three-layer split is `Channel` (dispatch core) → `Codec` +(`Frame` ↔ bytes; `ProtobufCodec` in production) → `Transport` +(`StreamTransport` length-prefixing over stdio / unix). ### Health & crash recovery @@ -452,7 +460,7 @@ only thing that differs is how the channel pair is materialised. | Plugin | Wire | When to use | |---|---|---| | `hass_client.testing.pytest_plugin` | in-memory channel pair, `SandboxRuntime` as an asyncio task | fast feedback, freezer-safe | -| `hass_client.testing.conftest_sandbox` | real stdio JSON-line (`python -m hass_client.sandbox_v2`) | pins the subprocess boundary, freezer tests auto-skip | +| `hass_client.testing.conftest_sandbox` | real stdio protobuf channel (`python -m hass_client.sandbox_v2`) | pins the subprocess boundary, freezer tests auto-skip | The compat lane runner [`run_compat.py`](run_compat.py) drives either plugin against a list of diff --git a/sandbox_v2/architecture.html b/sandbox_v2/architecture.html index dcaefebb511..d2b935c4764 100644 --- a/sandbox_v2/architecture.html +++ b/sandbox_v2/architecture.html @@ -366,7 +366,9 @@ >
  • - The channel — wire protocol over stdio + The channel — protobuf wire over pluggable transports
  • Config-flow forwarding
  • @@ -438,8 +440,8 @@

    Main holds the canonical view of the world. Each sandbox group is a separate Python process spawned lazily on first need. The two sides - communicate over a JSON-line channel pinned to the subprocess's - stdin/stdout. Inside the sandbox, a private + communicate over a protobuf channel that rides a pluggable transport + (stdio by default, unix socket opt-in). Inside the sandbox, a private HomeAssistant instance hosts the integration's real async_setup_entry, ConfigFlow, entities, services, and events. @@ -882,7 +884,7 @@ font-size="10" font-style="italic" > - stdio JSON + stdio protobuf - JSON-line framing, request/response, + protobuf framing, request/response, Any platform in SANDBOX_INCOMPATIBLE_PLATFORMS → main. - Audio / byte-stream platforms the JSON channel can't ferry: + Audio / byte-stream platforms the control channel can't ferry: stt, tts, conversation, assist_satellite, wake_word, camera. @@ -1463,16 +1465,21 @@ 

    python -m hass_client.sandbox_v2 \
     --name <name> \
-    --url ws://localhost:8123/api/websocket \
-    --token <sandbox access token> \
-    [--share-states] [--share-entity-registry] [--share-areas]
    + --url stdio:// \ + --token <sandbox access token>

    - The runtime prints sandbox_v2:ready on stdout once its - HomeAssistant instance is up. The manager parses that - marker, then opens a JSON-line Channel over the remaining - stdin/stdout traffic. Everything after the marker is length-delimited at - the newline — one message per line. + --url selects the control-channel transport: + stdio:// (the default — frames over the subprocess's + stdin/stdout) or unix://<path> (the manager opens a + unix socket and the runtime dials back). ws:// / + wss:// are reserved for the deferred websocket transport + and rejected for now. The runtime opens the channel and sends a + Ready frame (sandbox_v2/ready) as its first + message; the manager treats its arrival as “running” — + there is no stdout text marker, so stdout carries nothing but channel + frames. Frames are protobuf and length-prefixed (4-byte big-endian + length + body) on the stream transports.

    Health and crash recovery

    @@ -1553,19 +1560,52 @@ -

    5. The channel — wire protocol over stdio

    +

    + 5. The channel — protobuf wire over pluggable transports +

    + +

    + The channel is split into three layers so the wire format and the byte + transport can each change without touching the concurrency-critical + dispatch core: +

    + +
      +
    • + Channel — the dispatch core + (pending-id map, inflight semaphore, register / call / push / close). + It speaks Frame objects and never touches raw bytes. +
      • +
    • + Codec — Frame ↔ + bytes. ProtobufCodec is the production wire (a typed + protobuf Frame envelope; the codec owns the + type → message registry). The + line-oriented JsonCodec is kept only as the channel-core + test/debug wire. +
      • +
    • + Transport — moves whole frame + blobs. StreamTransport length-prefixes each frame (4-byte + big-endian length + body) over a reader/writer pair (stdio, unix + socket); a future WebSocketTransport drops in via + Channel.from_transport. +
      • +

    Both sides of the bridge share an identical - protocol.py — the constants are mirrored verbatim in + protocol.py (type-string constants, mirrored verbatim in homeassistant/components/sandbox_v2/protocol.py and - sandbox_v2/hass_client/hass_client/protocol.py. Each - message is a single JSON object on its own line over the subprocess's - stdio: + sandbox_v2/hass_client/hass_client/protocol.py) and an + identical messages.py / generated _pb2 pair. A + call_service request, for example, is a + CallService message (domain, + service, plus Struct target / + service_data for the dynamic fields) wrapped in the + Frame envelope.

    - 
    {"id": 17, "type": "sandbox_v2/call_service", "domain": "light", "service": "turn_on", "target": {"entity_id": ["light.kitchen"]}, "service_data": {}}
    -

    Three message shapes ride the channel:

      @@ -1583,7 +1623,7 @@
    • Graceful close. Either end can flush a final reply - and then close stdin/stdout cleanly. + and then close the channel cleanly.
    @@ -2182,7 +2222,8 @@ hass_client.testing.conftest_sandbox - Real stdio JSON-line (python -m hass_client.sandbox_v2python -m hass_client.sandbox_v2) Pins the subprocess boundary; freezer tests auto-skip @@ -2292,8 +2333,8 @@

    Sandbox lifecycle

    SandboxManager spawns one subprocess per group lazily; - restart-on-crash with a 3/60 s budget; - sandbox_v2:ready handshake. + restart-on-crash with a 3/60 s budget; Ready-frame + handshake.

    diff --git a/sandbox_v2/hass_client/hass_client/channel.py b/sandbox_v2/hass_client/hass_client/channel.py index 98abff0aa3f..946e780bf0d 100644 --- a/sandbox_v2/hass_client/hass_client/channel.py +++ b/sandbox_v2/hass_client/hass_client/channel.py @@ -147,9 +147,11 @@ class Codec(Protocol): class JsonCodec: """One-JSON-object-per-frame codec. - Line-compatible with the original wire shape (sans the trailing - newline, which the length prefix replaces). Kept as the default for - tests and debugging; production rides :class:`ProtobufCodec`. + The registry-free test/debug wire: it passes frame payloads through as + plain JSON (no ``type``-to-proto lookup), so the concurrency-critical + channel core can be exercised with synthetic message types and arbitrary + dict/int payloads. Production rides :class:`ProtobufCodec`; this stays + for the channel-core tests only. """ def encode(self, frame: Frame) -> bytes: diff --git a/sandbox_v2/hass_client/hass_client/sandbox.py b/sandbox_v2/hass_client/hass_client/sandbox.py index 4699ae525a9..8cb0d72fa8c 100644 --- a/sandbox_v2/hass_client/hass_client/sandbox.py +++ b/sandbox_v2/hass_client/hass_client/sandbox.py @@ -12,10 +12,12 @@ Composes the sandbox's per-process services: registrations and ``_*`` events up to main, gated by :class:`ApprovedDomains` (Phase 6). -The handshake: open the stdio channel, send a :data:`MSG_READY` frame -as the first message, warm-load restore state, register handlers, then -idle until SIGTERM (or until main asks for a graceful shutdown over the -channel — see Phase 9's :meth:`SandboxRuntime._handle_shutdown`). +The handshake: open the control channel (transport selected by the +``--url`` scheme — ``stdio://`` by default, ``unix://`` to dial back +to the manager's unix socket), send a :data:`MSG_READY` frame as the first +message, warm-load restore state, register handlers, then idle until +SIGTERM (or until main asks for a graceful shutdown over the channel — see +Phase 9's :meth:`SandboxRuntime._handle_shutdown`). """ import asyncio @@ -54,12 +56,13 @@ ChannelFactory = Callable[[], Awaitable[Channel | None]] class SandboxRuntime: """Runtime: Ready-frame handshake + length-prefixed control channel. - The websocket URL/token still come in on the CLI for forward-compat - with the deferred WS transport (the scoped sandbox token will travel - that path), but today the runtime only uses the stdin/stdout control - channel that the manager opens. The handshake is a :data:`MSG_READY` - frame sent as the channel's first message — there is no stdout text - marker. + The control-channel transport is chosen from the ``--url`` scheme: + ``stdio://`` (default — frames over the process's stdin/stdout) or + ``unix://`` (dial back to the manager's unix socket). ``ws://`` / + ``wss://`` are reserved for the deferred websocket transport and + rejected for now; the token still travels the CLI for forward-compat + with it. The handshake is a :data:`MSG_READY` frame sent as the + channel's first message — there is no stdout text marker. """ def __init__( diff --git a/sandbox_v2/plans/whats-changed.md b/sandbox_v2/plans/whats-changed.md index 67b21c4356f..4a64b14190b 100644 --- a/sandbox_v2/plans/whats-changed.md +++ b/sandbox_v2/plans/whats-changed.md @@ -70,14 +70,15 @@ opt-in is future work (`docs/design-share-states.md`). ## For sandbox / core contributors -- [ ] **Protobuf wire format.** Messages are protobuf (`Frame` envelope + typed +- [x] **Protobuf wire format.** Messages are protobuf (`Frame` envelope + typed per-message bodies; `Struct`/`ListValue` only for voluptuous schemas and `service_data`). `.proto` source + generated `_pb2` checked in; regen script - provided. (`plan-transport.md`) -- [ ] **Pluggable transports.** stdio + unix socket now; the `Transport` seam - accepts the deferred websocket drop-in (lands with share-states). - (`plan-transport.md`) -- [ ] **Handlers consume typed protobuf messages** (no dict adapters). + provided. (`plan-transport.md` T2 `360e4543300`) +- [x] **Pluggable transports.** stdio (default) + unix socket now; the + `Transport` seam accepts the deferred websocket drop-in (lands with + share-states). (`plan-transport.md` T3 `1eaa79d261e`) +- [x] **Handlers consume typed protobuf messages** (no dict adapters). + (`plan-transport.md` T2 `360e4543300`) - [ ] **Test Dockerfile** for running the client runtime against main. (`plan-docker.md`) - [ ] v1 reference code lives only in git history now.