ENGINEERING NOTE · 9 OCT 2026 · PROPOSED DESIGN
From conversation to a validated device event.
Specification · implementation pendingThis note defines the next Claude integration milestone. It documents intended interfaces and acceptance criteria, not a completed backend or measured hardware performance.
INTERACTION SPECIFICATION
Explore the proposed response contract.
This browser example shows the intended mapping between conversation and device events. It does not record audio, contact Claude or control hardware.
Explain why the sky is blue.
ILLUSTRATIVE DEVICE REPLYAir scatters blue light more strongly than red light, so more blue light reaches your eyes from the sky.
DEVICE STATEListening → Waiting → Speaking → Idle
This is an application-level event envelope, not an Anthropic API response. The real backend must parse the API response, validate it and produce these events.
Read the integration design and validation rules1. Scope of the first demonstration
One ESP32-S3 device captures a spoken question, shows a waiting state, plays a generated reply and renders an approved expression. The demonstration uses an explicit listening trigger; an always-on recording mode is outside this milestone.
Persistent memory, commercial onboarding, autonomous movement and arbitrary device control are excluded from the first demonstration. The long-term companion vision will be evaluated after the conversation loop is reliable.
2. Responsibilities and trust boundaries
| Layer | Responsibility | Boundary |
|---|---|---|
| Firmware | Audio capture, display states, playback and reconnect behavior | Accept only supported backend events. Keep charging and low-level pin control outside model authority. |
| Backend | Speech services, context construction, Claude requests and response validation | Keep API keys server-side. Cap context and output length; validate every proposed action. |
| Claude API | Dialogue and supported tool requests | Model output is not an executable hardware instruction. |
| Speech services | Transcription and synthesized speech | These are separate from Claude. Providers and audio formats remain to be selected. |
3. Proposed request flow
- The user explicitly starts a listening turn. Firmware changes the display to
listening. - Firmware sends audio and a request identifier to the project backend over an authenticated HTTPS connection.
- The backend transcribes the audio and builds a bounded context from the current session and project character instructions.
- The backend calls Claude with the transcribed request. Where tool use is enabled, supported tool schemas are supplied separately from user content.
- The backend validates text and expression arguments, rejects unknown commands and generates speech audio.
- Firmware receives a correlated reply event, plays the audio and maps the expression enum to local artwork.
- Firmware returns to
idle. A timeout or lost connection invokes a local error state; it does not wait indefinitely for model output.
The transport, audio encoding, endpoint paths and credential provisioning are implementation decisions still to be validated against the actual firmware. HTTPS here is a design choice; this note does not claim those endpoints exist.
4. An application event contract
The following envelope is produced by our proposed backend after validation. It is not the raw response shape of the Claude API.
{
"type": "companion.reply",
"reply_text": "Here is a happy face.",
"expression": "happy",
"action": "none",
"memory_proposal": null
}Download the proposed reply-event JSON schema
| Field | Proposed validation rule |
|---|---|
type | Must be companion.reply. |
reply_text | Non-empty string, capped at 600 characters for the initial interaction design. This is a design limit to evaluate, not a model limit. |
expression | Only neutral, happy or curious in the first version. Firmware maps the enum to local assets. |
action | Only none in the first version. Add an action only after firmware support and validation exist. |
memory_proposal | Must be null until the consent and persistence design is implemented. |
Unknown fields and invalid values must be rejected. A backend-generated fallback reply may use the neutral expression. Errors remain a separate event family; the browser illustration's device.error example is not covered by the reply schema.
5. Device states and recovery
| State | Enter when | Exit when |
|---|---|---|
| Idle | Boot completes or a turn ends | User starts listening |
| Listening | An explicit trigger starts capture | Capture completes or the user cancels |
| Waiting | A request is accepted for processing | A valid reply arrives, the timeout expires or a connection error occurs |
| Speaking | Validated reply audio begins | Playback ends or the user interrupts |
| Error | Network, transcription, validation or playback fails | Firmware presents a short recoverable message and returns to idle |
Late responses must be correlated to a request ID so a cancelled turn cannot unexpectedly start playback. Retries should not replay an old response or repeat a future physical action. The firmware implementation must define finite timeouts and interruption behavior.
6. How Claude contributes
Claude supplies language understanding and contextual replies, and may request a supported expression tool. The backend executes the tool request after validating its arguments. Audio capture, transcription, voice synthesis and expression rendering remain separate work.
Reference: Anthropic's tool-use documentation
Character instructions should be consistent and short enough to evaluate. Conversation context should have an explicit budget rather than accumulating indefinitely. The integration will log provider, model version, input/output token counts and a request identifier for reproducible evaluation, while excluding credentials and raw private conversations from published logs.
7. Evaluation and acceptance criteria
Run a fixed 20-prompt set, split between Mandarin and English, and repeat it under documented network conditions. This is a proposed protocol; no results have been collected for the planned Claude integration.
| Measure | Record |
|---|---|
| Response latency | Time from capture completion to first playback; median and p95 across repeated runs. Report speech, model and synthesis times separately when available. |
| Completion rate | Completed voice-to-reply turns divided by attempted turns, with reasons for failure. |
| Event correctness | Every delivered expression uses an allowed enum. Unknown commands are rejected. |
| Recovery | Lost Wi-Fi, backend timeout, malformed event and playback interruption return the device to an understandable state. |
| Cost per turn | Model tokens, transcription and synthesis usage. Distinguish estimated cost from actual billed cost. |
The milestone is accepted only after a recorded hardware conversation and its evaluation record are available. Latency and cost targets will be chosen from measurements rather than presented as invented benchmarks.
8. Evidence and remaining work
- Published build record and two original study photographs. The source report is a 13-slide independent-study presentation supplied by the builder.
- A component list checked against the current device
- Firmware source or a reproducible build/version record
- Claude-connected hardware demonstration and evaluation results
Earlier learning included 78/xiaozhi-esp32. The project does not claim authorship of that firmware or represent earlier third-party model integrations as a completed Claude integration.
EdgeTalk Labs is an independent, unincorporated project operated by 李冠志 in Taiwan. FateVault.org is the project’s web and email domain. Contact hello@fatevault.org.