#FrankenRemote

A tailnet-native remote workstation in Rust: open a machine on your Tailscale network and use its existing desktop, with hardware-accelerated HEVC, no separate account or pairing ceremony, and a system that refuses to accumulate invisible latency.

Status: early implementation; not an installable remote desktop yet. The Rust workspace contains tested authority and bounded media delivery, HEVC admission/normalization, supervised Linux media workers, and input records joined to submission-time checks with an opt-in native X11 pointer/button adapter. There is no complete frd/fr application, qualified live transport, or hardware-qualified capture/codec path yet. IMPLEMENTATION_STATUS.md records the implemented slices and exact verification evidence. The design source of truth remains COMPREHENSIVE_PLAN_FOR_THE_DESIGN_OF_FRANKENREMOTE.md (version 1.4). Every latency target, operating envelope, and platform claim below is a proposed engineering objective from that plan, not a measured FrankenRemote result.

The two planned binary names are:

• frd — FrankenRemoteDaemon, the host-side broker and its internal process-role family (interactive-session agent, on-demand media worker);
• fr — the FrankenRemote client and command-line interface.

The engineering thesis, from the plan:

Tailscale owns connectivity and machine identity. Asupersync owns asynchronous lifetimes and transport integration. Platform APIs and existing codec implementations own capture and hardware acceleration. FrankenRemote owns the small amount of policy that makes those pieces behave like one excellent remote workstation.

---

#TL;DR

The problem. Remote desktop tools accumulate the wrong things: their own accounts, pairing databases, relays, and identity systems layered on top of a network that already has all of those; television-shaped video defaults (cinematic rate control, frame-count vanity metrics, multi-second hidden buffers) applied to terminal text and document work; and input paths that happily replay stale clicks into a desktop that has moved on. The result feels slow in ways nobody can explain, and stops in ways nobody can trust.

The solution. FrankenRemote deliberately owns almost nothing except policy. The installed Tailscale client provides connectivity, machine identity, admission evidence, and HTTPS certificates — there is no FrankenRemote password, PIN, pairing store, coordination service, or relay. Asupersync provides the sole async runtime, cancellation-aware ownership, bounded channels, and a deterministic lab. Platform APIs (ScreenCaptureKit/VideoToolbox, Desktop Duplication, PipeWire/portals, MediaCodec, WebCodecs) and a trimmed FFmpeg boundary provide capture and hardware HEVC. What FrankenRemote itself builds is the connective discipline: a freshness contract with bounded queues at every stage, input authority that expires and ends cleanly, codec-aware loss recovery, and diagnostics that answer "why does this feel slow?" directly.

#Develop and verify

The current workspace contains fr-core, fr-wire, fr-media, fr-native, frd, and fr-lab. From a checkout with Rustup installed, the repository's rust-toolchain.toml selects the exact nightly:

./scripts/verify.sh fast                  # format, workspace check, strict clippy, tests
./scripts/verify.sh docs                  # verify markdown links & required core design docs
./scripts/verify.sh count                 # fixed line counter & size discipline budget check
./scripts/verify.sh audit                 # feature-resolved target dependency & unsafe audit
./scripts/verify.sh test-fixtures         # planted over-budget and architectural violation tests
./scripts/verify.sh crate <name> [action] # per-crate verification (fast|check|clippy|test|fmt)

The fast lane runs formatting, workspace compilation, strict Clippy, and tests with all features, including the explicitly test-only media backend. At source commit e7d57d5a1a284ec0b8da6374d3eda8613167c1e1, the retained Linux verification run passed 47 core unit tests, 17 media unit tests, 9 media contract tests, and 3 compile-fail doctests. These are source/policy and fake-backend contract results, not live Asupersync, Tailscale, OS-input, or HEVC hardware qualification.

The input implementation and its verification limits are described in PROTOCOL_INPUT.md and NATIVE_INPUT.md. Pointer/button effects have been exercised against real X11/XTest servers; native keyboard/text, the independent input-agent watchdog, and the authenticated live-session join remain unfinished. This is not an installable controlled desktop.

The full lane additionally requires UBS and fails explicitly when it is unavailable. The release lane remains blocked until native artifacts and qualification exist. CI calls the same repository-owned commands; it does not replace native builder or hardware evidence. See the implementation status for integration boundaries and remaining work.

#Bounded fragment repair

Selective repair covers every missing fragment even when disjoint holes exceed its negotiated range count. It coalesces the smallest already-received gaps instead of silently omitting later holes, minimizing duplicate fragments within the existing range limit. Sender byte/rate limits, retry counts and the original receiver reference deadline still bound recovery. The fr-media repair-coverage tests exercise actual wire requests and sender/receiver round trips, including all nonempty eight-fragment loss patterns with one through four repair ranges; these are deterministic delivery checks, not hardware or network qualification.

Once streaming, a complete independent picture supersedes older queued media, including unsent repair requests, without waiting for decoder availability. This prevents obsolete reference deadlines from killing an already-available recovery candidate. Reliable startup must still decode first; incomplete independent pictures do not replace the working chain, and in-flight decoder ownership, reference deadlines, HEVC validation and visibility checks are not bypassed.

#Native host publication

The Linux HostSession::publish_display path now joins an already-approved session to native display discovery, explicit viewer selection, negotiated media channels, decoder startup, and the same continuous capture owner. Both public host and observer APIs have been exercised together with actual HEVC and X11 pixel readback. This observation-only path preserves renewal during native work; it is not yet a complete daemon listener, control UI, or hardware-qualified application. See HOST_PUBLICATION.md for the interface, ownership rules, and verification boundaries.

#Executive decisions

Condensed from plan §1; each row is a settled decision, not an open question.

Four assumptions the plan makes explicit rather than burying: HEVC is not x265; HEVC-only does not imply every browser (the supported set comes from a real decode-and-present probe); high bitrate does not repair 4:2:0 chroma subsampling; and a daemon cannot erase OS consent boundaries.

#What can genuinely distinguish this project

Hardware video alone is not novel — Sunshine/Moonlight is the honest baseline to beat, not a deliberately weak VNC configuration. FrankenRemote earns its place in a narrower, coherent combination (plan §3):

1. Freshness rather than throughput. The optimization target is the age of the useful state shown to the user, not frames-per-second entering an encoder. Every stage has a bounded queue and a defined rule for obsolete work — a rule that respects codec references (raw captures are replaceable; encoded reference frames are not). The system never disguises a growing multi-second backlog as a healthy stream.
2. Workstation quality rather than television defaults. Terminal text, thin grid lines, and low-motion windows get spatial resolution preserved and redundant frame rate spent down. When motion stops, a sharper version of the current screen is sent ("settle-to-sharp"). Cursor movement doesn't force a full-screen encode; static desktops don't keep an encoder busy for an artificial 60-fps number.
3. Input authority that ends cleanly. Every controlling session owns a finite input lease with host-clock expiry checked immediately before OS submission. Revocation, disconnect, focus loss, and process failure have explicit release behavior. Reconnection never blindly replays old clicks, text, or modifier state.
4. Explainable performance. The product answers "why does this feel slow?" directly: relayed tailnet path, overloaded encoder, GPU-to-CPU copies, decoder backlog, thermal throttling, or queue growth — and a small sanitized metadata trace can replay session and adaptation decisions in Asupersync's deterministic lab without recording the screen.
5. A small, useful FrankenSuite extension. The reusable contributions are deliberately modest: a well-tested real-time transport profile, cancellation-aware foreign-worker supervision, bounded latest-value media channels, and stable observation/control receipts — upstreamed into Asupersync, not accreted into a distributed operating system.

#Architecture

                           Existing Tailscale network
                                     |
                    Native QUIC / WebTransport / WSS
                                     |
                  +------------------v------------------+
                  | frd: identity, admission, sessions  |
                  | small idle footprint; no codec FFI  |
                  +---------------+---------------------+
                                  | authenticated local IPC
                  +---------------v---------------------+
                  | interactive-session agent           |
                  | OS consent, input lease, revoke UI  |
                  | independent input watchdog          |
                  +---------------+---------------------+
                                  | bounded control + media IPC
                  +---------------v---------------------+
                  | on-demand media worker              |
                  | capture -> GPU conversion -> HEVC   |
                  | capture and encoder co-located      |
                  +-------------------------------------+

The diagram shows ownership and supervision, not byte routing: the media worker sends bounded encoded output directly to the broker; the session agent retains consent and input authority. The user installs one product — separate process roles are an internal safety and OS-session requirement, not services to configure by hand.

Key structural commitments (plan §5–§7):

• The idle daemon is genuinely idle. frd owns listeners, configuration, the peer-identity cache, the session registry, and browser assets. It does not capture, encode, or hold GPU surfaces while nobody is connected (proposed objective: <25 MiB resident, <0.1% of one core).
• Authority is small, typed, and generational. Distinct typed identifiers fence host boot, OS session, remote session, input lease, display geometry, media configuration, and recovery chains. Stale work — old datagrams, late callbacks, replayed heartbeats, resurrected leases — is rejected by construction, not by hope.
• Cancellation has a fixed teardown order: revoke input authority → release remotely held keys/buttons → invalidate generations → stop capture admission → cancel cooperative tasks → drain bounded sends → terminate a stuck foreign worker if necessary → publish closure. Acknowledgements name their stage — admitted, submitted to the OS, or observed — and are never labeled "exactly once."
• Shared pipelines belong to the OS share session, not to whichever viewer joined first. Closing one viewer never cancels another viewer's encoder; a slow read-only viewer is degraded or disconnected independently rather than backpressuring the controller.

#Platform implementation map

Target adapters from plan §8.3 — a statement of intended integration paths, not a certification of any driver or OS version:

FFmpeg integration is a deliberately narrow boundary (plan §9): a curated, allowlisted libavcodec/libavutil build per target, one pinned binding family, opaque GPU surfaces, observable copies, and no FFmpeg types crossing into session or wire layers. macOS/iOS use system VideoToolbox and Android uses MediaCodec, so bundled FFmpeg targets stay few. There is no hot-path ffmpeg subprocess. The pure-Rust HEVC candidate (oxideav-h265) is tracked as a possible future fallback/verification tool; no HEVC encoder is written from scratch for this project.

#Product boundary

Version 1 includes: Linux/macOS/Windows hosting; desktop native clients; iOS and Android clients with first-class native SwiftUI and Jetpack Compose UIs over the shared Rust core, developed in this monorepo; a browser client; keyboard and pointer control; a practical touch interface; audio in both directions (host playback to the client, explicit-enable client microphone to the host); a shared clipboard with automatic bidirectional text sync; explicit file send/receive with ATP-backed folder synchronization; display selection; reconnection; useful diagnostics. One host exposes one existing interactive user session; one remote controller owns input at a time, with a small explicit viewer limit (initially one controller plus two read-only viewers, subject to resource admission).

Non-goals (plan §2.3): remote filesystem mounting, USB passthrough, printer redirection, webcam forwarding, remote shell execution, session recording, an enterprise dashboard, mobile hosting, public-internet guest links, a proprietary cloud account, or a replacement for Tailscale. No AI model in the critical path; no synthesized application responses or fabricated state to hide latency (local cursor rendering is fine; fake application state is not).

Honest limits stated up front: sharing even one display can control the wider logged-in session through application actions — setup says so explicitly. Lock, logout, or user switching ends observation and control. A daemon does not bypass Wayland compositor consent, macOS TCC/login windows, or Windows secure desktops.

#Performance objectives

Every number here is a proposed target from plan §21, not a measurement. Measurement scope rules (process-family CPU/RSS, GPU allocations counted separately, cold vs. warm startup, direct vs. relay, optical input-to-photon lanes, Sunshine/Moonlight comparison baselines) are part of the plan and bind any future claim.

#Proposed workspace

From plan §22 — responsibility boundaries, not a requirement to create every crate before the first working slice. Crates enter the workspace only with a real vertical slice. The current members are fr-core, fr-wire, fr-media, fr-native, frd, and fr-lab; the other responsibility boundaries below remain planned.

frankenremote/
  crates/
    fr-core/          typed IDs, limits, session authority, common state
    fr-wire/          bounded codecs and golden protocol fixtures
    fr-transport/     Asupersync-native and browser transport adapters
    fr-tailnet/       local identity, discovery, certificates
    fr-media/         safe media contracts, scheduling, adaptation
    fr-platform/      per-OS capture, input, audio and lifecycle modules
    fr-ffi/           narrowly gated native media/OS boundary modules
    fr-client/        common viewer, input and presentation policy
    frd/              daemon and internal process-role entrypoints
    fr/               CLI and thin desktop launch integration
    fr-web/           WASM entrypoint and browser boundary
    fr-lab/           fixtures, deterministic scenarios, benchmark harness
  mobile/             native SwiftUI (iOS) and Kotlin/Compose (Android) apps over the shared core
  web/                small self-hosted JS/CSS/HTML shell
  native/             reproducible media build recipes and manifests
  xtask/              repository-owned verification/release commands

Budget: 194k handwritten Rust lines (tests and project-induced upstream work included) as the target, 240k planned maximum, hard stop below 250k, plus a separate ≤20k allowance for JS/Swift/Kotlin/build glue — raised in v1.4 because the mobile apps deliberately carry first-class native UIs. One fixed counting command in the repository; the counting method does not get redefined near the end.

#Implementation sequence

Six phases with explicit exit gates (plan §23). Phase 0 exists so that architectural risks die first, in small reproducible vertical experiments — not after most of the product is written.

An optional extension lane (native 4:4:4 precision, the HEVC-only chroma-carrier experiment, small-group FEC, authorized FrankenTerm semantics) opens only after the core is useful, each with a bounded budget and an independent off switch.

#Agent ergonomics

FrankenRemote is designed to be operated by coding agents as well as humans (plan §18). The planned command surface is small and JSON-first — fr hosts --json, fr doctor --json, fr robot session open/observe/input/close, frd status --json — with every robot response distinguishing success from partial submission, cancellation, refusal, or unknown external effect. Observations carry validity boundaries (geometry generation, source-freshness status, capture/presentation timestamps with uncertainty); actions can require preconditions and are refused rather than clicking an old coordinate system. An optional, explicitly granted adapter can expose FrankenTerm pane semantics through ft robot instead of forcing agents to read pixels. These are planned interfaces, not commands that exist today.

#Verification discipline

The plan's evidence rules bind this repository from day one (plan §24):

• Source reviewed, builds passed, simulated properties passed, independent wire interoperability passed, and hardware measurements passed are separate evidence categories; a plan review establishes none of the latter four.
• A capability row is passed, failed, blocked, or not tested — an untested row is never "supported with caveats."
• Deterministic lab tests exercise the production state machines and limits (lease expiry, generation fencing, recovery budgets, receiver credits), not generated traces that never touch production logic.
• Native fault tests, security tests (forged sources, origin attacks, hostile HEVC parameter sets, replayed tickets), and the hardware/network matrix are enumerated in the plan as requirements to execute during implementation, not tests already run.

#Repository map

#About contributions

Please don't take this the wrong way, but I do not accept outside contributions for any of my projects. I simply don't have the mental bandwidth to review anything, and it's my name on the thing, so I'm responsible for any problems it causes; thus, the risk-reward is highly asymmetric from my perspective. I'd also have to worry about other "stakeholders," which seems unwise for tools I mostly make for myself for free. Feel free to submit issues, and even PRs if you want to illustrate a proposed fix, but know I won't merge them directly. Instead, I'll have Claude or Codex review submissions via gh and independently decide whether and how to address them. Bug reports in particular are welcome. Sorry if this offends, but I want to avoid wasted time and hurt feelings. I understand this isn't in sync with the prevailing open-source ethos that seeks community contributions, but it's the only way I can move at this velocity and keep my sanity.

#License

The FrankenRemote source is licensed under the MIT License with an OpenAI/Anthropic Rider, Copyright (c) 2026 Jeffrey Emanuel (see LICENSE). The rider withholds all rights from OpenAI, Anthropic, their affiliates, and anyone acting on their behalf, including any use of the software or derivative works in a machine-learning dataset, training corpus, evaluation harness, or pipeline. In any conflict between the rider and the rest of the license, the rider controls.

Shipped binaries will additionally carry the recorded license/build provenance of their curated native media components (FFmpeg configuration, Opus, platform SDKs); that distribution review is a release gate (plan §9.3), not a solved question.

---

The final synthesis, from the plan:

Tailscale decides which machines can reach and identify one another. Asupersync makes their session lifetimes tractable. Existing hardware APIs move and compress pixels. FrankenRemote makes the resulting interaction fresh, readable, safe to stop, and easy to understand.

That is enough of a project to be valuable, and enough of a constraint to make it deliverable.