#macOS Virtualization Kubelet

Run native macOS virtual machines as Kubernetes pods.

macOS-vz-kubelet is a Virtual Kubelet provider that turns an Apple Silicon Mac into a Kubernetes node. Each pod's first container boots as a native macOS VM on Apple's Virtualization framework; optional Docker side-car containers run in the same pod for logging, monitoring, or artifact handling.

It targets teams scaling macOS CI/CD (GitLab runners, build and test farms) on a cluster, at near-native Apple Silicon performance and without QEMU/KVM.

#Contents

• Overview
• How it works
• Requirements
• Getting started
• Networking
• Imaging
• Feature overview
• Configuration
• Examples
• Roadmap
• Contributing
• Related projects
• License

#Overview

Running macOS under Kubernetes traditionally means QEMU/KVM on Linux, which cannot fully use Apple Silicon hardware. macOS-vz-kubelet instead runs VMs natively through Virtualization.framework on the Mac itself, so guests run at near-native speed.

The project is designed to:

• Run macOS VMs as first-class pods in a Kubernetes cluster.
• Support hybrid pods that combine a macOS VM with Docker side-car containers.
• Manage VM images with a custom, compressed, OCI-compliant format.
• Integrate with Kubernetes scheduling, resource requests, and lifecycle.

For exactly what is and is not supported, see the Feature overview.

#How it works

The kubelet runs on the Mac host as a Virtual Kubelet provider and orchestrates two backends from one pod spec.

• Virtualization framework provisions and runs the macOS VM (pod container index 0), directly on Apple Silicon.
• Custom OCI image format distributes VM images. Images are built with Virtualization.framework-compatible tooling and pushed/pulled as compressed OCI artifacts; the kubelet pulls them via the oras-go library. Packaging/push uses a dedicated ORAS CLI fork, oras-macos-vz.
• Hybrid pods. Container 0 is always the macOS VM. Containers 1..N run on the local Docker daemon as regular containers.
• Networking is wired automatically: local NAT by default, or bridged for routable IPs. See Networking.
• Resource and lifecycle. CPU/memory requests size the VM. Pod create and delete are supported; updates require recreating the pod.

Typical flow:

1. A pod manifest references a macOS VM image in the custom OCI format.
2. The kubelet pulls the image from the registry (cached locally, see Imaging).
3. The VM boots; any side-car containers start alongside it.
4. The kubelet reports pod status, IPs, and lifecycle back to the control plane.

#Requirements

• Apple Silicon Mac (M-series). The binary builds for darwin/arm64 only.
• A macOS host with Virtualization.framework support.
• A Kubernetes cluster to join the node to, plus a client certificate and key the kubelet uses to authenticate to the API server.
• A macOS VM base image in the custom OCI format, pushed to an OCI registry (see Imaging).
• SSH enabled inside the VM image. kubectl exec/attach into the VM, the readiness probe, postStart hooks, and stats all run over SSH.
• A Docker daemon on the host (for example Colima) if you run side-car containers.
• Code signing with Virtualization.framework entitlements. Ad-hoc signing is enough for NAT; bridged networking needs Apple-approved vmnet entitlements (see Networking).

#Getting started

#Build and sign

Tooling versions are pinned in .mise.toml (Go, goreleaser, Ruby). Builds run on macOS and produce a MacOSVK.app bundle.

make snapshot   # local development build
make release    # signed release build (set RELEASE_CERTIFICATE_NAME, RELEASE_PROVISION_PROFILE_PATH)

The binary needs Virtualization.framework entitlements. To ad-hoc sign a locally built binary:

codesign --entitlements resources/vz.entitlements -s - <YOUR BINARY PATH>

Bridged networking requires the additional vmnet entitlements in resources/release.entitlements, which need Apple approval. See Networking.

#Run a workload

1. Create a base VM image. Use a Virtualization.framework-compatible tool such as macosvm.

2. Package and push it in the custom OCI format with oras-macos-vz:

   oras-macos-vz push -h
   

3. Write a pod manifest that references the OCI image. See Examples.

4. Run the pod. The kubelet picks it up, pulls the image, and boots the VM. Status is reported back as for a regular container.

5. Interact with kubectl exec (and attach) into the VM or any side-car container.

Configure the kubelet with the flags and environment variables below.

#Networking

Networking runs in one of two modes.

#NAT (default)

• The VM gets a local IP via NAT.
• Fits most cases where external reachability is unnecessary.
• kubectl exec/attach reach the VM over SSH on its local IP. The VM IP is discovered from the host ARP table.

#Bridged

• Gives the VM a routable IP by attaching it to a host network interface (typically a tagged VLAN with DHCP).
• Each VM interface gets a generated MAC so its lease can be tracked.
• The kubelet discovers the VM IP by sniffing for that MAC with libpcap (tcpdump-style capture) and reports it to Kubernetes.

Enable bridged mode by setting VZ_BRIDGE_INTERFACE to the host interface name. Bridged mode requires the VMNet and VM Networking capabilities, which need Apple approval; see this Apple Developer Forums thread for how to request them. After approval, generate a Mac Development certificate, an App ID with those capabilities, and a provisioning profile, then build a release binary signed with resources/release.entitlements.

#Imaging

VM images use a custom, OCI-compliant format. See the OCI manifest example.

#Compression

macOS images are large (tens of gigabytes with Xcode and simulators preinstalled). The packaging step compresses image layers (parallel gzip) so they store and transfer at a fraction of the raw size, and the kubelet decompresses on pull.

#Copy-on-write overlays

A host runs at most two VMs at once (MaxVirtualMachines, enforced by a semaphore; a third pod waits for a free slot). This reflects a Virtualization.framework limit. To let concurrent VMs share one base image while keeping independent state, each pod gets a copy-on-write clone of the disk and auxiliary storage. Overlays are removed when the VM stops; the base image is never mutated.

#Digest validation

Local image files carry a recorded digest to guarantee integrity:

• On first download, a digest is computed and stored alongside the image file.
• On every pod start, the digest is checked against the expected digest from the registry manifest.
• If the digest file is missing or is not newer than the image file, the digest is recomputed from the image; a mismatch invalidates the local cache and the image is re-pulled.

#Feature overview

The tables below list supported Kubernetes features. Anything not listed is unsupported.

#Node

#Pod

#Containers

Readiness gating. Pod-spec probes are not evaluated, but every macOS pod is held NotReady until an internal SSH readiness check succeeds (the guest sshd must answer), bounded by VZ_SSH_READINESS_TIMEOUT. If a pod defines an exec postStart hook, Ready is gated on that hook finishing successfully (bounded by VZ_POSTSTART_TIMEOUT); a permanently unreachable VM fails the pod rather than hanging.

#Storage

A projected volume maps several volume sources into one directory. Kubernetes adds one by default carrying the service account token, API server CA, and namespace, so a pod can call the API server.

#Image pull secrets

The kubelet resolves registry credentials from the pod's spec.imagePullSecrets and from the pod ServiceAccount's imagePullSecrets in the same namespace.

Supported secret types:

• kubernetes.io/dockerconfigjson (preferred), requires .dockerconfigjson data.
• kubernetes.io/dockercfg (legacy), requires .dockercfg data.

Secrets of other types are ignored with a warning event. A referenced secret that is missing or invalid fails pod creation.

#Configuration

#Command-line flags

All flags are optional. Duration flags take Go duration strings (for example 30s, 1m).

Standard klog logging flags are also available under a --klog. prefix (for example --klog.v).

#Environment variables

#Kubernetes connection

#macOS VM access (SSH)

Used for kubectl exec/attach into the VM, the readiness probe, postStart hooks, stats, and graceful shutdown. Provide one of the key variables or VZ_SSH_PASSWORD.

#Networking and side-cars

#Tracing (OpenTelemetry)

#Local cache

The cache directory is ~/Library/Caches/com.agoda.fleet.virtualization. It holds OCI images and their digest files, plus pod mount volumes when emptyDir volumes are used.

#Examples

The example directory has ready-to-adapt manifests:

• pod.yml - a minimal macOS VM pod.
• pod-gitlab-sidecar.yml - a hybrid pod with a macOS VM plus a GitLab Runner Docker side-car.
• deployment.yml - a Deployment of macOS VM pods.
• oci_manifest.json and config.json - the custom OCI image format.

#Roadmap

• Higher test coverage, including an open end-to-end testing approach.
• A public CI/CD pipeline to make contribution easier.

#Contributing

Issues and contributions are welcome on the GitHub Issues page. See CONTRIBUTING.md for development and pull request guidance.

#Related projects

These projects are the foundation macOS-vz-kubelet is built on:

• Virtual Kubelet - the Kubernetes kubelet implementation this provider plugs into.
• Code-Hex/vz - Go bindings for Apple Virtualization.framework.
• oras-project/oras-go - ORAS Go library for OCI artifacts.
• oras-macos-vz - the ORAS CLI fork used to package and push VM images.

#License

Licensed under the Apache License, Version 2.0. See LICENSE.