#lele: Bare-Metal Rust AI Inference Engine

lele is a standalone, dependency-free inference engine for ONNX models, built from scratch in pure Rust.

It rejects the "general-purpose runtime" approach (wrapping C++ libs like ORT or using heavy Torch ports) in favor of AOT compilation — converting ONNX graphs into specialized Rust source code with hand-crafted, domain-specific kernels.

#What Can It Do?

lele runs real-world models across speech, vision, and text-to-speech — all in pure Rust, on CPU, with zero runtime dependencies:

All of the above also run in the browser via WebAssembly.

#Performance (2026-06-13)

Single-threaded comparison against ONNX Runtime (CPU) on macOS (Apple Silicon). ORT configured with intra_op_num_threads=1, inter_op_num_threads=1. Speech models use steady-state RTF (warmup + multi-run average).

PP-OCRv6 uses NEON depthwise convolution (4-accumulator FMA, vld2q stride-2), GELU fusion, Conv+Add(bias) fusion, and NEON reduce_mean for SE blocks. Hojo-TTS-Light is a 3-stage pipeline (encoder → AR-LLM → decoder); lele uses a custom KV-cache for the AR loop. MOSS-TTS-Nano uses SDOT int8 GEMV (137 GFLOPS on Apple M2 Pro NEON).

#Key Features

#AOT Compilation Pipeline

lele is not an interpreter. It compiles ONNX graphs into Rust source code at build time:

• ONNX → Rust codegen: Each operator becomes a direct function call with static buffer allocation. No graph traversal, no dispatch overhead at runtime.
• Operator fusion: 12 built-in graph patterns are detected and fused into single optimized kernels:
  • LayerNorm: 9-node subgraph (ReduceMean→Sub→Pow→ReduceMean→Add→Sqrt→Div→Mul→Add) → one layer_norm call
  • GELU: 5-node subgraph (Div→Erf→Add→Mul→Mul) → one gelu call
  • Conv + Add(bias): Conv→Add → single-pass fused bias convolution
  • Conv + SiLU: Conv→Sigmoid→Mul → single-pass conv2d_silu
  • Conv + ReLU, SiLU (single / double / triple), Linear (MatMul→Add), Embedding Concat, Quantized Linear (6-node DynamicQuantizeLinear chain)
• Constant folding: Shape, Unsqueeze, Squeeze, Concat, Slice, Cast, and ConstantOfShape with all-constant inputs are evaluated at compile time and removed from the graph.
• Buffer allocation: Liveness analysis assigns each tensor to a reusable workspace buffer with zero-copy aliasing for Reshape/Squeeze/Unsqueeze/Identity/Cast.
• Weight deduplication: Identical initializers share a single offset in the binary blob.

#Extensible Compiler API

The Compiler builder lets you customize code generation for any model:

use lele::compiler::Compiler;

let compiler = Compiler::new()
    .with_name("MyModel")           // Set generated struct name
    .with_default_optimizations()   // Load fusion patterns + helper methods
    .with_constant_folding(true)    // Enable compile-time folding
    .with_override("MyCustomOp", |node, inputs, outputs, buf, w, indent| {
        // Generate custom Rust code for any operator
        Ok(())
    })
    .with_pattern("MyFusion", my_matcher, my_generator);  // Custom fusion

#Runtime / Compiler Split

The compiler feature is optional (default = ["compiler"]). For deployment, you can depend on lele with default-features = false and ship only the generated Rust code + weights binary — no protobuf parser, no ONNX dependency, no overhead.

#SIMD-Optimized Kernels

• AArch64 (NEON): Hand-written intrinsics for matmul, convolution, activations, normalization. Includes specialized depthwise convolution (4-accumulator FMA latency hiding, vld2q stride-2 decimation), per-channel broadcast add/mul for SE blocks, NEON reduce_mean, and SDOT/I8MM int8 GEMV for quantized inference.
• x86_64 (AVX2/FMA): SIMD paths for GEMM, convolution, LSTM/GRU gates, normalization.
• WASM (SIMD128): Tiled matmul micro-kernel with f32x4 4× blocking, SIMD activations.
• Apple Accelerate: Optional cblas_sgemm FFI for AMX-accelerated GEMM.

#Advanced Kernel Capabilities

• INT8 quantization: DynamicQuantizeLinear, MatMulInteger, fused quantized linear with pre-packed weights
• FP16 weights: f16 weight inference with dedicated GEMV kernel
• Recurrent: LSTM and GRU with SIMD gate kernels
• Autoregressive KV-cache: Custom cache for AR-LLM generation (used in Hojo-TTS)
• STFT: Built-in Short-Time Fourier Transform for audio processing

#Audio Feature Extraction

Built-in DSP frontend for speech models:

• FFT: Radix-2 real FFT with precomputed twiddle factors
• Mel spectrogram: HTK mel scale with sparse filterbank (only nonzero weights stored)
• LFR: Low Frame Rate stacking (reduces frame rate by N×)
• CMVN: Cepstral Mean-Variance Normalization
• Integrated pipeline: SenseVoiceFrontend — scale → pre-emphasis → Hann window → FFT → mel → log → LFR

#WebAssembly Support

lele compiles to WASM and runs ML inference directly in the browser with no server required.

The web demo includes ASR, TTS, object detection, and instance segmentation — all client-side.

cd examples/web-demo
./build_wasm.sh
python3 -m http.server 8080 -d web
# Open http://localhost:8080

See examples/web-demo/README.md for details.

#Supported ONNX Operators

60+ operators across all categories:

• Math: Add, Sub, Mul, Div, Mod, Pow, Sqrt, Neg, Exp, Log, Sin, Cos, Erf, Softplus, Clip, Round, Floor, Ceil, Reciprocal, CumSum, Max, Range, Einsum (partial)
• Neural Network: Conv, ConvTranspose, ConvInteger, Gemm, MatMul, MatMulInteger, LSTM, GRU, BatchNormalization, LayerNormalization, MaxPool, AveragePool, GlobalAveragePool, Resize
• Activation: Relu, LeakyRelu, Sigmoid, Tanh, Softmax, PRelu, HardSigmoid, ArgMax
• Tensor: Reshape, Transpose, Concat, Split, Slice, Gather, GatherElements, Pad, Expand, Tile, Where, TopK, Flatten, Squeeze, Unsqueeze
• Reduction: ReduceSum, ReduceMean, ReduceMax, ReduceL2
• Comparison: Equal, Less, Greater, LessOrEqual, GreaterOrEqual, And, Or, Xor, Not
• Control Flow: If (then/else subgraph recursion)
• Signal: STFT
• Other: Shape, Size, Cast, ConstantOfShape, DynamicQuantizeLinear, Identity, Constant

Silu is available as a fused pattern (Sigmoid→Mul). GELU is fused via compiler pattern (Div→Erf→Add→Mul→Mul).

#Getting Started

#Prerequisites

• Rust (latest stable)
• cargo

#Option 1: CLI Codegen

Convert an ONNX model into Rust source code:

cargo run --release --bin lele_gen -- <model.onnx> <output.rs>

This produces <output.rs> (the model code) and <output>_weights.bin (the weights).

#Option 2: Build-Script Integration (lele-build)

For a seamless build workflow, use the lele-build crate in your build.rs. It supports model download from Hugging Face Hub, URL, or local path, with caching and automatic regeneration:

# model.toml
[model]
source = "hf-hub"
repo = "your-org/your-model"
files = [{ file = "model.onnx" }]

[codegen]
class_name = "MyModel"

// build.rs
use lele_build::{config::ModelConfig, *};

fn main() {
    let cfg = ModelConfig::load("model.toml").unwrap();
    if need_regenerate_with_model("MyModel", "src/gen", "model.toml", None) {
        // Download + generate
    }
}

Set HF_ENDPOINT to use a mirror. Set LELE_SKIP_MODEL_GEN=1 to skip generation. If download fails, a compiling stub is generated so cargo build always succeeds.

#Running Examples

./run_sensevoice.sh          # SenseVoice ASR
./run_silero.sh              # Silero VAD
./run_supertonic.sh          # Supertonic 2 TTS (5 languages)
./run_supertonic3.sh         # Supertonic 3 TTS (31 languages)
./run_yolo26.sh              # Yolo26 object detection
./run_yolo26n_seg.sh         # Yolo26n-Seg instance segmentation
./run_hojo_tts.sh "text"     # Hojo-TTS-Light voice cloning
cd examples/moss-tts-nano && cargo run --release -- "text"  # MOSS-TTS-Nano
cd examples/ppocr && cargo run --release -- image.png      # PP-OCRv6 OCR

#Supported Models

#Architecture

ONNX Model
    │
    ▼
┌──────────────┐     ┌───────────────────────┐
│  lele-build  │────▶│      Compiler         │
│ (download,   │     │  ┌─────────────────┐  │
│  cache)      │     │  │ Constant Fold   │  │
└──────────────┘     │  │ Pattern Fusion  │  │
                     │  │ Buffer Alloc    │  │
                     │  │ Code Gen        │  │
                     │  └─────────────────┘  │
                     └──────────┬────────────┘
                                │
                    ┌───────────▼───────────┐
                    │  model.rs + .bin      │
                    │  (pure Rust, no deps) │
                    └───────────┬───────────┘
                                │
              ┌─────────────────┼─────────────────┐
              ▼                 ▼                  ▼
        Native (NEON/AVX)   WASM (SIMD128)   Embedded

#Roadmap

1. Broader operator coverage — support as many ONNX operators as possible so any model "just works"
2. CPU-first performance — push SIMD (NEON/AVX/WASM) and multi-threading to be the fastest CPU inference engine
3. Lightweight & edge — stay zero-dependency, minimal binary size, ideal for embedded and resource-constrained devices
4. More models — Whisper, CosyVoice, and other popular edge-friendly models
5. Advanced techniques — FlashAttention, PagedAttention for long-sequence efficiency (still CPU)
6. Voice API server — RESTful ASR/TTS/Denoise endpoints for edge deployment

#License

MIT