Flowlog

status · under active development; interfaces may change.

FlowLog compiles Datalog into efficient and scalable Differential Dataflow rust executables. As such, FlowLog has first-class incremental maintenance — outputs update without recomputation as facts change. On DOOP points-to it runs ~3.6× faster than Soufflé across 20 DaCapo programs (32 threads).

#Quick Start

1 — Install the toolchain. One-time setup — Rust (1.80+) and required OS packages, then a cargo check smoke test.

$ bash env/env.sh     # Linux / macOS
PS> .\env\env.ps1     # Windows (elevated PowerShell)

2 — Build. The compiler lands at target/release/flowlog-compiler.

$ cargo build --release

3 — Run an example. example/graph_analysis/reach.dl — nodes reachable from a seed set:

.decl Source(id: int32)
.input Source(IO="file", filename="Source.csv", delimiter=",")
.decl Arc(x: int32, y: int32)
.input Arc(IO="file", filename="Arc.csv", delimiter=",")

.decl Reach(id: int32)
Reach(y) :- Source(y).
Reach(y) :- Reach(x), Arc(x,y).
.printsize Reach

Make a tiny dataset, then compile and run:

$ mkdir -p reach
$ printf '1\n'        > reach/Source.csv
$ printf '1,2\n2,3\n' > reach/Arc.csv

# Compile to a binary, then run it on 4 worker threads
$ target/release/flowlog-compiler example/graph_analysis/reach.dl -F reach -o reach_bin -D -
$ ./reach_bin -w 4

Flag reference: Compiler CLI. For incremental mode and the profiler, see https://www.flowlog-rs.com/.

#System requirements

FlowLog's performance and stability depend on a number of host and OS-level settings. For example, on Linux a large analysis can abort with memory allocation of <N> bytes failed even when memory is plentiful, because the allocator maps many memory regions and exhausts a low vm.max_map_count. Raising it resolves this:

$ sudo sysctl -w vm.max_map_count=1048576

Before running, review the recommended host configuration in the setup guide: https://www.flowlog-rs.com/tutorial/getting-started/system-config.

#Architecture

A .dl program compiles through five stages; three side modules assist the planner and codegen:

                                                   profiler
                                                       ┊
                                                       ↓
.dl → parser → typechecker → stratifier → planner → codegen → executable
                                             ↑
                                             ┊
                                    catalog · optimizer

Pipeline

• parser — .dl → typed AST, each node source-located.
• typechecker — resolves literal types (1 → int32).
• stratifier — groups rules into dependency-ordered strata; a stratum with a cycle recurses to fixpoint.
• planner — lowers rules to a Differential Dataflow plan, sharing sub-plans to reuse arrangements.
• codegen — emits the plan as Timely + Differential Dataflow Rust.

Side modules

• catalog — per-rule metadata for the planner (signatures, pushdown filters, range checks).
• optimizer — cardinality-based join ordering and worst-case optimal joins (WIP).
• profiler — runtime metrics from Timely / Differential Dataflow operators.

Crates

• flowlog-build — library; compile .dl to Rust from build.rs.
• flowlog-compiler — CLI; compile .dl to a standalone executable.
• flowlog-runtime — linked into output (interning, IO, sort/merge, incremental-txn state); not a direct dep.

#Compiler CLI

$ flowlog-compiler <PROGRAM> [OPTIONS]

<PROGRAM> is a path to a .dl file, or all / --all to compile every program in example/. Common options:

• -F, --fact-dir <DIR> — default directory for relative .input filenames; the executable can override it at runtime.
• -o <PATH> — output executable path; defaults to the program stem (reach.dl → ./reach).
• -D, --output-dir <DIR> — default directory for .output files; - prints tuples to stdout. The executable can override it at runtime.
• -B, --build-dir <DIR> — keep the generated Rust project in this directory for subsequent builds.
• -T, --target-dir <DIR> — share Cargo artifacts across build directories; overrides CARGO_TARGET_DIR. Relative paths start at the compiler's working directory.
• --mode <MODE> — batch (default) or inc.
• --str-intern — intern string columns at load for faster joins and lower memory (off by default).
• -P, --profile — collect execution statistics.
• -h, --help — full help text.

#Testing

A green oracle run is the definition of correct — see tests/README.md for per-suite contracts and recipes.

#vs Soufflé — DOOP

On DOOP default points-to analysis (doop/default.dl) across all 20 DaCapo programs at 32 threads (FlowLog -w 32, Soufflé -j 32). The Soufflé program is the same default.dl of identical rules and join order; all 20 produce identical VarPointsTo.

Run time (run only; one-off compile excluded) — FlowLog is faster on 20/20, geomean 3.62× (range 1.41–6.07×).

Peak memory — Soufflé is leaner: Soufflé/FlowLog geomean 0.43× (FlowLog trades memory for speed).

Benchmark suite: flowlog-bench.

#Publication

FlowLog: Efficient and Extensible Datalog via Incrementality
Hangdong Zhao, Zhenghong Yu, Srinag Rao, Simon Frisk, Zhiwei Fan, Paraschos Koutris
VLDB 2026, Boston

• Paper — PVLDB Vol. 19
• Artifacts — flowlog-rs/vldb26-artifact

#Contributing

Issues and pull requests are welcome. Target main and sign off your commits with git commit -s. PRs must pass CI before merge. See the contributor guide and release process.

Let's make Datalog fast — and incremental.