Building an in-tree LLVM obfuscator solo, with an AI pair

I work in cybersecurity for a large enterprise these days. This project has nothing to do with my employer — it’s a personal, after-hours thing — but the day job is why I care about obfuscation and anti-tamper as a topic in the first place, and why I wanted to actually understand one from the inside instead of just reading papers about it.

The result is ollvm: an obfuscation framework fused directly into an LLVM 22 checkout (a parallel ollvm-21.x branch tracks LLVM 21), built by one person, over evenings and weekends, with heavy use of an AI coding agent. This post covers three things: what obfuscation actually does at each layer (for readers who don’t live in this space), what it takes to get a project like this correct rather than just impressive-looking, and how a solo hobbyist gets something this large built at all.

Disclaimer: built for legitimate software-protection use cases — IP protection, anti-tamper research, CTFs with permission. Obfuscation is not a security boundary; treat it as one layer in a broader defensive strategy.

Obfuscation, briefly

Skip this section if you already know the vocabulary. If you don’t, here’s every technique this project implements, in plain terms, before any architecture talk:

• Opaque predicate — a condition that always evaluates the same way (always true, always false), but written so that’s not obvious from local analysis. if ((x*x) % 4 != 2) is always true for integer x, but a decompiler can’t tell that without arithmetic reasoning. Used to plant fake branches that look real.
• Bogus control flow (bcf) — inserts opaque predicates and dead branches into the CFG so a chunk of the graph is genuinely unreachable, but looks reachable to anything short of data-flow analysis. Cheap to apply, cheap-ish to see through once you know the trick, still raises the floor for casual analysis.
• Control flow flattening (flattening) — takes a function’s structured control flow (if/else, loops) and turns it into a single dispatcher loop with a state variable, so every basic block becomes a case reached only through the dispatcher. The CFG stops looking like the original program’s logic and starts looking like a state machine — which is accurate, because that’s what it now is.
• Instruction substitution — replaces an instruction with a longer, semantically-equivalent chain (a + b becomes something like a - (-b) expanded further, a ^ b becomes an equivalent built from &/|/~). Individually trivial to simplify back; in bulk, and layered with other passes, it’s noise a human has to wade through.
• Mixed Boolean/Arithmetic (mba) — a more rigorous version of substitution: rewrites integer expressions using identities that mix boolean (&, |, ^, ~) and arithmetic (+, -, *) operators, which don’t simplify under either boolean algebra or arithmetic algebra alone. This is what most of the “wall of bitwise garbage” you see in obfuscated decompiler output actually is — see the screenshot further down.
• Virtual call (vcall) — turns a direct call into an indirect call through a synthetic vtable-like structure, so a disassembler’s cross-reference graph (which relies on direct call targets) loses an edge.
• Anti-decompiler tricks (adec) — techniques aimed specifically at breaking decompiler heuristics rather than a human reading disassembly: indirect-branch trampolines, inline asm junk, pointer-alias confusion, fake loops, dead decoy blocks. This is the category most precisely targeted at Hex-Rays/Ghidra-style data-flow propagation choking on ambiguous memory aliasing.
• String encryption — literal strings are encrypted at rest and decrypted at first use, so strings/static string scanning gives you nothing.
• Code virtualization (“VM obfuscation”) — the deep end. Instead of transforming the function’s native instructions, you compile the function to a custom bytecode and ship a small interpreter that executes it at runtime. A disassembler sees the interpreter’s dispatch loop, not the original logic — the actual program only exists as data. This is the technique commercial protectors (VMProtect, Themida, and similar) are built around, almost always applied post-compile at the binary level rather than as a compiler pass.

Everything past this point in the post is about how these get implemented correctly, at scale, inside a real compiler.

Why this is hard

None of the individual techniques above are novel — bcf, flattening, and substitution have been public since the original Obfuscator-LLVM project over a decade ago. The hard part was never “know the trick.” It’s:

• LLVM’s API surface moves under you. The New Pass Manager, analysis invalidation contracts, and IR utility APIs change release to release. A pass that’s correct against LLVM 14 is not guaranteed to compile against LLVM 22, let alone still be correct — PreservedAnalyses semantics, PassBuilder callback registration points, and even basic IR builder ergonomics shift.
• You can’t diff IR to check correctness. The entire point of the transform is that the IR is different afterward. The only ground truth is: does the compiled binary still do the same thing. That pushes correctness checking out of the compiler and into an external runtime test harness — a very different discipline from most compiler-pass work.
• CFG-rewriting passes break invariants in non-obvious ways. Flattening and virtualization restructure control flow wholesale; PHI nodes, exception-handling edges (invoke, callbr, EH funclets), and SSA dominance all have to survive. Get this wrong and you get a binary that works most of the time — which is worse than one that never works, because it fails in the field instead of in CI.
• The passes interact. Run bcf before substitution and you get different (and sometimes invalid) results than the reverse order. Run vm and flattening on the same function and you get two passes trying to own the entire CFG at once. Managing N passes pairwise is combinatorial; it needs to be an explicit ordering model, not a convention people remember.
• “Looks obfuscated” and “is obfuscated” are different bars. A pass that inflates instruction count without adding real analytical cost is theater. Measuring that honestly (cyclomatic complexity delta, opaque predicate density, MBA expression depth) is its own piece of engineering, covered later in this post.

This is why most hobby-grade forks stop at bcf + flattening + substitution and call it done — that trio is achievable in a long weekend. Everything past it (deterministic builds, a real budget/safety system, EH correctness, a reporting pipeline, and especially a working VM pass) is where the effort actually lives.

Where LLVM-based obfuscators stand today

The lineage matters for context. Obfuscator-LLVM (o-llvm), out of EPFL, introduced bcf, flattening, and substitution as an LLVM fork and was, for years, the reference implementation everyone cited. It stalled in the LLVM 4-ish era and was never meaningfully carried forward by its original authors. Since then the ecosystem has mostly been forks-of-the-fork: Hikari (most actively maintained, popular in the iOS jailbreak/tweak scene, adds a handful of techniques like indirect branching and anti-class-dump), and smaller derivatives like Goron and Akira/Arkari that each bolt on one or two extra passes (string encryption, extra flattening variants). Nearly all of them are pinned to an LLVM version from several years ago, because re-basing hand-written passes onto a new LLVM release is expensive, unglamorous work that doesn’t show up in a feature list — so projects freeze at whatever LLVM version they started on and slowly stop building against current toolchains at all.

None of the public forks I’m aware of do all of: track a current LLVM release branch, integrate cleanly with the New Pass Manager (rather than shimming the legacy pass manager LLVM itself has been trying to retire for years), enforce deterministic/reproducible output, or go past CFG-level transforms into full code virtualization with a shared runtime engine. Virtualization specifically tends to live in the commercial, closed-source, binary-level tooling category (VMProtect, Themida) rather than as an open compiler pass — which made it the most interesting (and hardest) thing to build here.

None of this is a claim that this project is “better” in some absolute sense — bcf and flattening are decades-old ideas and skilled analysts have automated deobfuscation techniques (symbolic execution against opaque predicates, SSA-based CFG un-flattening) for most of it. The point is narrower: getting an obfuscator that’s correct, current, and instrumented is a different, less-traveled problem than getting one that’s clever.

One person, an AI pair, and a 15-million-line codebase

I did not have a team. I had evenings, an LLVM checkout, and a coding agent. Being honest about how that actually worked, because “AI helped me build a compiler pass” undersells the real mechanics:

Scope the agent’s world before writing anything. LLVM is enormous — clang, MLIR, LLDB, every backend, decades of history. An agent (or a human, for that matter) let loose on the whole tree produces slop: unrelated files touched, upstream conventions guessed instead of matched, huge diffs nobody can review. The project’s CLAUDE.md explicitly tiers the repository: full read/write on the obfuscator’s own directories, read-only with escalation-required on the three or four upstream files that need a registration line added (PassRegistry.def, PassBuilder.cpp), hard refusal on everything else. That single constraint — “you don’t get to explore” — did more for output quality than any amount of prompting.

Treat the agent as fast but untrusted. Code that looks plausible and code that’s correct are not the same thing, and an LLVM pass that’s subtly wrong (a missed PreservedAnalyses, an SSA invariant broken under a rare EH shape) doesn’t announce itself — it corrupts a downstream analysis or crashes three passes later. The only thing that made rapid AI-assisted iteration safe was having ground truth to check against on every change: the IR verifier, and a runtime correctness harness that compiles and executes obfuscated output and diffs it against a clean baseline. Generated code that doesn’t pass both isn’t done, no matter how confident it looks.

Write the docs as you go, not after. DEV.md, USER.md, VM.md, and TESTS.md in the repo total roughly 2,500 lines. That’s not for show — it’s the mechanism that lets both me and the agent stay oriented across sessions. An agent with no memory of yesterday’s session needs the same onboarding a new hire would; a written architecture doc with an explicit pass-ordering table and a JSON schema reference does that job whether the reader is human or not.

The human still owns the things a test suite can’t check. What “correct” means for a security tool, which combinations of passes are a bad idea (vm + flattening together, rejected outright by the pipeline driver), what belongs in the threat model versus what’s out of scope — none of that comes from an agent iterating fast. It comes from deciding what “good” looks like and then using the agent to get there quickly instead of slowly.

Net effect: a solo hobbyist produced something with the shape of a small team’s output — in-tree integration, ten interacting passes with a real ordering model, a bytecode VM with four independent hardening layers, a JSON reporting schema, an HTML report viewer, and a Python runtime test harness — because the AI collapsed the typing cost of LLVM’s API surface, while the verification loop below is what kept the correctness cost from collapsing along with it.

Verify, harden, iterate — the loop per pass

Every pass, new or modified, goes through the same cycle before it’s considered done:

1. Implement as an NPM function (or module) pass, honest PreservedAnalyses return required.
2. -obf-verify — run LLVM’s own IR verifier immediately after the pass. Catches structural breakage (bad SSA, dangling references) before it has a chance to propagate.
3. Runtime correctness diff — compile a test program with the pass enabled, at a fixed seed, run it, diff stdout against a clean (-O2, no obfuscation) build of the same program. Repeat across multiple seeds and multiple pass combinations — llvm/utils/obfuscator/cases/ covers exhaustive combinations, EH edge cases, budget-exhaustion scenarios, and per-feature regressions. This is the step that actually matters: an obfuscator that produces IR the verifier likes but a binary that segfaults or returns the wrong value is worse than useless.
4. Report and CFG inspection — with -obf-report-dir set, look at the actual before/after CFG and the difficulty score for a handful of representative functions. Did the pass do nothing on cases it should have handled? Did instruction count balloon past the budget for no analytical benefit? The report makes both visible instead of implicit.
5. Budget and perf sanity — check budget_util_after in the report; if one pass is chewing 80% of the growth budget on a trivial function, something’s mis-tuned.
6. Iterate — fix, re-run steps 2-5. Because seeding is deterministic (-obf-seed + -obf-seed-manifest), a failing case reproduces byte-for-byte on the next run — critical when the failure showed up on seed -8433793424924233352 out of a fuzzed sweep and needs to be handed back for a fix.

Because seeds cascade base -> module -> function -> pass, and every function gets an independently derived seed, the same test corpus effectively fuzzes pass ordering and RNG interaction for free — a regression introduced in mba on one seed shows up as a runtime diff failure on some function somewhere in the corpus, not a silent miscompile that ships.

Architecture

llvm.global.annotations
  -> ObfuscationAnnotationAnalysis   (Function -> Config cache)
  -> ObfuscationModulePass
       -> StringEncryptionPass       (strenc, module-only)
       -> ObfuscationFunctionDriverPass
            -> per-function: topologically-sorted pass pipeline
            -> ObfReportAnalysis     (JSON + CFG DOT sink)

Configuration is entirely annotation-driven — nothing is passed on the opt command line beyond global safety knobs:

// Light: expression-level only
__attribute__((annotate("obf: mba(prob=70,maxDepth=3), substitution(loop=2)")))
int light(int x) { return x * 3 + 7; }

// Heavy: structural + post-hardening
__attribute__((annotate("obf: mba(prob=70), bcf(prob=30,loop=1), flattening(minBlocks=3), shield, adec(prob=60)")))
int heavy(int x, int y) { return x ^ y; }

// Maximum: replace the entire function body with a bytecode interpreter
__attribute__((annotate("obf: vm(hardened=1,useAES=1,regEncrypt=1)")))
int secret(int key, int data) { return key ^ (data + 0xDEAD); }

clang -S -emit-llvm -O0 -g test.c -o test.ll
opt -passes=obfuscation -S test.ll -o test.obf.ll -obf-seed=1 -obf-deterministic -obf-verify
clang test.obf.ll -O2 -o test.obf

ObfuscationAnnotationAnalysis parses every obf: string exactly once per module — a parenthesis-aware tokenizer, passName(params...) grammar, canonical key aliasing — into a Function* → ObfuscationConfig cache that everything downstream reads from.

Ten function-level passes, one module-level pass:

Pass	ID	Category
Mixed Boolean/Arithmetic	mba	Expression
Instruction substitution	substitution	Expression
Virtual call	vcall	Call hardening
Basic block split	split	CFG
Semantic diffusion	sdiff	Expression / CFG
Bogus control flow	bcf	CFG
CFG flattening	flattening	CFG
Anti-optimization shield	shield	Post-hardening
Anti-decompiler	adec	Post-hardening
Code virtualization	vm	Virtualization
String encryption	strenc	Module-only

You don’t control the order — the driver does, via a partial-order graph (mba before substitution, bcf after everything expression-level, vm conflicts with flattening and the driver rejects that combination outright) resolved with Kahn’s algorithm and a deterministic tie-break. Annotate in any order; the same functions always get the same pipeline.

Seeding cascades base -> module -> function -> pass, and a per-function instruction budget (limit = clamp(insts_before × multiplier, 1, hardcap), default 50×) stops any single pass from running away — later passes skip themselves and record why once the budget’s exhausted, rather than silently no-op-ing.

The VM pass

This is the deep end, and it gets its own ~800-line reference doc in the repo. Virtualizing every function independently would mean duplicating 51 opcode handlers per function — so instead there’s a single module-level shared engine, __vm_engine, built once per module. Every virtualized function becomes a thin wrapper: PHI nodes get demoted to allocas (the bytecode has no PHI concept), the body is compiled to a private bytecode stream, and the wrapper tail-calls the shared engine with its own globals.

The interesting part: the opcode encoding is different for every virtualized function. Each gets a Fisher-Yates-shuffled logical↔physical opcode bijection seeded from its own RNG stream, so the physical byte for “add i32” in function A is not the physical byte for “add i32” in function B. A signature built from one virtualized function’s dispatch table tells you nothing about the next one in the same binary.

Four hardening layers stack independently:

Layer	Default	Mechanism
Register-index XOR	on	Every register-index byte in the bytecode XOR’d with a compile-time salt; handlers re-XOR with a volatile-loaded salt
Bytecode encryption (AES-CTR)	on	Per-function 128-bit key, .init_array constructor, shared __obf_aes_ctr_decrypt() runtime (also used by strenc)
Bytecode encryption (LCG)	opt-in	Cheaper alternative: key = ptrtoint(@bytecode) XOR SEED, so it’s ASLR-derived
Register-value XOR	opt-in	Every register file access XOR’s against a per-slot key table — defeats a straight memory dump of the register files mid-execution

hardened=1 adds MBA on the handler control flow, opaque predicates in the dispatch loop, RDTSC anti-debug timing gates on the dispatch and a random subset of handlers, an FNV-1a bytecode integrity check, and callee-pointer XOR masking — all as .init_array constructors so they run before main.

Anti-decompiler: per-architecture gadgets

adec is the one pass that goes below IR abstraction and cares about the target. It’s a technique pool (llvm/lib/Transforms/Obfuscator/ADec/) with an ArchBackend abstraction and concrete backends for x86-64 and AArch64, sampling from nine techniques per site: alias confusion, inline asm gadgets, call trampolines, constant laundering, fake loops, indirectbr trampolines, RDTSC timing stretches, stack pollution, dead decoy blocks. This is the pass most directly aimed at the human on the other end of a decompiler — asm-level junk and pointer aliasing are exactly what Hex-Rays-style data-flow propagation chokes on.

Correctness hardening

A few things treated as non-negotiable, because CFG-rewriting obfuscation is notorious for producing IR that happens to work until it doesn’t:

• Every pass returns an honest PreservedAnalyses — the driver’s “did this pass actually change anything” signal (used for report deltas and analysis invalidation) comes straight from !PA.areAllPreserved(). A pass lying about preservation corrupts the report and lets stale analyses leak into the next pass.
• ObfRepairSSA runs after heavy CFG transforms to re-thread PHI nodes and clean up dead blocks — vm is explicitly flagged NeedsSSARepair = true by the driver.
• invoke, callbr, and EH funclets are hostile territory by default. vm rejects any function with EH pads, invoke, callbr, or indirectbr outright rather than try to virtualize around them.

Reports: knowing what actually happened

-obf-report-dir=<dir> turns on a JSON obfuscation map plus per-function CFG snapshots as DOT files, including a per-pass diff DOT that color-codes added/removed blocks and edges against the previous snapshot — you can point at exactly which pass introduced which piece of the CFG. Each function report carries resolved seeds, instruction counts before/after, budget utilization, a per-pass ran/skipped/changed/delta table, and a computed “difficulty” score (cyclomatic delta, opaque-predicate density, MBA node count/depth, indirect branch and call counts) — the mechanism that turns “looks obfuscated” into a number instead of a vibe. llvm/utils/obfuscator/obf_report_html.py turns the JSON into a self-contained HTML file with an interactive Cytoscape.js CFG viewer.

Report viewer: per-pass instruction delta table on top, interactive before/after CFG panes below.

What it looks like from the other side

Straight-line CFG before, versus after flattening collapses it into a dispatcher loop:

Same function through Hex-Rays, baseline versus after bcf (bogus control flow):

mba alone turns a handful of arithmetic ops into a wall of bitwise identities that decompiles cleanly but tells you nothing at a glance — this is what “MBA” in the glossary above looks like in practice:

Try it

cmake -S llvm -B build -G Ninja -DLLVM_ENABLE_PROJECTS="clang;lld" \
  -DLLVM_ENABLE_RTTI=ON -DLLVM_ENABLE_EH=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target opt clang

opt -passes=obf-dump-config -S test.ll -o /dev/null -obf-verbose -obf-seed=1

Source and full docs (README.md, USER.md for the annotation grammar and pass reference, DEV.md for architecture, VM.md for the bytecode ISA, TESTS.md for the harness) are at github.com/und3ath/ollvm, branches ollvm-21.x / ollvm-22.x.

und3ath.