🔨 I Built a C Compiler in PHP That Can Compile SQLite

on August 30, 2026

I wanted to find out how much of a C compiler I could implement in PHP before it became capable of compiling SQLite.

The answer, as of this writing, is: enough. On macOS ARM64, a PHP-hosted C frontend and ARM64 code generator turn the SQLite 3.46.0 amalgamation into textual assembly. Apple’s assembler and linker finish a native executable. A small harness opens an in-memory database, inserts a row, selects it, and prints:

darkwood

Alongside that acceptance test sits a regression suite of 46/46 passing fixtures. The outer build/validation pipeline is orchestrated with darkwood/flow v8.1.5.

This article is a guided tour through that implementation: how source becomes tokens, how macros expand, how declarators become types, how semantic analysis feeds code generation, how ARM64 is emitted, and which bugs forced the abstractions to become honest. The interesting material is not a dashboard of timings. It is the PHP that implements C.

What “compile SQLite in PHP” actually means

Precision matters, because the phrase is easy to oversell.

The PHP compiler implements:

C source
  → lexer / preprocessor
  → parser
  → AST
  → semantic analysis
  → ARM64 code generation
  → .s assembly text

Native Apple tools then perform:

ARM64 .s
  → Apple assembler (as)
  → Mach-O object

objects
  → system linker via the clang driver
  → native executable

Clang is not used to compile sqlite3.c on the acceptance path. It is used as a link driver, and sometimes as a differential oracle for small probes. PHP does not write Mach-O. The interesting boundary is:

PHP implements the C frontend and ARM64 code generator; the platform assembler and linker finish the native executable.

The deliberately small architecture

There is no CIR, no SSA form, no optimizer pipeline. The driver in src/Compiler/Driver/Compiler.php is almost linear:

$preprocessor = new Preprocessor($this->sourceManager, $options->includePaths, $options->defines);
$tokens = $preprocessor->preprocess($fileId);

$parser = new Parser($tokens, $this->diagnostics);
$decls = $parser->parse();
unset($parser, $tokens);

$sema = new Sema($this->diagnostics);
$tast = $sema->analyze($decls);
$enumConstants = $sema->enumConstants;
unset($sema);

$codegen = new Codegen($enumConstants);
$assembly = $codegen->generate($tast);

Symfony hosts console commands (app:compiler-check, app:compiler-fixtures, app:compiler-sqlite, and friends). It is not part of the compiler algorithm. The interesting code lives under src/Compiler/ as ordinary PHP classes. Headers under include/ provide a minimal Darwin/libc surface for the frontend — enough for stdio, stdlib, threads, and related declarations — not a full SDK rewrite.

Textual assembly was a scoping decision. Emitting .s means inheriting Apple’s assembler diagnostics and a readable debugging path, without building a Mach-O writer in PHP.

As a picture:

                     PHP

sqlite3.c
   │
   ▼
┌──────────────┐
│ Preprocessor │
└──────┬───────┘
       ▼
┌──────────────┐
│    Parser    │
└──────┬───────┘
       ▼
┌──────────────┐
│     AST      │
└──────┬───────┘
       ▼
┌──────────────┐
│     Sema     │
└──────┬───────┘
       ▼
┌──────────────┐
│ ARM64 Codegen│
└──────┬───────┘
       │
       ▼
    sqlite3.s

                  macOS toolchain

sqlite3.s → as → sqlite3.o
smoke.s  → as → smoke.o
                 │
                 ▼
              linker → executable → darkwood

Source files and tokens

Compilation begins with source identity, not with grammar.

SourceLoc is deliberately tiny: a file id and a byte offset.

// src/Compiler/Common/SourceLoc.php
final readonly class SourceLoc
{
  public function __construct(
    public int $fileId,
    public int $offset,
  ) {
  }
}

Line and column are computed later by SourceManager::lineCol(). The manager keeps file contents and a cached table of line-start offsets, then binary-searches for the largest start <= offset:

// src/Compiler/Common/SourceManager.php (excerpt)
$starts = $this->lineStarts($loc->fileId);
// Binary search: largest line-start index with start <= offset.
$lo = 0;
$hi = count($starts) - 1;
while ($lo <= $hi) {
  $mid = intdiv($lo + $hi, 2);
  if ($starts[$mid] <= $offset) {
    $lo = $mid + 1;
  } else {
    $hi = $mid - 1;
  }
}
$lineIdx = max(0, $hi);
$lineStart = $starts[$lineIdx];

return ['line' => $lineIdx + 1, 'col' => $offset - $lineStart + 1];

Tokens carry spelling, kind, location, and — critically for the preprocessor — a hide set:

// src/Compiler/Common/Token.php
final readonly class Token
{
  /**
   * @param list<string> $hideSet Names that should NOT be expanded for this token (blue-painting)
   */
  public function __construct(
    public TokenKind $kind,
    public string $spelling,
    public SourceLoc $loc,
    public array $hideSet = [],
  ) {
  }

  public function withHideSet(array $newHideSet): self
  {
    // ...
    return new self(
      kind: $this->kind,
      spelling: $this->spelling,
      loc: $this->loc,
      hideSet: $newHideSet,
    );
  }
}

TokenKind is a small enum: identifier, keyword, integer/float/char/string literals, punctuation, and EOF. Keeping locations on every token is what makes diagnostics of the form file.c:line:column: error: ... possible after later stages fail.

This is also a distinctly PHP shape: immutable value objects, arrays of tokens, and instanceof/match dispatch later in the pipeline.

Building a C preprocessor in PHP

The preprocessor is not string substitution. It is a token machine with conditional compilation and recursive macro expansion.

Preprocessor::processTokens() walks an indexed token stream. A condition stack tracks #if / #elif / #else / #endif with frames of the form {parentActive, anyBranchTaken, currentActive, seenElse}. Only when every frame is currently active do tokens enter the expansion path. Inactive regions still parse directives so nesting stays balanced.

When the region is active, non-directive tokens are batched until the next line-start #, then handed to MacroExpander::expand():

// src/Compiler/Preproc/Preprocessor.php (excerpt)
if ($isActive()) {
  $batch = [];
  while ($i < count($tokens)) {
    $t = $tokens[$i];
    if ($t->kind === TokenKind::Punct && $t->spelling === '#'
        && $this->isAtLineStart($output, $tokens, $i)) {
      break;
    }
    $batch[] = $t;
    ++$i;
  }
  if ($batch !== []) {
    array_push($output, ...$this->expander->expand($batch));
  }
  continue;
}

#if expressions are expanded through the same expander after defined(...) rewriting, then evaluated as constant expressions.

Macro expansion and hide sets

MacroExpander (in MacroTable.php) uses an indexed cursor over the original input plus a reversed pending stack for rescanning. Expansions are pushed in reverse so the next token to consume is the first token of the expansion — classic rescan semantics without repeatedly array_shift()-ing a large list.

When an identifier names a macro and is not disabled by its hide set, the expander unions the macro name into the hide set of the replacement tokens (“blue painting”). That prevents infinite recursion when a macro expands to its own name:

// src/Compiler/Preproc/MacroTable.php (excerpt)
if ($token->hideSet !== [] && in_array($token->spelling, $token->hideSet, true)) {
  $output[] = $token;
  continue;
}

$macro = $this->table->lookup($token->spelling);
// ...
$newHideSet = $this->unionHideSet($token->hideSet, [$macro->name]);
$expanded = $this->expandObjectLike($macro, $token->loc, $newHideSet);
for ($i = count($expanded) - 1; $i >= 0; --$i) {
  $pendingReversed[] = $expanded[$i];
}

Function-like macros parse argument lists, support # stringification and ## pasting in substitution, and paint hide sets onto body tokens. Object-like macros are the simpler sibling of the same mechanism.

Substitution walks the macro body token by token. Parameter names are replaced by the corresponding argument token lists (themselves subject to expansion rules), # stringifies an argument into a string literal token, and ## pastes adjacent tokens into a new spelling. __VA_ARGS__ participates for variadic function-like macros. Every produced token carries a hide set that includes the macro currently expanding, so a later rescan will not re-enter the same name through the same painted token.

The point for a technical reader: this is a real preprocessor subsystem. SQLite’s amalgamation is macro-heavy; without hide sets, argument collection, stringification/pasting, and conditional compilation, the parser would never see a coherent token stream.

One early regression fixture, 011-macro-eof, encodes a class of edge case where EOF interacting with macro replacement and conditional evaluation corrupted preprocessing. The fixture form is the project’s habit: shrink the amalgamation failure into a tiny program that still dies if the preprocessor regresses. Another early fixture, 012-octal-escape, lives one stage later in the lexer/string decoder — "\040" must become the correct byte — but it is part of the same lesson: C’s “boring” lexical rules are load-bearing once real headers appear.

Parsing C without a parser generator

The parser is a hand-written recursive-descent / Pratt hybrid over the token array. There is no yacc grammar. Declarations, statements, and expressions are methods on Parser.

AST nodes are ordinary PHP classes in src/Compiler/Common/Ast.php. Representative shapes:

final class FuncDecl extends Decl
{
    public function __construct(
        public string $name,
        public CType $returnType,
        public array $params,
        public bool $variadic,
        SourceLoc $loc,
        public ?CompoundStmt $body = null,
        public StorageClass $storageClass = StorageClass::None,
        public bool $isInline = false,
    ) {
        parent::__construct($loc);
    }
}

final class BinaryExpr extends Expr
{
    public function __construct(
        public BinaryOp $op,
        public Expr $left,
        public Expr $right,
        SourceLoc $loc,
        public ?CType $resolvedType = null,
    ) {
        parent::__construct($loc);
    }
}

final class CallExpr extends Expr
{
    public function __construct(
        public Expr $function,
        public array $arguments,
        SourceLoc $loc,
        public ?CType $resolvedType = null,
    ) {
        parent::__construct($loc);
    }
}

final class CompoundLiteralExpr extends Expr
{
    public function __construct(
        public CType $type,
        public Expr $initList,
        SourceLoc $loc,
        ?CType $resolvedType = null,
    ) { /* ... */ }
}

Expression nodes may carry resolvedType after semantic analysis; the parser builds structure first. Semantic analysis annotates and type-checks in place rather than producing a wholly separate IR.

Why C declarators are difficult

C’s declarator grammar is the classic trap. These are not the same type:

int *p[4];      /* array of 4 pointers to int */
int (*p)[4];    /* pointer to array of 4 int */

parseDeclarator() implements that distinction. The ungrouped path applies * first, then [N] wraps the current type — so *p[4] becomes an array of pointers. The grouped path (*p) builds a pointer, may collect dimensions inside the parentheses, then applies trailing suffixes with applyArraySuffix(), which rebinds [N] onto the pointee when the current type is already a pointer:

// src/Compiler/Parser/Parser.php
private function applyArraySuffix(CType $type, ?int $count): CType
{
    if ($type instanceof PointerCType) {
        $to = $type->to;
        $array = $count !== null
            ? new ArrayCType($to, $count)
            : new IncompleteArrayCType($to);

        return new PointerCType($array);
    }

    return $count !== null
        ? new ArrayCType($type, $count)
        : new IncompleteArrayCType($type);
}

That helper exists because a naive “apply array suffix to the whole type” fix for int (*p)[N] can accidentally rewrite int *p[N] and break real code — including, historically, the SQLite path. Fixture 051-pointer-to-array locks the correct reading:

int a[2][2] = { {1, 2}, {3, 4} };
int (*p)[2] = a;
return (*p)[0] + (*p)[1];

Compound literals and designated initializers

Compound literals are recognized in cast position: (type){ ... } becomes CompoundLiteralExpr instead of CastExpr when a brace initializer follows the closing parenthesis.

Designated initializers parse a C99 designator-list (.field/[index]repeated, then=) into either a single designator or a nested path stored on InitListExpr. That path is later normalized for nested struct and array members — the machinery behind fixture 044-nested-designated-init`.

Representing C types in PHP

All core type classes live in src/Compiler/Common/CType.php: primitives, PointerCType, ArrayCType, IncompleteArrayCType, FunctionCType, StructCType, UnionCType, EnumCType, qualifiers, and typedef wrappers.

Questions the compiler asks repeatedly — size, alignment, “is this an integer?” — are methods on CType using match (true) and instanceof:

public function isInteger(): bool
{
    return match (true) {
        $this instanceof BoolCType,
        $this instanceof CharCType,
        // ...
        $this instanceof EnumCType => true,
        $this instanceof QualifiedCType => $this->base->isInteger(),
        $this instanceof TypedefCType => $this->base->isInteger(),
        default => false,
    };
}

EnumCType being an integer type matters for ABI: enum parameters must spill from general-purpose registers like other integers. Size/align follow an LP64-ish layout (int 4, pointers 8, enum 4; structs/unions consult their record’s laid-out size and alignment). Arrays multiply element size by count; incomplete arrays have no size.

The C tension between array and pointer shows up everywhere. Arrays decay in many expression contexts, yet array types remain first-class in declarators and in pointer arithmetic scaling. Getting only one of those right is how you ship a compiler that links and still fails *(a+1).

Semantic analysis and symbol resolution

Sema does more than “check types.”

It maintains nested Scope objects (global, function, block), a typedef map, record types, and a public enumConstants table:

// src/Compiler/Sema/Sema.php (excerpt)
final class Sema
{
  private Scope $globalScope;
  private Scope $currentScope;
  /** @var array<string, int> */
  public array $enumConstants = [];

  public function analyze(array $decls): array
  {
    foreach ($decls as $decl) {
      $this->collectTopLevel($decl);
    }
    foreach ($decls as $decl) {
      $this->analyzeDecl($decl);
    }
    return $decls;
  }
}

Pass one inserts top-level functions, variables, and typedefs, and registers enum cases. Pass two type-checks bodies. Enum registration writes both the flat map and a global Symbol::enumConstant(...):

private function registerEnumConstants(EnumType $enumType): void
{
  foreach ($enumType->cases as $case) {
    $this->enumConstants[$case->name] = $case->value;
    $this->globalScope->insert(Symbol::enumConstant(
      name: $case->name,
      value: $case->value,
      type: IntCType::$instance,
    ));
  }
}

Typedef enums and anonymous enums nested in structs also feed that registration path. Identifier lookup walks scopes, then falls back to enumConstants so VALUE in int result = VALUE; resolves even when the only declaration was an enum case.

Scope is a parent-linked map of Symbol values (Variable, Function, Typedef, EnumConstant). Function bodies open a function scope under global; compound statements nest further. Parameters are inserted before the body is analyzed.

The driver then hands enumConstants to codegen explicitly:

$codegen = new Codegen($enumConstants);

That design makes constant folding of enum names in static initializers a shared contract between sema and the backend, not a second independent lookup table invented during emission.

A concrete travel path for a global enum initializer looks like this:

enum { VALUE = 42 };
int result = VALUE;

The parser builds an EnumDecl / enum type with a case VALUE = 42, then a VarDecl whose initializer is an Identifier("VALUE"). Sema’s first pass registers VALUE → 42 in enumConstants and in the global scope. The second pass type-checks the initializer as an integer. Codegen, when emitting the global result, resolves the identifier through the enum constant map and emits a numeric directive — typically .long 42 — rather than a symbolic relocation. Fixture 015-enum-global exists so that path cannot silently regress to “undeclared identifier” or “emit zero.”

Expression typing also covers the mundane hazards that SQLite still depends on: array decay in most expression contexts, lvalue requirements for assignment and unary &, and statement-position oddities such as (void)sizeof(x); (fixture 013-sizeof-stmt). sizeof is parsed as a unary operator that may take either an expression or a parenthesized type name; getting that classification wrong at statement position is a parser bug that never shows up in return 42.

Constant expressions and static initialization

SQLite lives on static data: tables, strings, pointers into arrays, compound initializers. Global emission is therefore not a footnote.

Codegen::emitInitializer() chooses assembly directives from the target type. Narrow integers and characters become .byte; wider integers become .long / .quad; arrays recurse per element and .zero any remaining tail; string literals into char arrays copy bytes and pad. Char array globals are asserted by fixture 010-char-array-global, which requires .byte and forbids .quad 84 in the assembly.

A live compile of that fixture currently emits:

.section __DATA,__data
.globl _table
.p2align 3
_table:
.byte 84
.byte 92
.byte 134
.byte 0

That output depends on instanceof ArrayCType matching the real class in App\Compiler\Common. In PHP, a bare ArrayCType name inside App\Compiler\CodeGen resolves to App\Compiler\CodeGen\ArrayCType unless imported. A missing:

use App\Compiler\Common\ArrayCType;

made the array branch unreachable. Globals fell through a wider path and emitted .quad where .byte belonged. The program could assemble and link. Runtime layout was still wrong. This is a cross-language bug in the purest sense: PHP name resolution corrupted C static data layout.

Pointer global initializers (&arr[i]), tentative definitions, and global compound literals are sibling problems in the same neighborhood: the backend must understand not only “emit an integer,” but “emit a relocatable address,” “merge tentative definitions,” and “materialize a constant aggregate in the data section.”

From typed AST to ARM64

The backend is src/Compiler/CodeGen/Codegen.php, with register conventions in Arm64.php:

const SCRATCH_REGS = [
    Arm64Reg::X9, Arm64Reg::X10, Arm64Reg::X11, Arm64Reg::X12,
    Arm64Reg::X13, Arm64Reg::X14, Arm64Reg::X15,
];

const ARG_REGS = [
    Arm64Reg::X0, Arm64Reg::X1, Arm64Reg::X2, Arm64Reg::X3,
    Arm64Reg::X4, Arm64Reg::X5, Arm64Reg::X6, Arm64Reg::X7,
];

RegAlloc hands out scratch registers. Arguments arrive in x0–x7 per AAPCS64-style usage in this backend. Returns use x0 (and x1 for larger small aggregates). Assembly is accumulated as strings and flushed as text — another PHP-native choice.

A concrete function

Given:

int add(int a, int b)
{
    return a + b;
}

this compiler emits (representative):

.text
.globl _add
.p2align 2
_add:
stp x29, x30, [sp, #-16]!
mov x29, sp
sub sp, sp, #16
str x0, [x29, #-8]
str x1, [x29, #-16]
add x9, x29, #-8
ldrsw x9, [x9]
add x10, x29, #-16
ldrsw x10, [x10]
add x9, x9, x10
mov x0, x9
mov sp, x29
ldp x29, x30, [sp], #16
ret

Reading that against the implementation:

1. Prologue saves frame pointer and link register, sets x29
2. Locals/parameters get negative offsets from x29; arguments are stored from x0/x1
3. Loads use ldrsw for signed 32-bit values into 64-bit registers
4. add produces the sum in a scratch register, then moves it to x0
5. Epilogue restores sp from x29 and returns

Frame size is computed from local needs and patched into a placeholder sub sp, sp, #... emitted early in emitFunction(). Alignment to 16 bytes matters on AArch64; the backend rounds spill/aggregate temporary slots accordingly.

Stack frames and local variables

Local allocation is deliberately simple and explicit. emitFunction() resets per-function state, emits a placeholder frame subtraction, then discovers how much space parameters, locals, and temporaries need:

$this->emitLine('stp x29, x30, [sp, #-16]!');
$this->emitLine('mov x29, sp');
$this->emitLine('sub sp, sp, #0  ; FRAME_SIZE_PLACEHOLDER');
$this->framePlaceholderIndex = count($this->lines) - 1;

ensureLocalSpace() grows a running localOffset, aligning each reservation to 8 bytes. allocLocal() stores the chosen negative offset from x29 in localVarOffsets:

private function ensureLocalSpace(int $size): void
{
    $aligned = ($size + 7) & ~7;
    $this->localOffset += $aligned;
    if ($this->localOffset > $this->frameSize) {
        $this->frameSize = $this->localOffset;
    }
}

private function allocLocal(string $name, CType $type): void
{
    $size = $type->sizeInBytes() ?? 8;
    $this->ensureLocalSpace($size);
    $offset = -$this->localOffset;
    $this->localVarOffsets[$name] = $offset;
    $this->localVarTypes[$name] = $type;
}

Parameters arriving in x0–x7 are immediately spilled into those slots (str x0, [x29, #-8], and so on). That gives every named local a stable address for &var, for taking field addresses, and for reload after calls. Large struct returns (>16 bytes) also reserve space for the invisible x8 sret pointer and store it early.

When the function body is fully emitted, the placeholder line is spliced with the real sub sp, sp, #<rounded frame>. The frame size is rounded up so sp stays 16-byte aligned — a hard requirement on AArch64. Epilogue restores sp from x29, pops the saved pair, and rets.

Compound-literal temporaries, small-aggregate return slots, and call argument staging all compete for the same frame machinery. That is why ABI bugs often look like “random” stack corruption until you notice a temporary was allocated with the wrong size or a nested call clobbered a slot that still held an unevaluated argument.

Pointer arithmetic

Binary + / - involving pointer-like operands scale the integer side by pointee size. Arrays count as pointer-like for that purpose:

// src/Compiler/CodeGen/Codegen.php (excerpt)
$leftIsPtrLike = $leftType->isPointer() || $leftType->isArray();
$rightIsPtrLike = $rightType->isPointer() || $rightType->isArray();
// ...
} elseif ($ptrType instanceof ArrayCType || $ptrType instanceof IncompleteArrayCType) {
    $t = $ptrType->of->unqualified();
    $pointeeSize = $t->isPointer() ? 8 : ($t->sizeInBytes() ?? 4);
}
if ($pointeeSize > 1) {
    // lsl #1/#2/#3 or mul by immediate size
}

Before Loop 32’s fix, *(a+1) could emit add ..., #1 in bytes. Fixture 035-array-ptr-arith now requires element scaling.

Calling functions on ARM64

emitCallExpr() evaluates arguments, spills them through the stack in a disciplined order, loads x0–x7 (and stack args beyond), then either bl _name for direct calls or blr xn for indirect calls.

Function pointers are first-class in the type system (PointerCType to FunctionCType) and in codegen. A minimal example:

int call(int (*fn)(int), int value)
{
    return fn(value);
}

emits an indirect call sequence that loads the callee address and uses blr — not bl to a fixed symbol. Direct calls from main to call still use bl _call, while taking the address of id uses adrp/add with @PAGE / @PAGEOFF relocations.

Variadic calls

Variadic calls are sharper. Argument evaluation can have side effects; register and stack lifetimes interact badly with naive left-to-right emission that clobbers earlier results. Fixture 016-variadic-spill is the reduced form:

printf("%d%d%d", next(), next(), next());
return counter == 3 ? 42 : 1;

Expected stdout is 123 with exit 42. The backend’s printf-shaped path evaluates arguments into stack temporaries before assembling the final variadic home, so increments are not lost to register reuse. Generating syntactically valid ARM64 is much easier than implementing a calling convention that survives side-effecting arguments.

Small aggregate returns

Apple ARM64 returns aggregates of at most 16 bytes in x0 (and x1 if larger than 8). emitSmallAggregateReturn() materializes the value into a stack slot, then loads x0/x1:

} elseif ($value instanceof CallExpr) {
    $this->emitExpr($value);
    $this->emitLine('str x0, ['.$addrReg->x().']');
    if ($size > 8) {
        $this->emitLine('str x1, ['.$addrReg->x().', #8]');
    }
}
// ...
$this->emitLine('ldr x0, [sp]');
if ($size > 8) {
    $this->emitLine('ldr x1, [sp, #8]');
}

The CallExpr branch exists because forwarding return a(); inside another struct-returning function must preserve the full register result of the nested call. Storing only a 32-bit view of x0 loses a field. That is Loop 48 / fixture 054-struct-return-chain.

In the generated assembly for b, you can see the pattern: bl _a, then str x0, [...], then ldr x0, [sp] before returning — the small aggregate is treated as a register-width value, not as a single w0 leftover from a scalar mindset.

Structs, unions, and compound literals

Struct and union layout is computed while parsing the record definition, then stored on a RecordType consulted later by StructCType / UnionCType. For structs, each field is aligned to its own alignment, offsets accumulate, and the total size is rounded to the record’s maximum alignment:

// src/Compiler/Parser/Parser.php (struct field layout excerpt)
$fieldAlign = $fieldType->alignOf() ?? 1;
$fieldSize = $fieldType->sizeInBytes() ?? 0;
$maxAlign = max($maxAlign, $fieldAlign);
$byteOff = ($byteOff + $fieldAlign - 1) & ~($fieldAlign - 1);
$fields[] = new RecordField(
    name: $fieldName,
    type: $fieldType,
    bitWidth: null,
    offset: $byteOff,
    bitOffset: 0,
);
$bitOffset = ($byteOff + $fieldSize) * 8;
// ...
$totalBytes = intdiv($bitOffset + 7, 8);
$totalBytes = ($totalBytes + $maxAlign - 1) & ~($maxAlign - 1);
$rec = new RecordType(name: $tag ?? '', fields: $fields, size: $totalBytes, alignment: $maxAlign);

Unions take the max member size (again rounded to max alignment) and place every field at offset 0. Bit-fields exist in the parser’s struct path as well; they track bit offsets inside storage units. Codegen uses field offsets when emitting member access (add of a constant to a base address) and when laying out global struct initializers (inserting .zero padding between designated field positions).

That layout information is what makes aggregate arguments, returns, and compound literals possible at all: without trustworthy offsets, “field b of an 8-byte struct” is guesswork.

Compound literals are lowered by materializing a temporary:

// emitAddr() path for CompoundLiteralExpr
if ($expr instanceof CompoundLiteralExpr) {
    // allocate frame slot, compute address in a register
    $this->emitLocalInit($reg, $expr->initList, $expr->type);
    return $reg;
}

Passing (struct S){20, 22} as an argument therefore means: build the temporary, then pass the aggregate from memory into argument registers or stack according to size — not “evaluate the initializer list as if it were the first field only.” Fixture 034-compound-lit-arg exists because that mistake returned 20 instead of 42.

Nested designated initializers ({.i.y = 2, .i.x = 1}, {.a[1] = 40}) require the initializer normalizer to walk designator paths into nested structs and arrays, including through anonymous members when present. Fixture 044-nested-designated-init is the reduced probe.

The native toolchain boundary

Toolchain is intentionally thin:

// src/Compiler/Platform/Toolchain.php
public function assemble(string $asmPath, string $objectPath): ProcessResult
{
    return $this->run(['/usr/bin/as', $asmPath, '-o', $objectPath]);
}

public function link(array $objectPaths, string $outputExecutable, array $extraArgs = []): ProcessResult
{
    $args = array_merge(
        ['/usr/bin/clang', ...$objectPaths, '-o', $outputExecutable, '-lm', '-lpthread', '-ldl'],
        $extraArgs
    );
    return $this->run($args);
}

run() uses proc_open, captures stdout/stderr, and returns an exit code. That is enough. PHP writes .s; as writes .o; the clang driver links. Clang still does not compile the C source on the acceptance path.

SQLite as an integration oracle

Compiling return 42 proves the pipe is connected. Compiling SQLite proves subsystems interact.

The harness (fixtures/compiler/sqlite_smoke_test.c) opens :memory:, creates a table, inserts 'darkwood', selects it, and prints the text. That exercises a huge translation unit, macro-heavy headers, global data, function pointers, structs, enums, and enough of the calling convention that wrong spills fail at runtime even when assembly “looks fine.”

SQLite is not a standards suite. It is an integration oracle: many features must be simultaneously almost right, or the smoke test fails.

Bugs the implementation exposed

The strongest technical material in this project is the set of bugs that only became obvious when abstractions collided. Each case below follows the same shape: small C program, expected behavior, wrong behavior, root cause in PHP, fix, fixture.

Case study: ArrayCType and corrupted static data

Program (010-char-array-global):

static const unsigned char table[4] = {84, 92, 134, 0};
int main(void) { return table[0]; }

Expected: exit 84; assembly contains .byte, not .quad 84.

Failure mode: missing use App\Compiler\Common\ArrayCType; in Codegen.php made instanceof ArrayCType test the wrong class. The array initializer branch never ran. Wider directives laid out the table incorrectly.

Fix: import the real class; emit per-element .byte for char arrays. Retained explicitly in the project status notes and locked by fixture assembly assertions.

Lesson: a PHP namespace mistake is a C ABI/layout bug. “Links” is not “works.”

Case study: array pointer arithmetic scaling

Program (035-array-ptr-arith):

int a[3] = {10, 20, 12};
int *p = a + 1;
return *(a + 2) == 12 && *p == 20 ? 42 : 0;

Expected: exit 42 (also compared with clang).

Failure mode: a + 1 advanced by bytes because arrays were not treated as pointer-like for scaling.

Fix: in emitBinaryExpr, treat ArrayCType like a pointer for +/-, scale by sizeof(element) via lsl/mul.

Lesson: array decay and pointee scaling are not the same bug, but they live next door.

Case study: pointer-to-array declarations

Program (051-pointer-to-array):

int (*p)[2] = a;
return (*p)[0] + (*p)[1];

Expected: match clang.

Failure mode: declarator suffix application that repaired (*p)[N] could break *p[N] if applied too broadly.

Fix: close the grouped (*name) declarator before trailing array suffixes; use applyArraySuffix() so [N] binds to the pointee of a pointer type. Keep the ungrouped path constructing array-of-pointer the ordinary way.

Lesson: C declarators are a parser problem with semantic consequences; fixtures must cover both readings.

Case study: compound literal arguments

Program (034-compound-lit-arg):

return f((struct S){20, 22});

Expected: 42.

Failure mode: evaluating a compound literal / init list as a value returned only the first field (20).

Fix: materialize the compound literal into a frame temporary in emitAddr, then pass the ≤16-byte struct from memory into argument registers.

Lesson: ephemeral aggregates need addresses, not “first scalar.”

Case study: nested designated initializers

Program (044-nested-designated-init):

struct Outer o = {.z = 3, .i.y = 2, .i.x = 1};
struct Arr s = {.a[1] = 40, .a[0] = 2};

Expected: field and array slot values match clang’s reading.

Failure mode: single-level designators worked; nested paths and array designators inside structs did not.

Fix: parse designator-lists into paths; normalize those paths when applying initializers, including through nested and anonymous members.

Lesson: C99 initializer sugar is not optional if you claim “real C” against SQLite-shaped code.

Case study: variadic spills

Program (016-variadic-spill):

printf("%d%d%d", next(), next(), next());

Expected: stdout 123, exit 42.

Failure mode: argument evaluation order / register lifetime destroyed earlier next() results before the call.

Fix: evaluate into stack temporaries, then assemble the variadic call home (the printf-shaped path in emitCallExpr).

Lesson: side-effecting arguments are an ABI test, not a parsing test.

Case study: struct return chains

Program (054-struct-return-chain):

struct S b(void) { return a(); }
// main: struct S s = b(); return s.a + s.b;

Expected: exit 3 (clang oracle).

Failure mode: nested aggregate return preserved only part of x0 (symptomatically exit 2).

Fix: in emitSmallAggregateReturn(), when the returned expression is a CallExpr, store full x0 and, if needed, x1 before reloading for the current function’s return.

Lesson: generating ret after bl is not the same as implementing aggregate return forwarding.

From failures to regression fixtures

The methodology is more important than any single KEEP:

SQLite failure (or clang mismatch)
  → identify subsystem
  → reduce to a tiny C program
  → compare observable behavior
  → create fixtures/compiler/NNN-*.c + .json
  → fix one narrow behavior
  → run app:compiler-fixtures
  → run app:compiler-sqlite
  → KEEP only if still valid

Fixture metadata can assert exit codes, stdout, assembly substrings, and compareWithClang. Assembly assertions catch layout bugs that a lucky exit code might miss. The suite ending at 46/46 is not a vanity count; it is a memory of interactions that already bit the amalgamation once.

Differential probes

For small programs, compiling and running the same source with clang provides an executable oracle: exit code and stdout. Commands such as app:compiler-compare-clang and probe flows exist to support that workflow. Clang is a runtime differential tool here, not the compiler of sqlite3.c on the acceptance path.

That distinction matters. Differential testing answers “does our ABI match observable behavior on this probe?” Acceptance testing answers “does our compiler compile SQLite?”

Where Darkwood Flow fits

Compilation contains two different pipelines. Confusing them produces either useless abstraction or a private reimplementation of something that should be shared.

The algorithmic pipeline is tightly coupled:

tokens → AST → semantic model → assembly

Those stages exchange compiler-specific structures — Token lists, Decl trees, CType graphs, register allocators. Wrapping every parse step in an orchestration framework would clarify nothing. The hide-set algorithm does not become better because it travels through a job queue.

The operational pipeline is different:

compile → assemble → link → execute → validate

Each step has a clear artifact boundary, can fail independently, can record timing on shared state, and can be replaced without rewriting the lexer. That is where Darkwood Flow belongs.

This project depends on Composer package darkwood/flow v8.1.5. The compiler core under src/Compiler/ does not import Flow. The outer workflows under src/Flow/ do.

SqliteValidationFlow yields timed jobs. The API surface is intentionally small: a FlowFactory builds a flow from a generator of jobs; each job receives an Ip (information packet) carrying SqliteValidationState; await() runs the sequence to completion:

// src/Flow/SqliteValidationFlow.php
$flow = (new FlowFactory())->create(function () {
    yield $this->timed('compile_sqlite', $this->compileSqlite);
    yield $this->timed('assemble_sqlite', $this->assembleSqlite);
    yield $this->timed('compile_harness', $this->compileHarness);
    yield $this->timed('assemble_harness', $this->assembleHarness);
    yield $this->timed('link', $this->linkSmokeExecutable);
    yield $this->timed('run', $this->runSmokeTest);
});

$flow(new Ip($state));
$flow->await();

The timed() wrapper is ordinary PHP around a JobInterface: skip remaining work if failure is already set, invoke the job, store elapsed milliseconds on $state->timings. Concrete jobs (CompileSqlite, AssembleSqlite, CompileHarness, AssembleHarness, LinkSmokeExecutable, RunSmokeTest) call into Compiler and Toolchain. Flow does not know what a Token is.

From the user’s point of view there is one command:

php -d memory_limit=4G bin/console app:compiler-sqlite

Internally that expands into independently meaningful stages sharing SqliteValidationState: PHP produces sqlite3.s, as produces sqlite3.o, PHP produces harness assembly, as produces the harness object, the linker produces an executable, the process prints darkwood, validation asserts success. Sibling flows (CompileFlow for single-file compile/assemble/link, ProbeRunFlow for clang baseline probes) apply the same idea at smaller scale.

There was an architectural correction: a local miniature Flow runtime under src/Flow/Runtime/ was removed. The compiler experiment should consume the real Composer package rather than maintain a private copy. The finished separation is:

sqlite-compiler-php
  ├── src/Compiler/   plain PHP algorithms
  └── src/Flow/       darkwood/flow orchestration

Equally important is what Flow was not asked to do. It does not replace AST nodes, implement macro expansion, allocate registers, emit ARM64, replace Symfony, or replace the OS toolchain. It is not a performance layer and was never introduced to make SQLite compile faster. Its value is structure and visibility around a multi-stage native build — a practical use of Darkwood Flow on something very different from a web request.

In short: the compiler shows what native PHP can do; Darkwood Flow turns those capabilities into a repeatable experiment.

The iteration loop

Development proceeded as a discovery process over the implementation, not as a single inspired rewrite.

The pattern is differential compiler development:

construct probe
  → compile with this compiler
  → compile/run with clang when useful
  → compare observable behavior
  → isolate mismatch
  → reduce fixture
  → inspect parser / sema / codegen
  → modify one narrow behavior
  → run fixture suite
  → run SQLite
  → KEEP only if still valid

The useful claim is not that a model can emit thousands of lines of PHP. It is that an agent can repeatedly interact with real compiler code, real as/clang, real fixtures, and a large acceptance target — then keep only changes that survive validation.

Discipline still appeared as refusal to invent late work, and as stopping when further changes were no longer justified by failing probes.

The wake loop

After correctness-focused engineering loops through Loop (struct return chain / 054-struct-return-chain), a wake loop continued to resume an agent every five minutes with instructions to probe for the next highest-value improvement.

Three terms must stay distinct:

Term	Meaning
Wake-loop tick	A periodic resume: probe, validate, decide whether anything is worth changing.
Engineering loop	A numbered KEEP entry .
Retained change	Code that survived validation and was logged KEEP.

Ticks are not code changes. The wake loop was terminated. Between those facts sits a long stretch of ticks that kept finding the suite green — 46/46, SQLite still printing darkwood — without a mismatch valuable enough to justify another KEEP.

If the wake loop had kept inventing refactors without a failing probe, it would have been noise dressed as progress. The interesting ending is methodological:

eventually the feedback loop converged toward no-change decisions.

What the compiler supports

Enough of C to compile and run the SQLite 3.46.0 amalgamation smoke path on macOS ARM64, plus a 46-fixture regression net covering preprocessor edges, globals, compound literals, enums, variadic spill patterns, declarators, designated initializers, and aggregate ABI cases.

What it deliberately does not try to be

• Not a claim of full C99/C11/C17 compliance
• No CIR/IR optimizer by design
• Not a replacement for Clang/GCC in production
• Not a complete Darwin SDK compatibility story
• Floating-point ABI and other larger gaps remain outside the “SQLite smoke + current fixtures” bar
• AI does not remove the need for executable acceptance tests

Prefer the accurate phrase: enough of the C subset exercised by SQLite and the current regression fixtures.

What I learned

PHP can host a non-trivial compiler implementation using ordinary objects, arrays, instanceof dispatch, string-built assembly, and proc_open for native tools. A direct lexer → preprocessor → parser → sema → ARM64 path can handle enough C to compile SQLite’s amalgamation and run a real smoke harness.

The hardest bugs were not “forgot a semicolon in the emitter.” They were interactions: PHP namespaces vs C layout, array vs pointer equivalence, declarator trees, aggregate ABI forwarding, variadic evaluation order. SQLite forces those interactions. Fixtures remember them.

Darkwood Flow earned its place as the outer pipeline. It did not belong inside the token stream.

The late wake loop taught a quieter lesson. Probing without a retained change is evidence that the feedback loop is working — and that the experiment has reached a stopping point worth respecting.

Where I would take it next

Honest next milestones are larger: broader floating-point ABI, wider C coverage beyond the SQLite-shaped subset, perhaps streaming or leaner emission strategies. None of those are “one more five-minute KEEP” candidates, and none should restart without the same fixture-plus-smoke gate.

For Darkwood, the reusable takeaway is architectural: keep compiler algorithms as plain PHP; keep operational pipelines in Flow; insist that “works” means fixtures plus a brutal acceptance test — not merely that something assembled.

Sources

• Code source : https://github.com/matyo91/sqlite-compiler-php
• Slides : https://github.com/matyo91/slidewire

• Post a comment