Migration System

30.1 Overview

The ADL migration system automatically generates and executes database schema changes when a domain is updated. When you submit a new version of a domain YAML, the runtime compares the new schema against the stored snapshot from the previous version, generates DDL statements to transform the database, and executes them in a transaction.

The migration system is content-aware: it detects schema drift even when the version string doesn’t change. If you resubmit version 1.0.0 with a modified schema, the system detects the change and generates a migration.

30.2 The Three-Stage Pipeline

New domain YAML
    │
    ▼
┌─── Parser ─────────────┐
│ Parse YAML → DomainModel│
└──────────┬──────────────┘
           │
           ▼
┌─── SchemaDiffer ──────────┐
│ Compare new vs stored model│
│ → SchemaDiff               │
└──────────┬─────────────────┘
           │
           ▼
┌─── MigrationGenerator ────┐
│ SchemaDiff → MigrationPlan │
│ (DDL statements)           │
└──────────┬─────────────────┘
           │
           ▼
┌─── MigrationExecutor ─────┐
│ BEGIN TRANSACTION          │
│ Execute each statement     │
│ COMMIT or ROLLBACK         │
└──────────┬─────────────────┘
           │
           ▼
Store new model snapshot
Log migration entry
Activate domain

Stage 1: Schema Differ

The SchemaDiffer compares the new DomainModel against the stored snapshot and produces a SchemaDiff:

struct SchemaDiff {
    std::vector<ResourceChange>      resource_changes;
    std::vector<FieldChange>          field_changes;
    std::vector<IndexChange>          index_changes;
    std::vector<RelationshipChange>   relationship_changes;
};

Each change carries metadata about what changed (resource name, field name, old value, new value).

Stage 2: Migration Generator

The MigrationGenerator processes the SchemaDiff and produces a MigrationPlan:

struct MigrationPlan {
    std::vector<MigrationStatement>   statements;
    std::vector<MigrationWarning>     warnings;
    std::vector<MigrationError>       errors;
};

If errors is non-empty, the migration is refused and the domain submission fails with HTTP 422.

Stage 3: Migration Executor

The MigrationExecutor wraps all statements in a single transaction:

BEGIN TRANSACTION;
    statement_1;
    statement_2;
    ...
    statement_n;
COMMIT;

If any statement fails, the entire transaction is rolled back and the database remains in its original state.

30.3 Change Detection

The differ detects six categories of changes:

Resource Changes

Field Changes

Index Changes

Relationship Changes

30.4 Field Rename Detection

Field renaming requires an explicit hint because the differ cannot distinguish a rename from an add+remove pair. Use the renamedFrom property:

# v1.2.0
resources:
  Book:
    object:
      pages: { type: integer }

# v1.3.0
resources:
  Book:
    object:
      pageCount:
        type: integer
        renamedFrom: pages

The differ detects the renamedFrom hint, matches it against the removed fields list, and emits a single Modified change instead of separate Added and Removed changes. The generator produces:

ALTER TABLE library_books RENAME COLUMN pages TO pageCount

This is an O(1) operation — no data is moved. The renamedFrom hint is a one-time migration directive, not a permanent field attribute. After the migration executes, the stored snapshot contains pageCount without the hint.

30.5 The Four-Step Column Rebuild

SQLite does not support in-place constraint modification. When a field’s type, nullability, or FK cascade behavior changes, the migration system uses a four-step column rebuild:

1. ALTER TABLE {table} ADD COLUMN {field}_new {new_type} {new_constraints}
2. UPDATE {table} SET {field}_new = {field}
3. ALTER TABLE {table} DROP COLUMN {field}
4. ALTER TABLE {table} RENAME COLUMN {field}_new TO {field}

Example: Cascade Rule Change

# v1.8.0
relationships:
  book:
    type: belongsTo
    target: Book
    onDelete: restrict

# v1.9.0
relationships:
  book:
    type: belongsTo
    target: Book
    onDelete: cascade

The differ detects CascadeChanged. The generator produces:

ALTER TABLE library_loans ADD COLUMN bookId_new TEXT
    REFERENCES library_books(id) ON DELETE CASCADE
UPDATE library_loans SET bookId_new = bookId
ALTER TABLE library_loans DROP COLUMN bookId
ALTER TABLE library_loans RENAME COLUMN bookId_new TO bookId

All four statements execute in a single transaction. The column name remains the same; only the FK constraint changes.

SQLite Version Requirements

• ALTER TABLE RENAME COLUMN requires SQLite 3.25.0+ (2018)
• ALTER TABLE DROP COLUMN requires SQLite 3.35.0+ (2021)

If a DROP COLUMN statement is reached on an older SQLite build, the executor skips that single statement, logs a warning (“Verify schema manually”), and commits the rest of the migration. The result is a consistent, operational schema in which the obsolete column is simply retained alongside the new one — not a corrupt or half-applied state. To remove the leftover column, upgrade SQLite to 3.35.0+ and re-submit, or drop it manually. The migration as a whole still commits atomically (§30.4); a skip is the only non-fatal deviation, and it is recorded in the migration log detail with a skipped-statement count.

30.6 Destructive Operations

Destructive changes are guarded by default. The guard is enabled with the allow_destructive: true field in the submission request body (there is no X-Allow-Destructive header).

What’s Destructive

Two guard behaviors

Destructive changes fall into two categories, with different default behavior:

• Refused (hard error). A narrowing type change, or adding a required field with no default, is rejected outright with HTTP 422 (error_code: DESTRUCTIVE_MIGRATION). No statements run, no snapshot is stored, and the domain stays at its previous version. The author must either make the change non-destructive (e.g. add a default, rename instead of retype) or opt in with allow_destructive: true.
• Deferred (skipped, submission succeeds). Dropping a removed field, FK, or join table is deferred by default: the DROP COLUMN/DROP TABLE is not generated, a warning is logged, and the rest of the migration commits. The column/table is retained until you re-submit with allow_destructive: true.

Allowing Destructive Changes

POST /api/v1/domains
Content-Type: application/json

{
  "content": "<domain YAML or JSON string>",
  "format": "yaml",            // optional: "yaml" | "json" (auto-detected if omitted)
  "allow_destructive": true
}

With allow_destructive: true, both refused and deferred destructive operations are generated and executed inside the migration transaction.

30.7 Version Tracking

Migration Log Table

CREATE TABLE adl_migration_log (
    id              INTEGER PRIMARY KEY,
    domain_name     TEXT NOT NULL,
    from_version    TEXT NOT NULL DEFAULT '',
    to_version      TEXT NOT NULL,
    executed_at     TEXT NOT NULL,
    status          TEXT NOT NULL DEFAULT 'success',
    detail          TEXT NOT NULL DEFAULT '',
    steps_applied   INTEGER DEFAULT 0,
    steps_skipped   INTEGER DEFAULT 0
);

Each migration is logged with:

• from_version: Previous domain version (empty string for initial bootstrap)
• to_version: New domain version
• executed_at: ISO-8601 timestamp (exposed as applied_at in the migration-status API)
• status: one of success, failed, rolled_back (there is no partial status)
• detail: a DDL summary on success (e.g. "3 statement(s) executed, 1 skipped") or the error message on failure
• steps_applied / steps_skipped: DDL statement counts (queryable via GET /api/v1/domains/{name} migration status)

Domain Storage Table

CREATE TABLE adl_domains (
    name            TEXT PRIMARY KEY,
    version         TEXT NOT NULL,
    adl_json        TEXT NOT NULL,
    yaml_source     TEXT NOT NULL,
    checksum        TEXT NOT NULL,
    status          TEXT NOT NULL,
    created_at      TEXT NOT NULL,
    updated_at      TEXT NOT NULL
);

The adl_json column stores the serialized DomainModel as the snapshot for the next diff. The checksum is a SHA-256 hash of the JSON — this enables content-aware detection.

30.8 Content-Aware Detection

If you resubmit the same version with a modified schema, the system detects the change via checksum comparison:

v1.0.0 submitted → checksum A
v1.0.0 resubmitted with schema change → checksum B

Checksum mismatch → schema drift detected → migration generated

The from_version and to_version are both "1.0.0", but the migration log records the change. This prevents silent schema drift from version-controlled YAML.

30.9 Migration Workflow

Successful Migration

1. Submit new domain YAML
2. Parse → DomainModel
3. Load stored snapshot
4. Differ generates SchemaDiff
5. Generator produces MigrationPlan
6. No errors → Executor runs in transaction
7. Store new snapshot
8. Log migration entry (status: success)
9. Activate domain (routes, actions, schema endpoint)
10. Return HTTP 201

Migration With Skipped Steps (Deferred / Legacy SQLite)

When a destructive statement is deferred (§30.6) or a DROP COLUMN is skipped on legacy SQLite (§30.5), the migration still commits — the domain ends up active, not in any partial state:

1-5. Same as above
6. Warnings present, no errors → Executor runs in transaction
7. Non-destructive statements execute; deferred/unsupported drops are skipped
8. Transaction commits; new snapshot stored
9. Log migration entry (status: success; detail notes skipped count)
10. Log a warning ("Verify schema manually") if any step was skipped
11. Activate domain → status: active
12. Return HTTP 201 (status: "active", ddl_count = statements executed)

Refused Migration (validated before execution)

A destructive or unsafe change that is not opted into is rejected before any DDL runs. The domain is unchanged and stays at its previous version:

1-5. Same as above
6. Plan has errors → refuse
7. Do NOT execute any statements; do NOT store a snapshot
8. No migration-log entry is written
9. Return HTTP 422 (error_code: DESTRUCTIVE_MIGRATION) with details

Failed Migration (execution error)

If a generated statement fails at runtime, the whole transaction is rolled back — there is never a half-applied schema:

1-6. Plan executes in a transaction
7. A statement fails → ROLLBACK entire transaction
8. Do NOT store a snapshot; set domain status: failed
9. Log migration entry (status: failed) with the error
10. Return HTTP 500 (error_code: MIGRATION_ERROR)

Errors That Refuse Migration

• Adding a required field without a default value
• Type narrowing without allow_destructive: true
• Invalid SQL syntax in a custom index
• Circular dependency in computed fields

30.10 Integration Points

End of Chapter 30.

Next: Chapter 31 — Lua Scripting Plugin