Coframe Analytics Data Platform — Design Document, v2.2 Supplement¶

Status: v2.2 supplement (2026-05-25) to coframe_platform_design_v2_0.md + coframe_platform_design_v2_1_supplement.md Author: reeeneeee Compiled: May 2026 Spec authority: Coframe Core Manual (the Manual, chapter 11 on the Metric Engine landed alongside this supplement). Where this document and the Manual disagree, the Manual wins.

Relationship to v2.0 + v2.1: This supplement extends the previous two, picking up exactly where v2.1 §9 left off ("Phase 8/9 — open questions for v2.2+"). It captures the architectural advances landed since v2.1 shipped: a new acceleration component (the Metric Engine), a new persistent-columnar execution backend (DuckDB), refinements to the deployment topology that wire both into the runtime + workbench HTTP surfaces, and a polish pass on the canonical retail demo that exercises all of it end-to-end. Readers should consider v2.0 + v2.1 + v2.2 together as the current platform design baseline.

What this supplement adds:

Phase 8 — The Metric Engine. A per-AC multi-domain query engine layered on Polars + Parquet + a SQLite manifest. Memoises per-(family, anchor) entries, serves via three-branch dispatch (exact match → FD-DAG rollup → backend fallback), composes multi-metric Frames in one operation. Hosts both METRIC and QM (quasi-metadata) domains on a shared substrate.
Phase 9 — The DuckDB backend. Third execution backend after coframe-sqlite (Phase 2) and coframe-polars (Phase 7). Persistent + columnar + SQL-native — the canonical analytic-database archetype, and the right demo posture for an audience used to Snowflake/BigQuery/Postgres-with-columnstore.
Demo Polish (D1-D8). Eight visible polish slices that bring the canonical retail demo current with everything Phase 8 + Phase 9 enabled — engine-on installation, DuckDB backend, served-from indicator, Workbench Engine page, end-to-end smoke test, walkthrough docs, derived-metric example.
Updated package roles + dependency graph. Two new packages: coframe-metric-engine + coframe-duckdb. Engine integration as an optional dependency on existing packages via [engine] extras.
DataAPIBackend protocol — de-facto extensions. Two methods that aren't typed on the formal Protocol but are required-in-practice when the engine is enabled: extract_to_lazyframe(name) → pl.LazyFrame and the operator_registry property. All three backends ship them.
installation.yaml extensions. New metric_engine: block declaring opt-in + byte budget. Engine on-disk layout per design doc §3.4.
Deployment topology. EngineRegistry as a peer to BackendRegistry in both runtime and workbench HTTP apps. Keyed by (installation_id, ac_name). dev_server.py documented as the canonical bootstrap reference.
Cross-backend invariant validation. Three-way triangulation (SQLite ↔ Polars ↔ DuckDB) as the W4 abstraction's load-bearing validation. 6 canonical queries × 3 backend pairs = 18 pairwise Frame comparisons, plus FD-attestation + discovery invariants.
Forward look (v2.3+). AC-level derived metrics in Frame-QL (the resolver work D8 deferred). NLQ surface. MCP surface. Coframe Pro tier items (cross-process engine coordination, backend batching, cross-AC sharing, sketch operators).

1. Phase 8 — The Metric Engine¶

1.1 What it is, briefly¶

The Metric Engine is a per-AC multi-domain query engine with persistent memoisation, layered on Polars + Parquet + a SQLite manifest. Each AC gets its own engine instance bound at COMMIT time per v2.1 §5.2; the engine knows the AC's FD-DAG, declared filter, and name_map, and every memoised entry is pre-scoped to the AC.

The naming choice — Metric Engine over Cache — is deliberate. A cache passively wraps a backend; the engine is the execution substrate for analytical queries when it's enabled. It makes serving decisions (exact match? rollup? backend fallback?), composes multi-metric Frames, applies post-grain operations. Memoisation is a byproduct of execution, not the engine's purpose.

The Manual's new Chapter 11 carries the language-level spec; this supplement focuses on the platform-side concerns (packaging, deployment, opt-in, deployment topology).

1.2 Package: `coframe-metric-engine`¶

File	Role
`engine.py`	The `MetricEngine` class — constructor takes `ac` + `store_root` + optional `max_bytes`. Exposes `serve()`, `compose()`, `ingest()`, `lookup()`, `evict()`, `refresh()`, plus per-domain `profile_table()`, `column_profile()`, `table_profile()` for QM
`types.py`	`Domain` enum (`METRIC` / `QM`), `EngineEntry` (frozen Pydantic), `ServingPath`, `FDStep`, `anchor_signature`
`manifest.py`	`Manifest` class — SQLite-backed catalog of materialised entries; threadsafe within a single Python process via a mutex around writes
`storage.py`	Polars → Parquet IO; `parquet_path_for`, `write_parquet`, `scan_parquet`, `delete_parquet`. Anchor-segmented directory layout per design doc §3.4
`profiling.py`	Shared kind-hint heuristic (the formerly-duplicated per-backend logic, now unified per slice 4b)
`warmup.py`	`pre_materialise()` walker — Phase 8 F2; walks every `MetricFamily.cache_hint.materialize_at` entry and pre-fills the engine
`recommendations.py`	`promotion_recommendations()` — Phase 8 F5; emits paste-able `cache_hint` YAML stanzas for hot entries

Build dependencies: coframe-core (for AC + MetricFamily), coframe-connect (for DataAPIBackend typing), polars, pyarrow. No back-dependency on coframe-resolution — the engine is conceptually upstream of resolution.

1.3 Slice plan (landed)¶

Slice	Deliverable
2	Package skeleton + `Domain` + `EngineEntry` types + stub methods
3	Storage + manifest — Polars→Parquet writes, SQLite manifest CRUD
4	QM as engine-managed substrate — `profile_table()` replaces per-backend kind-hint duplication
4b	Per-backend migration — SQLite + Polars `compute_table_profile` delegates to `engine.profile_table()`
5	METRIC `serve()` with FD-DAG traversal (exact / rollup / backend)
6	METRIC `compose()` — merge + frame_expression + post-grain ops
7	Eviction (LRU + FD-DAG-aware) + stability-window invalidation
8	`coframe-resolution` integration — `execute_query(..., metric_engine=engine)` opt-in

1.4 F-follow-ons (landed)¶

The slice plan above lands the engine as a standalone substrate. The F-follow-ons connect it to the rest of the platform:

F	Deliverable
F1	`installation.yaml metric_engine` block + `EngineRegistry` + runtime/workbench HTTP `/query` wiring
F2	`cache_hint` on `MetricFamily` + COMMIT-time pre-materialisation walker (`pre_materialise`)
F3	FD-DAG rollup at the execute_query layer (widen cache-hit path beyond exact match)
F4	§13.1 full rewire — `build_plan_per_metric` + `compose()` replaces legacy `merge_blocks` + `post_grain_ops` when engine is enabled
F5	Promotion recommendations — engine surfaces its own evidence for `cache_hint` candidates

Phase 8 is complete end-to-end.

1.5 Verification level interaction¶

Per Manual ch. 7 + 11.12: a materialised engine entry inherits the minimum verification level of its source schemas. A partition-invariant rollup (SUM / COUNT / MIN / MAX / BOOL_AND / BOOL_OR) from an AAA-attested source is itself AAA-attestable by construction — the entry is the answer to the sibling-coherence check, so no re-attestation is needed. Non-partition-invariant operators (AVG / MEDIAN / COUNT_DISTINCT) produce entries flagged not_further_rollable so the engine doesn't attempt to roll them up.

1.6 Where the engine sits in the v2.1 §3 L1/L2/L3 layering¶

The engine is best understood as an L2 substrate extension with per-AC scope:

Layer	What v2.1 §3 says	Where the engine fits
L1	Physical data	Engine doesn't touch L1; reads via the backend
L2	Installation-level derived state (raw profiles, table inventory, FD candidates, operator registry binding)	Engine extends L2 with materialised metric values + QM column profiles, but per-AC, not per-installation
L3	Per-AC declarations + overrides + condition statuses	Engine entries are scoped per-AC (one engine instance per AC), so they belong here in spirit — the per-AC scoping is structural, not just convention

A cleaner reading: the engine is L2-shaped substrate, L3-scoped lifecycle. It uses L2 mechanisms (derived state, refreshable cache, well-defined invalidation contract) but the boundary is per-AC, not installation-wide. The design-doc decision to keep one engine per AC for v0.1 (with cross-AC sharing as Coframe Pro tier work) is what locks this into L3-scoped.

2. Phase 9 — The DuckDB backend¶

2.1 Why DuckDB now¶

Three reasons converged:

The demo posture. SQLite is universally familiar but reads as a toy choice for analytics. Polars is in-memory + library-shaped, which doesn't match how an audience pictures a production deployment. DuckDB sits in the same conceptual slot as Snowflake / BigQuery / Postgres-with-columnstore: persistent, columnar, SQL-native, in-process. For a demo trying to land "this is what Coframe over a real warehouse looks like," DuckDB is the right archetype.
W4 triangulation. Two backends (row-oriented SQLite + in-memory columnar Polars) was enough to de-risk the protocol abstraction. Three backends — with DuckDB filling the persistent-columnar slot — triangulates it. If the same query produces identical Frames across three structurally-different engines, the abstraction has earned its keep.
The metric engine surfaces it. With Phase 8 landed, the persistent-columnar slot in the backend cohort matters more — the engine itself is Parquet-based (persistent + columnar), so a backend with the same shape lines up architecturally even if it's a separate component.

2.2 Package: `coframe-duckdb`¶

Mirrors coframe-polars's shape:

File	Role
`backend.py`	`DuckDBBackend` — constructor takes `database` (`:memory:` or path) + optional `read_only` + `max_bytes`. Implements every `DataAPIBackend` protocol method + `extract_to_lazyframe` + `operator_registry` property
`operator_registry.py`	`DUCKDB_OPERATOR_REGISTRY` — L2 binding from L1 catalog ops to DuckDB physical names. Native MEDIAN, BOOL_AND/BOOL_OR, APPROX_COUNT_DISTINCT (no extensions needed)
`loaders.py`	`load_csv` + `load_csv_dir` — uses DuckDB's native `read_csv_auto` inside `CREATE TABLE AS SELECT`. Much simpler than coframe-sqlite's manual regex inference

Build dependencies: coframe-core, coframe-connect, coframe-metric-engine, duckdb>=1.0, polars>=1.0 (for extract_to_lazyframe's Arrow round-trip), pyarrow>=15.

Entry-point: [project.entry-points."coframe.backends"] duckdb = "coframe.duckdb:DuckDBBackend" — discoverable via importlib.metadata alongside the existing sqlite + polars entries.

2.3 Slice plan (landed)¶

Slice	Deliverable
9.1	Package skeleton + entry-point registration + smoke tests
9.2	Discovery (`describe_table`, `enumerate_distinct_values`) + CSV loader
9.3	`aggregate()` + `DUCKDB_OPERATOR_REGISTRY`
9.4	Stability filter (dtype-aware, branches on VARCHAR vs DATE/TIMESTAMP) + FD/sibling attestation
9.5	Engine integration — `extract_to_lazyframe` (via DuckDB's native `.pl()` Arrow round-trip) + `compute_table_profile` delegation
9.6	End-to-end retail demo through `execute_query`
9.7	Cross-backend invariant tests (DuckDB ↔ SQLite ↔ Polars) — 18 pairwise comparisons
9.8	BackendRegistry wiring + entry-point discovery tests

Phase 9 is complete end-to-end.

2.4 Why DuckDB was smaller than Phase 7 was¶

Reference: Polars (Phase 7) ~870 LOC in backend.py; DuckDB (Phase 9) ~500 LOC. Two factors:

SQL generation transferred from coframe-sqlite almost verbatim. Same ANSI identifier quoting, same ? parameter placeholders, same JOIN syntax. Where SQLite uses date('now', '-N days'), DuckDB uses current_date - INTERVAL '<N> days' — the only meaningful dialect difference for our patterns.
extract_to_lazyframe is a one-liner. DuckDB's Python API has .pl() which returns a Polars DataFrame via Arrow. SQLite's equivalent required hand-pivoting rows to columns; Polars's was trivial because the data was already a DataFrame; DuckDB sits in between with native Arrow IO that's even cleaner than either.

The takeaway for adding a fourth backend later (Snowflake / BigQuery / Postgres / Databricks): expect ~400-800 LOC depending on dialect similarity to one of the existing three.

2.5 What DuckDB exposes that the other backends don't¶

Native operator support DuckDB has that SQLite lacks (worked around via MIN/MAX trickery in SQLITE_OPERATOR_REGISTRY) or that Polars partially has:

Op	SQLite	Polars	DuckDB
`MEDIAN`	extension required	native	native
`BOOL_AND`	`MIN(b) = 1` workaround	native	native
`BOOL_OR`	`MAX(b) = 1` workaround	native	native
`APPROX_DISTINCT`	unsupported	unsupported	native (`APPROX_COUNT_DISTINCT`, HLL-based)

Sketch operators (HLL_MERGE / THETA_UNION / T_DIGEST_MERGE) remain unbound in v0.1 — DuckDB's sketch story is evolving and locking a binding ahead of upstream stabilisation invites churn. L3-override available for installations that need them.

3. Demo Polish (D1-D8)¶

Eight visible polish slices that bring the canonical retail demo current with Phase 8 + Phase 9. Cataloged here for completeness; the substantive narrative lives in docs/retail_demo_walkthrough.md and drafts/flywheel_demo_script_v1_0.md.

D	Slice	Where it landed
D1	Retail `installation.yaml` flipped to DuckDB + engine-on; `cache_hint` blocks on revenue + cost	`drafts/data/retail_demo/`
D2	`dev_server.py` rebuilt around DuckDB + `EngineRegistry` + `pre_materialise()` on startup	`packages/coframe-author/scripts/dev_server.py`
D3	Flywheel demo script Stage 7 ("the engine in the loop") + Stage 0 backend note	`drafts/flywheel_demo_script_v1_0.md`
D4	`served_from` indicator (Frame field → JSON → React badge)	execution.py + runtime + workbench + frontend
D5	Workbench Engine page — Manifest panel + Promotion Recommendations panel	`packages/coframe-frontend/src/workbench/EnginePage.tsx`
D6	End-to-end demo smoke test — programmatic bootstrap → query → cache hit → recommendations	`packages/coframe-duckdb/tests/test_demo_smoke.py`
D7	MkDocs retail walkthrough — 7-step copy-paste-able demo	`docs/retail_demo_walkthrough.md`
D8	`profit = revenue - cost` via compose's `frame_expression` hook — programmatic example + integration test	`test_demo_smoke.py` + walkthrough Step 7a

The polish work is what makes Phase 8 + Phase 9 visible in the canonical demo, rather than just present in the codebase. It also surfaces what's not yet there — D8's note that AC-level declaration of derived metrics is forward-compat resolver work.

4. Updated package layout¶

4.1 Dependency graph (replacing v2.0 §1.1)¶

┌──────────────────────────────────────────────────────────────────┐
│                        coframe-frontend                         │
│           (TS/React; Workbench / Management / Query)            │
└────────────────────────────┬─────────────────────────────────────┘
                             │ HTTP/JSON
            ┌────────────────┴────────────────┐
            ▼                                 ▼
   ┌────────────────┐                 ┌────────────────┐
   │ coframe-author │                 │ coframe-runtime│
   │  (workbench    │                 │   (runtime     │
   │   HTTP backend)│                 │  HTTP backend) │
   └───────┬────────┘                 └───────┬────────┘
           │                                  │
           │ ┌────────────────────────────────┘
           ▼ ▼
   ┌────────────────┐    ┌────────────────┐    ┌────────────────────┐
   │coframe-mgmt    │    │coframe-resolut.│    │coframe-metric-     │
   │ (install +     │    │ (resolver +    │◀──▶│ engine (per-AC     │
   │  AC lifecycle) │    │  planner +     │    │  acceleration:     │
   │                │    │  executor)     │    │  serve/compose)    │
   └───────┬────────┘    └───────┬────────┘    └─────────┬──────────┘
           │                     │                       │
           │                     ▼                       │
           │             ┌────────────────┐              │
           │             │coframe-ql      │              │
           │             │(Frame-QL parse)│              │
           │             └────────────────┘              │
           ▼                                             ▼
   ┌──────────────────────────────────────────────────────────────┐
   │                       coframe-core                            │
   │  (Installation + AC + integrity catalog + operator catalog   │
   │   + FD-DAG + quasi-metadata + verification levels)           │
   └──────────────────────────────────────────────────────────────┘
                                ▲
                                │ (all backends depend on coframe-core
                                │  + coframe-connect + coframe-metric-engine)
                                │
   ┌─────────────────┬──────────┴─────────┬───────────────────┐
   ▼                 ▼                    ▼                   ▼
┌──────────┐  ┌──────────────┐  ┌────────────────┐  ┌────────────────┐
│ coframe- │  │coframe-sqlite│  │coframe-polars  │  │coframe-duckdb  │
│ connect  │  │              │  │                │  │  (NEW Phase 9) │
│ (DataAPI │  │              │  │                │  │                │
│ Backend) │  │              │  │                │  │                │
└──────────┘  └──────────────┘  └────────────────┘  └────────────────┘

Changes vs. v2.1: - coframe-metric-engine is new (Phase 8); sits between resolution + backends; conceptually upstream of coframe-resolution (resolution depends on engine via optional [engine] extra) - coframe-duckdb is new (Phase 9); peer to coframe-sqlite + coframe-polars - All three backends now optionally depend on coframe-metric-engine (for extract_to_lazyframe's Polars round-trip + compute_table_profile's delegation to the engine's shared profiling) - coframe-author + coframe-runtime both gain an optional [engine] extra that pulls coframe-metric-engine for EngineRegistry instantiation

4.2 Package roles table (replacing v2.0 §2 table)¶

Package	Role	Status
`coframe-core`	AC catalog (`coframe.installation`), L1 operator catalog (`coframe.operators`), integrity catalog, FD-DAG, verification levels	Phase 1
`coframe-connect`	`DataAPIBackend` protocol + types; `PhysicalBinding` + `compute_effective_registry` (W4)	Phase 2
`coframe-sqlite`	Row-oriented OLTP-shape execution backend (demo + small ACs)	Phase 2
`coframe-polars`	In-memory columnar execution backend (validates the L2 registry abstraction in Phase 7)	Phase 7
`coframe-duckdb`	Persistent columnar execution backend (demo-of-choice; W4 third-axis validation)	Phase 9
`coframe-author`	Workbench HTTP backend — verification ops, session state, `.coframe/` IO	Phase 3-4
`coframe-management`	Installation + AC lifecycle, L2 stability coordination	Phase 3-4
`coframe-runtime`	Runtime HTTP — Frame-QL / NLQ / HTTP API / MCP host	Phase 5-6, W3
`coframe-ql`	Frame-QL lexer / parser / AST	Phase 5
`coframe-resolution`	Four-rule filter, plan construction, execution; optional engine routing	Phase 5 + F4
`coframe-metric-engine`	Per-AC multi-domain query engine (METRIC + QM); acceleration substrate	Phase 8
`coframe-frontend`	TS/React UI for Workbench + AC Management + Query	Phase 4

5. `DataAPIBackend` protocol — de-facto extensions¶

5.1 What's typed vs. what's required-in-practice¶

The formal Protocol in coframe-connect/src/coframe/connect/data_api.py lists 9 methods + 1 attribute. Two additional methods aren't part of the typed Protocol but are required-in-practice when the Metric Engine is enabled:

Method / property	Typed on Protocol?	Required when engine is on?	Implemented by
All 9 protocol methods	Yes	Yes (always)	sqlite + polars + duckdb
`extract_to_lazyframe(name) → pl.LazyFrame`	No (conventional)	Yes — engine's `profile_table()` calls it for QM ingestion	sqlite + polars + duckdb
`operator_registry: dict[str, PhysicalBinding]` (property)	No (conventional)	Yes — planner reads it when no explicit registry passed to `build_plan`	sqlite + polars + duckdb

5.2 Should they be typed on the Protocol?¶

Arguments for promoting both to the Protocol:

They're load-bearing when the engine is enabled, which is the normal case for production
All three existing backends implement them; the cost of typing is zero for the cohort
It clarifies the contract: "to be a Coframe backend, you must implement these"

Arguments against:

The current Protocol is minimal-by-design — backends that only do legacy direct-execute (no engine) can ship without these and still work
extract_to_lazyframe returns a polars.LazyFrame; typing it on the Protocol would force backends to take a hard dependency on polars even if they don't use the engine

Recommendation: leave the Protocol as-is for v2.2. Document the de-facto requirement here. If/when a new backend (Snowflake, etc.) lands without engine integration, we'll see the practical implication and can decide then. The Manual ch. 6 amendment in M1 captures the same de-facto-required posture.

6. `installation.yaml` extensions¶

6.1 The `metric_engine:` block (new)¶

Per Phase 8 F1; extends v2.1 §2.3:

name:         "retail-demo"
backend:
    type:     duckdb
    source:   ./retail_demo.duckdb

stability_filter:
    default_hold_off_days: 7
    overrides:
        transactions: { hold_off_days: 7 }

metric_engine:                    # NEW in v2.2
    enabled:           true       # default: false (engine opt-in per design doc §13.2)
    max_bytes_per_ac:  1073741824 # 1 GiB per design doc §10.3; per-AC byte budget for evict()

acs:
    - path:  ../retail.coframe/

Coframe-core's MetricEngineConfig Pydantic model carries the schema; Installation.metric_engine is the runtime property accessor.

6.2 Engine on-disk layout (per design doc §3.4)¶

<installation>/.coframe/metric_engine/<ac_name>/
├── manifest.sqlite                  # entry catalog + metadata
└── data/
    ├── <domain>/
    │   └── <dataset_id>/
    │       └── <anchor_signature>/
    │           └── part-000.parquet
    └── ...

The <store_root> is configured per-installation; dev_server.py defaults to .coframe-dev-data/metric_engine/. Each (installation_id, ac_name) pair gets its own subdirectory under the store root. Per design doc §3.3, v0.1 is single-process per AC; multi-process coordination is Coframe Pro tier territory.

7. Deployment topology¶

7.1 The dev_server.py shape (canonical reference)¶

The packages/coframe-author/scripts/dev_server.py script is the canonical reference for what a Coframe deployment looks like end-to-end. Per Tier-1 D2:

# 1. CSV → DuckDB (idempotent — skip if exists)
db_path = data_dir / "retail.duckdb"
load_csv_dir(csv_dir, db_path)

# 2. BackendRegistry — per-request DuckDB connection (read-only)
registry = BackendRegistry()
def make_backend():
    return DuckDBBackend(database=db_path, read_only=True)
registry.register("retail-demo", make_backend)

# 3. EngineRegistry — per-(installation_id, ac_name) engine cache
engine_store = data_dir / "metric_engine"
engine_registry = EngineRegistry(engine_store)

# 4. Pre-materialise the AC's cache_hint grains (Phase 8 F2)
engine = engine_registry.get_or_create(installation_id, ac)
pre_materialise(ac=ac, backend=make_backend(), engine=engine, ...)

# 5. Compose workbench + runtime apps
workbench_app = create_app(
    backend_registry=registry,
    installation_loader=loader,
    engine_registry=engine_registry,
)
runtime_app = create_runtime_app(
    backend_registry=runtime_registry,
    installation_loader=loader,
    engine_registry=runtime_engine_registry,
)
workbench_app.mount("/runtime", runtime_app)

# 6. Serve
uvicorn.run(workbench_app, host="127.0.0.1", port=8000)

In production, the workbench + runtime would deploy as separate processes (per v2.1 amendment §10 W3). The dev_server composes them onto one uvicorn instance so the Vite frontend dev server only needs one proxy target.

7.2 EngineRegistry — structurally parallel to BackendRegistry¶

	BackendRegistry	EngineRegistry
Keyed by	`installation_id`	`(installation_id, ac_name)`
Value	Backend factory (zero-arg callable returning a `DataAPIBackend`)	`MetricEngine` instance, lazily constructed on first `get_or_create`
Lifecycle	Per-request factory call (backends are cheap to construct)	Per-app-instance cache (engines are stateful + per-AC)
Defined in	`coframe-runtime` + `coframe-author` (structurally identical, mirrored to avoid cross-deps)	Same — duplicated in both packages
Optional?	No (required for `/query` to work)	Yes — `create_app(engine_registry=None)` keeps the engine path off

7.3 The two-axis opt-in¶

The engine is doubly opt-in: both the installation AND the surface must opt in.

Installation `metric_engine.enabled`	Surface `engine_registry`	Effect
False	None	Engine off (legacy direct-backend path)
False	Set	Engine off (installation didn't opt in)
True	None	Engine off (surface didn't materialise a registry)
True	Set	Engine on

This separation is deliberate: an installation can declare its intent (metric_engine.enabled: true) without forcing every deployment surface to pay the engine's wheel + storage cost. A deployment that wants the engine path explicitly opts in by constructing the EngineRegistry.

8. Cross-backend invariant validation¶

8.1 The W4 capstone claim¶

The W4 L1/L2/L3 operator registry abstraction (per Manual ch. 10 + v2.0 §3) makes a load-bearing claim: the same Frame-QL query, against the same AC, produces the same Frame regardless of which backend executes it. v2.0 + v2.1 articulated the claim; v2.2 validates it across three backends of meaningfully different shape.

8.2 The triangulation¶

Backend	Shape	Typical deployment
SQLite	Row-oriented, file-based, OLTP-shape	Demo + small ACs
Polars	In-memory, columnar, library-shaped (no SQL parser, no persistent store)	Embedded analytics, notebook workflows
DuckDB	Persistent, columnar, SQL-native, in-process	The canonical analytic-database archetype (Snowflake / BigQuery / Postgres-with-columnstore stand-in)

Three backends sitting at structurally-different points in the design space means the W4 abstraction is held against more force than a two-backend cohort could provide.

8.3 The test surface¶

Per Phase 9 slice 7 (packages/coframe-duckdb/tests/test_cross_backend_invariants.py):

Test category	Coverage
Query invariant	6 canonical retail queries × 3 backend pairs = 18 pairwise Frame comparisons, all with `rel_tol=1e-9` for numerics, string equality for the rest
FD-attestation invariant	5 known-good FDs (store_id→region, etc.) + 3 known-bad FDs (region→store_id, etc.) × 3 backends — all agree on `holds`, `distinct_x_count`, `violation_count`
Discovery invariant	`list_tables()` returns the same set; `describe_table().row_count` matches across backends for every retail table

The test parametrisation means a regression in any one backend gets pinpointed to a specific test ID — test_three_way_backend_invariant[limit_per_top3_stores_per_region] makes the failing combo obvious from pytest output alone.

8.4 What this proves¶

If three structurally-different backends produce identical Frames for every canonical query (within floating-point tolerance), then:

The DataAPIBackend protocol is genuinely backend-agnostic
The L2 operator registry pattern (POLARS_OPERATOR_REGISTRY / SQLITE_OPERATOR_REGISTRY / DUCKDB_OPERATOR_REGISTRY) successfully isolates dialect differences
The resolver + planner + execution layer make no assumptions a particular backend can't satisfy
Adding a fourth backend (Snowflake / BigQuery / Postgres / Databricks) is a known-effort task with predictable shape

Item 4 is the practical payoff: future backends inherit the cross-invariant test surface for free + the abstraction's correctness is a structural property of the design, not an empirical observation about the three engines that happened to be built.

9. Forward look (v2.3+)¶

What's not yet done that's been spotted but deferred:

9.1 AC-level derived metrics in Frame-QL¶

D8 (Demo Polish Tier 3) ships a programmatic example of computing profit = revenue - cost via engine.compose()'s frame_expression hook, but a Frame-QL author can't write SELECT region, profit AT region directly today. Reasons:

The IpReducer Pydantic model doesn't carry cross-family lineage
The resolver's _classify_select_item explicitly ignores composite expressions

A v2.3 design pass would add a derived-family declaration on MetricFamily (with a derived block carrying the formula + input families), teach the planner to treat derived families as schema-virtual (no Rule-3 / Rule-4 selection), and teach engine.serve() a new Branch 0 that recursively serves components and applies the AC-declared formula via compose()'s existing frame_expression hook. The design doc for this work — Reading B, where the AC + engine own the derivation contract and the planner stays unaware — lives at drafts/coframe_derived_metrics_design_v0_1.md. ~300-500 LOC + tests.

9.2 NLQ surface¶

Natural-language → Frame-QL. The position article + manual both reference NLQ as a first-class consumer surface but the implementation hasn't landed. v2.3 design questions:

LLM dialogue protocol for clarification rounds (the "is this what you meant?" cycle)
Integration with Frame-QL's structural refusal — NLQ can use the refusal explanations to drive clarification
Where the LLM call boundary lives (coframe-dialogue package per v2.0 §9? Or in a runtime extension?)

9.3 MCP surface¶

Model Context Protocol server so AI agents can query an AC via MCP rather than HTTP. The coframe-mcp package was deleted in the v1 cleanup; v2.3 would re-add it as a thin MCP wrapper around the runtime HTTP app.

9.4 Coframe Pro tier work¶

The Phase 8 design doc explicitly deferred several capabilities to Coframe Pro:

Cross-process engine coordination (§3.3): multi-worker uvicorn deployments sharing one engine instance
Backend batching (§7.2): coalesce multiple single-metric calls into multi-metric AggregateRequests when the engine has multiple misses against the same backend in flight
Cross-AC engine sharing (§15.5): two ACs sharing a physical-name cache when their filters agree
Sketch operators (§5.3 in DUCKDB_OPERATOR_REGISTRY): HLL_MERGE / THETA_UNION / T_DIGEST_MERGE bindings

None of these are blocking for Core; they're optimisations the Pro tier offers.

9.5 Frame's served_from field — promote to typed enum?¶

D4 made Frame.served_from an optional str | None field, with four documented values. v2.3 could promote it to a Literal["engine_cache", "engine_mixed", "engine_backend", "backend"] | None or even an enum for stronger typing.

10. Amendments tracker¶

Date	Change	Section
2026-05-25	Initial v2.2 supplement landing — covers Phase 8 + Phase 9 + Demo Polish through commit `014b655`	All

This section is the place future v2.2.x amendments register their edits, mirroring v2.1's §10. Subsequent supplements (v2.3+) would land as new sibling files (coframe_platform_design_v2_3_supplement.md) per the v2.0 → v2.1 → v2.2 pattern.

11. Summary — what the platform looks like as of v2.2¶

A reader coming to the platform fresh in this state sees:

Twelve packages (eleven Python + one TS/React), with three execution backends (SQLite + Polars + DuckDB) sharing a single DataAPIBackend protocol that's been triangulated across structurally-different engines
Per-AC Metric Engine as an optional but production-shaped acceleration layer, with cache_hint declarations from AC authors, FD-DAG-aware rollup, and promotion-recommendation lifecycle that closes the loop back to AC authoring
Workbench + Runtime HTTP surfaces that both opt into the engine via parallel EngineRegistry infrastructure; deployment can use either or both depending on the topology
Frontend with Workbench (declare, attest, verify, engine) + Management + Query UIs, all reading from the same session-scoped AC
Three-tier Phase plan complete through Phase 9, with v2.3+ work (NLQ, MCP, AC-level derived metrics, Coframe Pro tier) explicitly named

The canonical retail demo exercises every visible capability end-to-end in ~7 documented steps (per docs/retail_demo_walkthrough.md); the demo bootstrap takes ~47 minutes to author from CSVs forward; new-customer time-to-first-query is seconds.