AI Systems Lab

An experimental AI project

AI Systems Lab makes provider abstraction, model routing, lifecycle management, compatibility checks, and repeatable evaluation concrete enough to study. Local llama.cpp remains a first-class backend and default workflow, while cloud-hosted OpenAI-compatible providers participate in the same chat and evaluation paths through shared model interfaces. Cloud provider credentials are opt-in environment variables and never committed configuration.

What the lab explores

The lab uses workload-specific model aliases behind a provider-neutral request path. Provider-specific request construction handles local runtime lifecycle or cloud API transport without coupling routing and evaluation workflows to one vendor. A curated catalog records each model’s provider, capabilities, intended role, and lifecycle so candidates can be tested before promotion.

Where local models are useful

Local models can be useful for bounded experiments, small targeted changes, fixes, repository exploration, and other work with a clear acceptance contract. These are candidate workloads to evaluate, not a promise that every model will complete every task successfully.

Where they fall short

Local models fall short for higher-stakes, ambiguous, complex, large-context, or reliability-critical work when their results do not meet the task's evidence bar. A fast answer, a plausible answer, or a larger local model does not establish that the work is safe or correct.

When to escalate

Flagship frontier cloud models are the escalation path when risk or complexity exceeds demonstrated local capability. Missed acceptance criteria, malformed tool calls, repeated human repair, insufficient context, and observed quality gaps are signals to escalate rather than stretch a local model beyond the evidence.

Compatibility before promotion

Each candidate must return a visible completion and produce non-streaming structured tool calls and streaming tool-call deltas through its OpenAI-compatible aliases. Local runtimes must also preserve the configured residency rule and unload cleanly; cloud-provider requests must satisfy the same observable completion and tool-call contract. Promotion also requires real tool loops in supported coding-agent clients; an answer that merely looks correct is not proof that a tool ran.

Layered evaluation

The lab separates raw runtime throughput, the end-to-end request path, and task quality. That keeps speed and memory measurements subordinate to whether the result meets automated acceptance criteria and survives human review.

./scripts/lab bench-llama coder --all-candidates
./scripts/lab bench-server coder --all-candidates --unload-after
./scripts/lab bench-quality coder --all-candidates

• bench-llama isolates model and runtime throughput.
• bench-server measures the provider-neutral request path and request latency.
• bench-quality records answers for correctness checks and manual review by default.

Evidence boundaries

Public claims stay at the level of reproducible interfaces and methods. Raw model answers, benchmark artifacts, local paths, host details, private repository metadata, credentials, and machine-specific configuration remain unpublished. Results are not generalized into model parity, superiority, guaranteed privacy, or suitability for untested workloads.

What this demonstrates

• Designing provider-neutral routing around stable workload aliases while isolating provider-specific request construction and local lifecycle state.
• Testing client compatibility with observable completions and real structured tool calls.
• Separating performance, end-to-end behavior, and task quality while making escalation part of the evaluation method.