# Real local inference for staging

`createLocalModelBackend` runs fresh inference using the existing pinned Qwen3-1.7B Q4_K_M file. It is separate from the deterministic preview provider and never substitutes fixture text when loading or generation fails. There is no network inference, model download, ICP update call or token transfer in this backend.

The native CPU result is not evidence of ICP execution or output parity with the canister's chunked sampler. The staging payment orchestration must record this distinction in its result receipts.

## Pinned inputs

| Input | Pin |
| --- | --- |
| Model | `unsloth/Qwen3-1.7B-GGUF` / `Qwen3-1.7B-Q4_K_M.gguf` |
| Upstream model revision | `d7f544eead698dbd1f15126ef60b45a1e1933222` |
| Model bytes | `1107409472` |
| Model SHA-256 | `b139949c5bd74937ad8ed8c8cf3d9ffb1e99c866c823204dc42c0d91fa181897` |
| Python binding | `llama-cpp-python==0.3.36` |
| Local execution | CPU, at most four threads, zero GPU layers, 4,096-token context |
| Generation | At most 96 new tokens; Qwen3 no-thinking framing; request-ID-derived seed |

The model defaults to the sibling inference project's `models/Qwen3-1.7B-Q4_K_M.gguf`. Each Python invocation checks its complete SHA-256 and file size before importing the runtime or loading the model. The runtime version is checked as well. Copying a different model to that filename will fail.

## Isolated installation

Use a separate `.venv-staging` under Agent Kit. No existing inference environment needs to change. The maintainer's documented [installation options](https://github.com/abetlen/llama-cpp-python) include official CPU and Metal wheels.

The official `0.3.36` macOS arm64 Metal wheel was downloaded twice during setup, including an uncached retry. Both attempts failed the ZIP CRC check for `lib/cmake/ggml/ggml-config.cmake`. Its integrity check was not bypassed. A CPU-only source build is the selected installation route for this staging environment:

```sh
python3 -m venv .venv-staging
CMAKE_BUILD_PARALLEL_LEVEL=4 \
CMAKE_ARGS='-DGGML_METAL=OFF -DGGML_NATIVE=OFF -DLLAMA_CURL=OFF' \
  .venv-staging/bin/python -m pip install \
  --no-binary=llama-cpp-python llama-cpp-python==0.3.36
```

This compiles the pinned package and requires a working local C++ toolchain. The package installer supplies its isolated build dependencies. It does not fetch model weights. Set the absolute executable path in the staging configuration; there is no automatic fallback to a system Python.

The isolated CPU source build completed on 28 October 2026. Its locally built wheel SHA-256 was `15898d5b324f429d9b95600ba61e15d942e663db681a71e05f27df4bf5aa69f6`. Installed dependencies were NumPy `2.5.4`, diskcache `5.6.3`, Jinja2 `3.1.6`, MarkupSafe `3.0.4` and typing-extensions `4.16.0`.

One fresh standalone probe passed the backend's hash/version preflight and generated `Welcome to GH0ST.` from a request for a short developer welcome. It consumed 64 prompt tokens and 6 generated tokens, ended with `finishReason: "stop"`, and reported 6,777 ms including local hash checking and loading. This single local observation is not a throughput benchmark, an ICP result or a quality evaluation.

## Integration

```js
import { createLocalModelBackend } from '../src/local-model.mjs';

const backend = createLocalModelBackend({
  pythonPath, // absolute path to this kit's .venv-staging/bin/python
  modelPath,  // optional absolute path; defaults to the pinned sibling model
  threads: 4,
  timeoutMs: 120000,
});

const identity = await backend.preflight();
const result = await backend.generate({
  requestId,
  prompt: 'Write a short welcome for a new community member. Do not invent facts.',
  maxTokens: 48,
  signal,
});
```

`result` contains the newly generated `answer`, actual `inputTokens` and `outputTokens`, `modelSha256`, `runtimeVersion`, `elapsedMs`, `finishReason` and `backend: 'local-llama-cpp'`. A `length` finish reason means the token budget was reached; callers should decide whether that output satisfies their completion policy before settlement.

The Node wrapper passes a bounded JSON request through stdin. Python reserves stdout for one JSON result and routes Python/native diagnostics to stderr. The wrapper checks result provenance and usage bounds, limits captured output and terminates the subprocess on abort or timeout. It starts a fresh model process for each job; this favors isolation over throughput.

The model output is untrusted text. The backend does not execute it, publish it, or certify that the answer is correct. Payment, verified source admission, duplicate protection and settlement belong to the surrounding staging workflow.