Skip to the content.

Inference Runtime Guide

core/inference is the contract for model calls, and four packages around it are the runtime: core/inference/model (what a model is), core/inference/outcome (what one call returned: its identity and its accounting), core/inference/route (selection, retry, fallback), and core/runtime (assembly from a deployment document). core/inference-openai and core/inference-anthropic are the two protocol families that ship with the platform; every other vendor is a driver/ module.

type Provider struct {
    Generator            // required: Generate and GenerateStream
    Embedder  Embedder   // nil when the provider does not serve embeddings
    Decider   Decider    // nil when the provider does not serve judgments
    Catalog   Catalog    // nil when the deployment declared no models
}

Provider.Validate() is what turns "declared but uncallable" into a deployment error: a model that declares an embedding width while no Embedder is bound, or decision support while no Decider is bound, would be a capability a caller can discover and never use.

Transcription and image generation are not seams here — they are extensions the provider driver offers (core/inference-openai has both). A capability that only one vendor sells stays with the vendor.

Exact addressing

Every call names a concrete model:

ref := model.ModelRef{
    ID:      model.ModelID{Provider: "openai", Name: "gpt-5"},
    Profile: "luna",        // the deployment's provider block
}

The runtime never picks or replaces a model on its own. Routing is optional and explicit: it is a policy over the declared line-up, and an empty route.selector means declaration order.

The document

inference:
  providers:
    - name: luna # deployment identity, secret-free
      driver: openai # a name the host compiled in
      settings: # the driver's own written form (see below)
        base_url: https://api.openai.com/v1
        api_key: "${OPENAI_API_KEY}"
        api: responses
  models:
    - provider: luna
      model: gpt-5
      weight: 100
      # Fill in only what the driver's catalog does not declare; a
      # contradiction is an assembly error, not an override.
      capabilities: { inputs: [text, image], outputs: [text] }
      limits: { max_input_tokens: 200000, max_output_tokens: 32000 }
  route:
    selector: weighted # a name the host registered
    attempts_per_target: 2
    max_targets: 3
    base_delay: 200ms
    max_delay: 2s
Block Rule
providers[].name The deployment's identity for the account or endpoint; it is what models[].provider refers to.
providers[].driver Must be registered on the Runtime; an unknown one fails with the offered names.
providers[].settings Opaque to core: the driver's constructor decodes it.
models[] Position is priority — the first model is tried first. Declaring a model is the only way one exists.
route Optional. attempts_per_target, max_targets, base_delay, max_delay, selector, selector_settings.

Drivers and their settings

The runtime imports no driver, so the adapter that maps a written settings block onto a driver's typed Settings lives in the host:

runtime.WithDrivers(
    runtime.Driver{Name: openai.ProviderID, Build: buildOpenAI},
    runtime.Driver{Name: anthropic.ProviderID, Build: buildAnthropic},
)

Every driver's Settings is a plain Go value without JSON tags — a JSON block is not its wire format, and pretending otherwise would freeze the contract. The host writes the mapping, which means a deployment's settings keys are the host's vocabulary: examples/forge/internal/app/drivers.go is the reference implementation, including duration parsing and the wire sub-block the Anthropic driver takes.

That indirection is deliberate. It lets a driver rename a field without a breaking change to every deployment file, and it keeps a credential's spelling (api_key) a host decision rather than a vendor one.

Model declarations

A model exists because a deployment declares it. Nothing is inherited from a same-named model, so a generate model has to state its text output, and every published capability is written out.

Two sources meet here:

A contradiction between them is an assembly error, not an override. What the document adds and the catalog omits becomes the published descriptor (model.ModelDescriptor: id, label, capabilities, limits, lifecycle).

Limits are promises, not clamps

A Generate request whose max_output_tokens exceeds the target's declared maximum is rejected before the driver is opened, and the rejection is transport-safe — so a routed request falls back to a target that declares room for it instead of failing. The caller's budget is never silently lowered, and an undeclared limit rejects nothing.

max_input_tokens is not enforced: the module has no tokenizer, and an approximate count would be worse than none. A host that owns the conversation history reads the declared window from the catalog and trims where it already trims.

Capabilities route the call

Declared capabilities are consulted on both the selection and the fallback path: a target whose declared output kinds cannot serve the request intent is skipped, while a model with undeclared capabilities is treated as undeclared — not unsupported — and the request is attempted. The provider's own validation remains the final arbiter.

A capability declaration is also a promise checked per surface: a model declaring toggleable reasoning must accept reasoning_enabled=false on that surface, and one declaring always must reject it. Discovery bits such as hosted web search ride on the descriptor, so a host can surface per-model options without vendor-specific knowledge.

Routing

Selection, retry and fallback live in core/inference/route:

router, err := route.New(targets, route.Policy{
    Generate: mySelector{},          // nil = every target, in declaration order
    Retry: route.RetryPolicy{
        AttemptsPerTarget: 2,
        MaxTargets:        3,
        BaseDelay:         200 * time.Millisecond,   // doubles per attempt
        MaxDelay:          2 * time.Second,
    },
})

A route.Target is one candidate: its provider, weight (for selectors that use one), the model it serves, and the deployment profile it belongs to. Dispatch stamps the winning target's model identity onto the request, so a caller names a model and the answer carries the provider that actually served it.

Field Meaning
Generate, Decision One selector per seam; a nil selector means every target in order.
Retry.AttemptsPerTarget Tries of one target for a transient failure; zero means one.
Retry.MaxTargets How many candidates are tried; zero means all the selector returned.
Retry.BaseDelay, Retry.MaxDelay Backoff, doubling per attempt, capped.

A selector is a Go value, and runtime.Selector is only the name that lets a document refer to one — two typed slots, because route.Selector[inference.GenerateRequest] and route.Selector[inference.DecisionRequest] are different Go types and one factory cannot produce both:

runtime.WithSelectors(runtime.Selector{
    Name:     "weighted",
    Generate: func(json.RawMessage) (route.Selector[inference.GenerateRequest], error) {
        return weighted{}, nil
    },
})

There are no built-in selectors, and an unregistered name fails the build with the offered list. A selector has one method — Select(ctx, request, targets) — because a selector answering for several seams would be the operation table this package exists to avoid. Pure Go callers skip the name table entirely and put the value in route.Policy directly.

Extensions

Provider-specific options travel as typed extensions, not as a raw map the driver merges blindly:

request:
  extensions:
    - { name: reasoning, settings: { effort: high } }

An extension name is resolved against the ExtensionDecoders the host registered (nodes.WithModelDecoders, runtime options), so unknown settings are a build error rather than a silently dropped field, and the driver receives a typed value. inference.Extensions also carries Metadata, which is passed through untouched and is the place for a trace tag rather than a wire field.

Streaming

GenerateStream returns a Stream the caller must close. The stream carries incremental events — text deltas, reasoning deltas, tool-call fragments, usage — and one terminal event; a consumer that stops early must close it, or the provider connection leaks. A node that does not need deltas calls Generate instead: a call whose answer is data should not pay for an event stream.

Media

Media is a first-class part kind (see media.md). A request's parts and the model's declared inputs/outputs are checked against each other during routing, which is what keeps "this turn has an image, this target has no vision" from reaching a provider. Audio and video parts follow the same rule; a provider that does not accept a kind refuses the request, and flattening it into text is the host's decision, not the runtime's.

Judgment

Decider.Decide answers a set of questions in request order and returns one answer per question:

Type Is
Question One question with a QuestionKind: a choice among labelled options, a probability, a free answer.
ChoiceValue, Probability, Answer The shapes an answer takes.
DecisionRequest, DecisionResponse The batch and its results, with usage and the same error vocabulary as generation.

Routing applies to judgment too — runtime.Selector.Decision is the other typed slot — because "which model judges this" is the same question as "which model answers this".

Errors

Every seam shares one vocabulary — an inference.ErrorKind plus a Field naming the request member at fault: InvalidRequest, UnsupportedOperation, UnsupportedFeature, InvalidExtension, UnknownModel, PolicyDenied, ProviderFailure, InvalidProviderResponse, UndefinedTool, …

A routing decision classifies from that vocabulary, not from a string, and it has three outcomes rather than two:

Outcome Kinds Meaning
retry the same target ProviderFailure, ProviderTruncated Transient, or a target worth one more attempt.
move to the next target UnknownModel, UnsupportedOperation, UnsupportedFeature, and anything unclassified This target cannot serve the request; another may.
stop InvalidRequest, InvalidExtension, PolicyDenied, UndefinedTool, CompilerContractViolation, OperationInterrupted, and a cancelled context Repeating it anywhere produces the same answer.

That is what makes a fallback safe: a wrong request is not retried three times against three vendors, and a cancelled call is not retried at all.

What this is not

Not here Where
A provider registry with init() registration The host's runtime.WithDrivers table: "which drivers exist" is a fact of the binary.
A vendor's model line-up The deployment declares it; drivers ship none.
Transcription, image generation Driver extensions (core/inference-openai).
Realtime Reserved; no seam yet.
Token counting No tokenizer in core.