Inference API

myco-model provides backend-independent messages and streaming drivers for Anthropic Messages, OpenAI Responses, and OpenAI Chat Completions. It has no dependency on Myco's server, profile configuration, tool runtime, or session store.

Start with GenerativeModel, GenerativeModelConfig, and Message.

Construct a driver

myco_model::new takes a ModelSpec, matching BackendConfig, system prompt, and tool catalog. ModelSpec::key is your application's name for a model; api_id is what the endpoint receives. The backend variant must match the spec's protocol. Credentials and endpoint settings are supplied directly; the library does not read ~/.myco or load .env for you.

This complete example sends one prompt to a Chat Completions compatible endpoint, streams text, and validates the completed response:

use std::io::{self, Write};

use futures::StreamExt;
use myco_model::{
    BackendConfig, Content, ContentDelta, GenerativeModelConfig, Message, MessageAccumulator,
    MessagePart, ModelSpec, OpenAIBackendConfig, Protocol, ThinkingMode,
};

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let model = myco_model::new(GenerativeModelConfig {
        model: ModelSpec {
            key: "example".into(),
            api_id: std::env::var("MYCO_EXAMPLE_MODEL")?,
            protocol: Protocol::OpenAICompletions,
            thinking: ThinkingMode::None,
            context_window_tokens: 32768,
            max_image_base64_bytes: 5 * 1024 * 1024,
            max_truncated_resumes: 0,
            auto_compact_at_tokens: None,
        },
        tools: vec![],
        system_prompt: "Give a brief, helpful answer.".into(),
        backend_config: BackendConfig::OpenAICompletions(OpenAIBackendConfig {
            base_url: std::env::var("MYCO_EXAMPLE_BASE_URL")?,
            auth_token: std::env::var("MYCO_EXAMPLE_API_KEY").unwrap_or_default(),
            effort: None,
            max_output_tokens: Some(512),
            ..Default::default()
        }),
    })?;
    let history = vec![Message::UserMessage {
        content: vec![Content::Text {
            text: "What makes a good library interface?".into(),
        }],
    }];

    let mut stream = model.generate(&history);
    let mut response = MessageAccumulator::default();
    while let Some(event) = stream.next().await {
        // This example performs one attempt; errors propagate to the caller.
        let part = event.into_result()?;
        response.push(&part)?;
        if let MessagePart::ContentDelta(ContentDelta::Text { delta, .. }) = part {
            print!("{delta}");
            io::stdout().flush()?;
        }
    }
    let output = response.finish()?;
    println!();
    eprintln!(
        "stop: {:?}; usage: {:?}",
        output.turn_end_reason, output.usage
    );
    Ok(())
}

Run it from the repository with a server and model you have configured:

MYCO_EXAMPLE_BASE_URL=http://localhost:11434/v1 \
MYCO_EXAMPLE_MODEL=YOUR_SERVED_MODEL_ID \
cargo run --locked -p myco-model --example inference

For endpoints requiring authentication, set MYCO_EXAMPLE_API_KEY in the environment. This example makes a real model request. Its model window and output budget are illustrative; adjust them for your endpoint.

Consume the stream

Each generate(&history) call is one attempt. Its stream yields GenerationEvent::Part(MessagePart) or terminates with a GenerationEvent::Failure(GenerationFailure). Dropping the stream cancels the in-flight request.

MessagePart carries message start, content starts/deltas, tool starts/JSON deltas, token usage, and a stop reason. Feed all parts to MessageAccumulator, then call finish() to validate and obtain GenerateOutput. Text shown during streaming is provisional until the whole attempt succeeds.

When incremental output is unnecessary, GenerateOutput::from_generation(model.generate(&history)).await performs that accumulation for you. It returns the error cause but discards retry metadata; inspect GenerationEvent::Failure yourself if you need that metadata.

Own policy above the driver

Drivers do not retry. A failure's retryable flag and optional retry_after are inputs to caller policy. Retry only if no response parts were emitted, with an attempt limit and bounded delay. myco-agent implements that policy; passing retry settings to a backend alone does not create a retry loop.

GenerateError::recovery() distinguishes ordinary retry eligibility at the history level from Recovery::OmitLastMessage, used when input must shrink. It is not a promise that retrying will succeed or an instruction to retry all errors indefinitely.

Preserve message structure

Messages contain user input, assistant output with optional tool calls, or tool results. Results pair positionally: result j answers call j of the immediately preceding assistant message. Keep the ordering and counts when storing, slicing, or replaying history. Drivers generate protocol-specific wire IDs; callers do not need to store provider call IDs.

A tool specification advertises a name, description, and JSON input schema. The model crate never executes the tool. Use the agent crate, or implement your own dispatch loop that preserves the message contract.

Content supports text, images, and thinking blocks. answer_content selects text and image blocks for an answer. For images, provide a URL or a typed data URL; raw base64 is treated as PNG. Token usage may be absent. Cached input tokens are a subset of input tokens, not an extra quantity to add.