Model RemoteLlm

ml/remote/llm::RemoteLlm

Parameters

↳ const api_key: Option<string>
↳ const backend: string = "mistral"
↳ const base_url: string = ""
↳ const max_tokens: Option<u64>
↳ const model: string = ""
↳ const system: string = ""
↳ const temperature: Option<f32>
↳ const timeout: Option<u64>
↳ const top_p: Option<f32>

Remote LLM provider configuration.

Holds connection and inference parameters for a remote large language model service.

• backend: provider name (default "mistral"); see the table below.
• api_key: API key for authentication (omit for Ollama or unauthenticated endpoints).
• base_url: override the provider base URL — required for Ollama, Azure OpenAI, or custom OpenAI-compatible endpoints; see the table below.
• model: model identifier — see the table below for recommended values per backend.
• system: system prompt injected at the start of every conversation.
• max_tokens: maximum tokens to generate per response (omit to use the backend default).
• temperature: sampling temperature 0.0–2.0 (omit to use the backend default; some backends reject temperature and top_p being set simultaneously).
• top_p: nucleus sampling cutoff 0.0–1.0 (omit to use the backend default).
• timeout: request timeout in seconds (omit to use the backend default).

Backends

ℹ️ Use RemoteLlm together with chat, stream, or visionChat.

use ml/remote/llm::RemoteLlm
use ml/remote/llm::stream
use std/engine/util::startup

treatment example()
  model llm: RemoteLlm(
    backend    = "mistral",
    api_key    = "...",
    model      = "mistral-small-latest",
    system     = "You are a helpful assistant.",
    max_tokens = 2048
  )
  input  prompt: Stream<string>
  output token:  Stream<string>
{
    stream[llm=llm]()
    Self.prompt -> stream.prompt,token -> Self.token
}