Treatment stream

ml/remote/llm::stream

Configuration

⬡ llm: ml/remote/llm::RemoteLlm

Inputs

⇥ prompt: Stream<string>

Outputs

↦ error: Stream<string>
↦ failed: Stream<void>
↦ token: Stream<string>

Stream token-by-token output from a remote LLM.

For each string received on prompt, sends a streaming chat request and emits decoded token strings on token as they arrive. Generation for one prompt ends when the provider closes the stream; the next prompt is then processed.

If the provider does not support streaming, or if an error occurs, failed and error are emitted.

graph LR
     T("stream()")
     P["🟩 🟩 …"] -->|prompt| T
     T -->|token|  K["🟩 🟩 🟩 🟩 …"]
     T -->|failed| F["🟩 🟩 …"]
     T -->|error|  E["🟩 🟩 …"]

     style P fill:#ffff,stroke:#ffff
     style K fill:#ffff,stroke:#ffff
     style F fill:#ffff,stroke:#ffff
     style E fill:#ffff,stroke:#ffff

use ml/remote/llm::RemoteLlm
use ml/remote/llm::stream

treatment example()
  model llm: RemoteLlm(backend = "mistral", api_key = "...", model = "mistral-small-latest")
  input  prompt: Stream<string>
  output token:  Stream<string>
{
    stream[llm=llm]()
    Self.prompt -> stream.prompt,token -> Self.token
}