How to score answers from a local model (Ollama, llama.cpp)

Get the answer from your local model, then send the text to ClearRun with clearrun.evaluate(). The model runs on your machine, but scoring runs on api.clearrun.net, so the answer text is sent there.

Last updated

Steps

  1. Ollama

    import { createKernelClient } from '@clearrun/sdk';
const clearrun = createKernelClient(); // reads CLEARRUN_API_KEY

const prompt = 'When was the Eiffel Tower finished?';
const res = await fetch('http://localhost:11434/api/generate', {
  method: 'POST',
  body: JSON.stringify({ model: process.env.OLLAMA_MODEL, prompt, stream: false }),
});
const { response: answer } = await res.json();

const { verdict } = await clearrun.evaluate({ prompt, responseText: answer });

  2. llama.cpp server

    const res = await fetch('http://localhost:8080/v1/chat/completions', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ messages: [{ role: 'user', content: prompt }] }),
});
const answer = (await res.json()).choices[0].message.content;

    Then call clearrun.evaluate the same way.

Fully offline?

Scoring can't run offline. For offline tests, use SDK mock mode. Mock scores are local test values, not real evaluations. See How to use ClearRun mock mode for local development (forceMock).