How to fail a build when a score is too low (--threshold)

Add --threshold to clearrun score. If any answer scores below that number, the command exits with code 1, and your CI step fails.

Last updated

Steps

  1. Pick a number from 0 to 100

    ClearRun does not choose one for you. Score a few answers you already trust and a few you don't, and set the line between them.

  2. Run the score with the threshold

    clearrun score responses/ --threshold 70

  3. Check the exit code

    echo $?

    It prints 1 after a failed run and 0 when every input passed.

What counts as a fail

The run exits 1 if any input:

  • scores below the threshold,
  • comes back with no score (only when a threshold is set),
  • has more issues than maxIssues, if you set it,
  • or hits an error, such as a missing key or a rate limit.

Otherwise it exits 0. There are no other exit codes.

Save the rule in a config file

{
  "paths": ["responses/"],
  "threshold": 70,
  "maxIssues": 3
}

Save this as clearrun.config.json, then run clearrun score with no file. --threshold on the command line overrides the file.

What the threshold does not do

It does not block or change the answer. It only sets the exit code. A pass does not mean the answers are true.