Enable user-defined steering for a checkpoint

note

User-defined steering concepts are available since feature set 250600.

Determine a good steering strength

First, you need to figure out a good steering strength for your checkpoint. This is usually done by evaluating a set of completions balancing the impact of the steering examples with the correctness of the output.

At the moment, the steering strength can be set only in the worker configuration and changing it requires restarting the worker. These limitations might get lifted in a future release.

You can refer to this table for a good starting point:

Checkpoint	Strength
llama-3.1-8b-instruct	0.062

Setting the default steering strength for your worker

If you deployed llama-3.1-8b-instruct with our default configuration, steering will already be enabled. In case you're using a custom worker deployment, you need to overwrite inference-worker.checkpoints in values.yaml and set a default steering strength to enable steering.

inference-worker:
  checkpoints:
    ...
    - generator:
        type: "luminous"
        pipeline_parallel_size: 1
        tensor_parallel_size: 1
        tokenizer_path: "llama-3.1-8b-instruct/tokenizer.json"
        weight_set_directories: ["llama-3.1-8b-instruct"]
        steering:
          default_in_context_vectors_strength: 0.062
      queue: "llama-3.1-8b-instruct"
      replicas: 1
      modelVolumeClaim: "models-llama-3.1-8b-instruct"
      version: 0
      models:
        llama-3.1-8b-instruct:
          ...

The Helm chart must now be redeployed again for the changes to take effect.

Creating and using a steering concept

This document describes how to set up a checkpoint for steering. Creating and using steering concepts in completion and chat requests is described on the Steering page.