Large language models must sometimes refuse to answer. A user might ask the LLM for harmful instructions, pose a question based on a false premise, or request information the model simply does not have. Getting this right is hard. The standard solution is to retrain the model with different amounts of refusal data until the refusal rate feels right. This is slow, expensive, and imprecise. Jain et al. (2024) propose a much simpler idea called refusal tokens.

During training, a special [refuse] token is placed at the start of every refusal response, and a [respond] token at the start of every normal response. The model learns to predict the correct token before generating the rest of its answer. At inference time, the probability the model assigns to the [refuse] token tells us how likely it thinks a refusal is needed. By setting a threshold on this probability, a developer can make the model refuse more or less often, all without touching the training pipeline again.

The idea extends naturally to different types of refusal. Instead of one refusal token, the model can be trained with several, one for safety concerns, one for questions beyond the model’s knowledge cutoff, one for ill-posed queries, and so on. Each token can be thresholded independently at test time, meaning a developer can tighten refusals on safety questions while leaving other categories unchanged. Experiments on the CoCoNot benchmark (Brahman et al., 2024) confirm this category-level control works with minimal interference between refusal types. The tokens also improve refusal F1 scores during training itself, even before any thresholding is applied, rising from 0.918 to 0.940 on CoCoNot.

The practical appeal is clear. Refusal tokens add practically nothing to the training process, require no external classifiers, and give, us, developers a direct, interpretable way to tune refusal behavior after deployment. As safety standards and use cases continue to shift, having a lightweight mechanism like this is a meaningful advantage over retraining from scratch every time the requirements change. This technique can be, and is, applied across LLM’s and allows for increased suitable answers.

Sources