Q1. What is logit bias? How does it work?
Logit bias is a parameter that allows you to directly adjust the logits (pre-softmax scores) of specific tokens before sampling. By increasing or decreasing the logit of a token, you make it more or less likely to be generated. It is a powerful way to enforce constraints (e.g., never output a certain word) or guide the model toward desired outputs. In OpenAI's API, you provide a dictionary mapping token IDs to bias values between -100 and 100. Positive values increase probability, negative values decrease. Extreme values like -100 effectively ban the token (though not guaranteed), while +100 makes it almost certain.
Q2. Give examples of using logit bias to constrain model output.
Example 1: Prevent generation of a specific word. Suppose you want to avoid the word "violence". Find its token ID(s) and set bias to -100. Example 2: Force a specific output format. For generating JSON, you could bias tokens like '{' and '}' positively. Example 3: Encourage the model to output only numeric digits for a math answer. Bias digits (0-9) positively and bias letters negatively. Example 4: In a multiple-choice QA, bias the letters A, B, C, D positively to ensure the model chooses one. Example 5: Bias a specific token for brand name (e.g., "Quipoin") to increase its likelihood in product descriptions.
# OpenAI API example
logit_bias = {token_id: bias}
response = openai.Completion.create(..., logit_bias=logit_bias)Q3. How do you find token IDs for logit bias?
Token IDs depend on the tokenizer used by the model (e.g., GPT-2/GPT-4 use Byte-Pair Encoding). Methods to find token IDs: • Use the model's tokenizer library:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base") # for GPT-4
ids = enc.encode("violence")Q4. What are the limitations and risks of using logit bias?
Limitations include: • Not a guarantee: extreme bias like -100 makes a token very unlikely but not impossible (especially with high temperature). • Complexity: need to know token IDs and handle multi‑token words. • Can interfere with natural language fluency; over-biasing may produce unnatural output. • May cause the model to fail to follow other instructions. • Works per generation step; the model may still produce the banned token in earlier steps via different tokenization (e.g., 'viol' + 'ence' with spaces). • Only applicable to certain APIs (OpenAI, Cohere). • Using logit bias carelessly can degrade performance. • Best for simple constraints like numeric output or ensuring a particular format; for complex constraints, use post‑processing instead.
Q5. Compare logit bias with other parameters like frequency penalty and stop sequences.
Logit bias: per‑token, pre‑generation adjustment; static (does not change during generation). Frequency penalty: dynamic based on token repetition. Stop sequences: end generation based on string match. Use cases: • Logit bias is for encouraging/banning specific tokens (e.g., force A/B/C). • Frequency penalty is for diversity and reducing repetition of any tokens. • Stop sequences are for controlling output length/termination. They are complementary. Example: To generate a comma-separated list of 5 numbers without repetition, you might use frequency_penalty=0.5 to avoid repeats, set stop sequence = "n", and use logit bias to favor digits over letters. Logit bias gives fine‑grained control but is more complex; frequency penalties are easier for general repetition.