Q1. What are the key dimensions for evaluating prompt quality?

Key dimensions include: • Accuracy: Does the output correctly answer the query? • Completeness: Does it include all required information? • Format compliance: Does it follow the specified structure (JSON, bullet points, etc.)? • Consistency: Same input yields similar output across runs. • Conciseness: No unnecessary verbosity or repetition. • Tone and style alignment: Matches desired persona/voice. • Robustness: Works across edge cases and variations. • Efficiency: Token usage (cost) and latency. • Safety: No harmful or off-topic content. • Explainability: Can the model's reasoning be understood? Different tasks may prioritize different dimensions.

Q2. How can you measure prompt quality automatically?

Automated metrics include: • Exact match or substring match for expected answers. • JSON schema validation (using libraries like `jsonschema`). • Regular expression checks for format (e.g., date format, phone number). • BLEU/ROUGE scores for summarization (compare to reference summaries). • Using another LLM as a judge (e.g., prompt: "Rate the following answer from 1 to 5 on accuracy and helpfulness"). • Unit tests: predefined input-output pairs that must pass. • Token count (cost) and response time (latency) measurement. • Classification metrics (precision, recall, F1) for discrete outputs. • Custom scoring functions (e.g., verify that extracted JSON fields are non‑empty).

Q3. What is an LLM‑as‑a‑judge evaluation? Give an example.

LLM-as-a-judge uses a separate LLM call to evaluate the quality of another LLM's output. Example: For a summarization task, you ask a stronger model (e.g., GPT-4) to rate summaries produced by a weaker model on criteria like factual accuracy, conciseness, and completeness. The judge prompt: "You are an evaluator. Rate the following summary from 1 to 5. Original text: {text}. Summary: {summary}. Rate on accuracy (1–5) and clarity (1–5). Provide a brief explanation." This method correlates well with human judgments and is scalable. However, it introduces additional cost and potential bias; calibrating the judge prompt is important.

Q4. How do you evaluate prompt quality when there is no ground truth (e.g., creative writing)?

For subjective tasks, use: • Human evaluation (crowdsourcing or expert raters) with rubrics (e.g., creativity, coherence, engagement). • Pairwise comparison: A/B test two prompts and ask humans to prefer one. • LLM-as-a-judge with detailed criteria (e.g., "Does the story have a clear beginning, middle, and end?"). • Proxy metrics: e.g., length, vocabulary diversity, sentiment alignment. • User feedback: click-through rates, thumbs up/down in production. • Adherence to constraints (e.g., must contain certain keywords). Since no single correct answer, evaluation becomes multi‑dimensional and often requires mixed methods.

Q5. What is the role of a validation set in prompt evaluation?

A validation set is a collection of representative input-output examples that are used to measure prompt performance during development. It helps: • Avoid overfitting to the test set. • Compare different prompt versions objectively. • Identify which types of inputs the prompt struggles with. • Tune hyperparameters (temperature, etc.). The validation set should be separate from the training set (if any) and final test set. It typically contains 50–500 examples. When you iterate on a prompt, you evaluate on the validation set; only after finalizing do you evaluate on the holdout test set to estimate real-world performance. This prevents optimistic bias.