Defining what good means for your task

Before you can measure quality you have to define it, and generic definitions are useless. Good for a customer support reply, good for a SQL generator, and good for a document summarizer are three different things. A human has to decide what matters for this specific task; the model cannot decide it for you.

Break the definition into named criteria you can check one at a time. For a support answer that might be: grounded in the provided policy, correct, complete, on the right tone, and free of leaked personal data. For a code generator it might be: runs, passes tests, uses the approved libraries, and includes no hard-coded secrets. Each criterion becomes a separate check, because a single output can be fluent and correct while quietly failing on safety.

Make each criterion concrete enough that two reviewers would agree on the verdict. Vague words such as helpful invite disagreement; testable statements such as cites the section it used do not. Write down the priority order too, because tradeoffs are real: for a regulated feature, grounded and safe usually outrank clever.

The reliability angle: these named criteria are also your safety spec. When you can list what must always be true, you can measure how often it is not, and you can refuse to ship when a critical criterion regresses.