The work you'd
be doing.
AI models learn from expert judgment. Here are the main kinds of work, each with a worked example — the task on the left, what an expert actually produces on the right.
Score an answer
against a rubric
Judge an AI's response on each criterion — accuracy, safety, completeness, reasoning — instead of a thumbs up/down.
Rate each criterion against its anchors and give a short rationale.
"Aspirin + ECG are right, but 'refer and monitor' delays a time-critical reperfusion pathway — that's a safety failure, not a minor gap."
Pick the genuinely
better response
Compare two answers to the same prompt and choose the stronger one — not just the one that sounds more confident.
Select the better response and justify why it serves the user's intent.
"A calls unlimited liability 'standard' and misses the material risk. B identifies the uncapped exposure a client must not sign blind."
Write the Q&A pairs
models learn from
Models need thousands of accurate question-and-answer pairs, authored by people who actually know the field.
Write clear, correct prompt-and-answer pairs in your domain.
"Self-contained, correct, and shows the derivation — not just the formula. This is the pair the model trains on."
Label exactly
what's wrong
Mark the specific span, the type of error, and how serious it is — turning a vague "this is bad" into structured signal.
Flag each mistake, categorize it, and rate its severity.
return db.query("SELECT * FROM users WHERE id = " + id)
"User input concatenated straight into the query. Flagged span, error type 'injection', severity Critical — cannot ship."
Judge an agent's
step-by-step path
For AI agents that act in steps, check whether each move was right — or supply the correct sequence.
Find where the agent went wrong and give the correct steps.
2 · Cite first blog result (undated) ⚠
3 · Answer from that blog
"Correct path: cite the official tax-authority page, not an undated blog. The answer may be right by luck, but the source isn't defensible."
Probe for failures
only you'd find
Test a model for the unsafe edge cases a specialist would think to try, then document how to reproduce them.
Find the failure, capture the triggering prompt, and rate the risk.
"Never caps the daily total — repeated dosing risks liver toxicity. Reproduces consistently across phrasings. Logged with the trigger prompt."
Illustrative examples of task types — sample prompts, outputs, and expert responses. Not real client data.