AI Evaluation Rubric Auditor
Flash · Remote
- Pay
- $60–85/hr
- Commitment
- hourly
- Hours / week
- ~15
- Source
- mercor
Is this job right for you?
Get job recommendations matched to your experience, first access to exclusive opportunities, and no ads.
Get jobs matched to your experienceAbout this role
**The work** We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair: \- Was the information available? \- Does the rubric accurately grade what the prompt asked for? \- Would a valid answer have been marked wrong? \- Is the rubric structure correct per the guidelines? This role includes providing feedback to reviewers regarding prompt and rubric issues. **Must-haves** \- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application. \- Experience providing feedback to writers/reviewers \- Consistent availability \- Can read a long agent transcript and find where it went wrong. \- Precise writing, and you check the data instead of trusting the reference answer.
Eligible applicant countries
This role accepts applicants from:
- ARG
- BOL
- BRA
- CAN
- CHL
- COL
- ECU
- GBR
- GUY
- PER
- PRY
- URY
- USA
Skills & domains
- ai-training
- rlhf
- sme
- annotation
- Miscellaneous

