Applied Engineer, Evaluations Role
$100–120/hr
Remote part time STEM contract with Mercor, $100–120/hr. Posted Jun 2, 2025.
$100–120/hr
Apply on MercorThis post is over a year old. It was still on Mercor's site on Nov 6, 2026, so it is listed here, but it may be a standing pool rather than a project starting now. Expect a wait after you apply.
Is this job worth applying to?
- Pay
- Above typicalTops out at $120/hr. The typical top rate across 148 priced STEM listings is $70/hr.
- Getting in
- Some screening~30 min: resume upload + 20-min AI interview. Resume showing real experience in your field.
- Steady work
- Work is patchyProjects can pause or end with little notice.
- Payout record
- Pays on timeSchedule: weekly, every Wednesday. Paid via Stripe (Wise in a few countries).
- Freshness
- Over a year oldPosted over a year ago. Still listed on Mercor on Nov 6, 2026.
- Where you live
- Not statedThe listing names no country.
The job description
Mercor is seeking software engineers to build and refine evaluations. You’ll turn completed pull requests into engineering tasks and use our in-house evaluation framework to multiple frontier coding agents, including Claude Code, Codex and more.
We work alongside an in-house research team and have produced industry leading benchmarks to compare the performance of frontier large language models.
In this role, you will:
- Identify repositories with enough substantive work to support challenging evaluations, and select suitable completed PRs.
- Investigate each problem, its reference solution, and the repository’s architecture, tests, and conventions.
- Write task prompts and build evaluations using automated tests, shell commands, and LLM grading prompts.
- Validate evaluations against reference solutions and deliberately flawed implementations. Find missing checks, incorrect grades, and criteria that unnecessarily constrain how a problem can be solved.
- Run evaluations repeatedly across multiple models and harnesses. Investigate whether failures come from the agent’s solution, the environment, or the grading, and establish that tasks expose meaningful weaknesses in agent performance.
- Refine evaluations through repeated testing and review. Document findings and grading decisions, and work through feedback.
You’ll need:
- Strong programming fundamentals and practical experience working in substantial and complex codebases. Languages we create evals for include TypeScript/JavaScript, Python, Java, Kotlin, Go, Ruby, PHP, C++ and Rust
- The ability to understand unfamiliar code, investigate subtle behavior, and assess whether different implementations solve the same problem correctly.
- Experience writing meaningful tests, including edge cases and regression coverage.
- Attention to detail and patience for repeated investigation and refinement.
- Practical experience using AI coding tools, with the judgment to verify their output and catch mistakes.
- Confidence using Git, test runners, and CI tooling.
- Clear and fluent written and verbal communication with the ability to own a task independently while raising questions when requirements are ambiguous.
Relevant experience can come from open-source, private, or enterprise repositories. Familiarity with a particular language or ecosystem is helpful; the ability to learn the repository and make sound engineering judgments matters more than its popularity or your public contribution history.
We provide onboarding to the evaluation framework and ongoing review feedback. After onboarding, you’ll be expected to own task selection, evaluation development, and iteration without step-by-step direction.
How do you apply to Mercor?
Apply on Mercor
The Apply button opens this post on Mercor's own site. You apply with them, not with Rubric. Budget about 30 min: resume upload + 20-min AI interview.
Get through screening
What Mercor asks for after you apply:
- Take the recorded AI video interview (practice round available)
- Complete any role-specific assessment and the ID check
- Accept the offer and sign the contract
Start work and get paid
Pay schedule: weekly, every Wednesday. Paid via Stripe (Wise in a few countries). You're paid for the work they hand out, not a fixed salary.
Mercor also needs
- Resume showing real experience in your field
- Webcam with your full face visible for the AI interview
- Government ID
Red flags
- Red flagLong waits with no offer after passing the interview
- Red flagProjects ending abruptly with little warning
What is Mercor like to work for?
Hiring marketplace that screens professionals with an AI video interview and places them on hourly contracts training and evaluating models for AI labs.
| Advertised pay | $20–200/hr |
|---|---|
| Getting in | Some screening |
| Application takes | ~30 min: resume upload + 20-min AI interview |
| How you get paid | Weekly, every Wednesday, via Stripe (Wise in a few countries) |
| Payout record | Pays on time |
| Countries | Hires in many countries and runs its pay week on India Standard Time; Mercor keeps a list of locations it cannot onboard, and some projects are US-only. |
Most listings sit around $50–100/hr; healthcare, legal and finance roles pay the most, language and generalist roles the least.
