Code Quality Specialist / Research Engineer
This is a referral link. If you join through it, RemoteStack may earn a referral bonus.
About the role
Pay: $75 - $105/hour
Client: Google DeepMind
About the Opportunity
Our client, a leading AI research organization, is looking for engineers and computational researchers to help evaluate how AI systems solve coding tasks.
You’ll investigate whether an unsuccessful result comes from the AI’s code, an issue in the tests, or unclear task requirements. This work combines code reading, debugging and technical judgment to help make AI evaluations more accurate and reliable.
The role is suited to people who enjoy understanding unfamiliar codebases, examining test behavior and explaining technical findings clearly.
What You’ll Do
- Review code, tests, execution logs and the steps an AI coding agent took while attempting a task.
- Investigate reported problems in coding benchmarks, including undocumented expectations, missing definitions and overly restrictive test assertions.
- Determine whether a reported issue is a genuine flaw, a gap that still permits reasonable solutions to pass, or an error in the AI’s implementation.
- Assess whether an identified issue affected the task’s pass/fail result.
- Write concise technical explanations supported by relevant excerpts from task instructions, tests and run logs.
- Diagnose test-harness and environment issues using Docker configuration, dependencies and execution logs.
- Participate in daily calibration discussions with research scientists to compare findings and align evaluation decisions.
Individual claim investigations are expected to take approximately 30–60 minutes. You’ll use review tools and supporting technical resources to examine the evidence and document a clear conclusion.
What We’re Looking For
- Either 2+ years of professional experience writing, reviewing and debugging software, or a completed STEM master’s degree or PhD with intensive programming and computational research experience.
- Strong Python code-reading, debugging and comprehension skills, including familiarity with complex frameworks and test assertions.
- Reading familiarity with at least one additional language: JavaScript/TypeScript, Go, C++ or Rust.
- Practical experience with Git and Docker.
- Familiarity with unit-testing frameworks, such as pytest, unittest, Jest or Mocha, and software build environments.
- Careful attention to edge cases, ambiguous requirements and discrepancies in test logic.
- Fluent professional English, with the ability to communicate technical findings clearly in writing and conversation.
Nice to Have
- Experience with LLM benchmarking, AI evaluation, prompt engineering or coding-agent workflows.
- Familiarity with benchmarks such as SWE-bench or CodeContests.
- Experience in AI red teaming, safety or reliability auditing.
- Previous collaboration with AI or machine-learning research scientists.
Engagement Details
- Expected commitment: 35 hours per week.
- Start: Immediate.
Compensation
$75–$105 USD per hour, depending on relevant skills and experience.
Interview Process
Shortlisted candidates will be invited to a 15-minute AI interview. The hiring team will then review your profile and interview responses to determine next steps, with no additional interview rounds required.
