Braintrust
Braintrust

Code Quality Specialist / Research Engineer

datacontractRemote - United States
SALARY
$75–105/hr
WORK TYPE
remote
JOB TYPE
contract
INDUSTRY
ai
Apply for this position

This is a referral link. If you join through it, RemoteStack may earn a referral bonus.

✦ AutoApply Sick of applying? We apply to roles like this for you, up to 20 a month.
Learn more

About the role

Pay: $75 - $105/hour

Client: Google DeepMind

About the Opportunity

Our client, a leading AI research organization, is looking for engineers and computational researchers to help evaluate how AI systems solve coding tasks.

You’ll investigate whether an unsuccessful result comes from the AI’s code, an issue in the tests, or unclear task requirements. This work combines code reading, debugging and technical judgment to help make AI evaluations more accurate and reliable.

The role is suited to people who enjoy understanding unfamiliar codebases, examining test behavior and explaining technical findings clearly.

What You’ll Do

  • Review code, tests, execution logs and the steps an AI coding agent took while attempting a task.
  • Investigate reported problems in coding benchmarks, including undocumented expectations, missing definitions and overly restrictive test assertions.
  • Determine whether a reported issue is a genuine flaw, a gap that still permits reasonable solutions to pass, or an error in the AI’s implementation.
  • Assess whether an identified issue affected the task’s pass/fail result.
  • Write concise technical explanations supported by relevant excerpts from task instructions, tests and run logs.
  • Diagnose test-harness and environment issues using Docker configuration, dependencies and execution logs.
  • Participate in daily calibration discussions with research scientists to compare findings and align evaluation decisions.

Individual claim investigations are expected to take approximately 30–60 minutes. You’ll use review tools and supporting technical resources to examine the evidence and document a clear conclusion.

What We’re Looking For

  • Either 2+ years of professional experience writing, reviewing and debugging software, or a completed STEM master’s degree or PhD with intensive programming and computational research experience.
  • Strong Python code-reading, debugging and comprehension skills, including familiarity with complex frameworks and test assertions.
  • Reading familiarity with at least one additional language: JavaScript/TypeScript, Go, C++ or Rust.
  • Practical experience with Git and Docker.
  • Familiarity with unit-testing frameworks, such as pytest, unittest, Jest or Mocha, and software build environments.
  • Careful attention to edge cases, ambiguous requirements and discrepancies in test logic.
  • Fluent professional English, with the ability to communicate technical findings clearly in writing and conversation.

Nice to Have

  • Experience with LLM benchmarking, AI evaluation, prompt engineering or coding-agent workflows.
  • Familiarity with benchmarks such as SWE-bench or CodeContests.
  • Experience in AI red teaming, safety or reliability auditing.
  • Previous collaboration with AI or machine-learning research scientists.

Engagement Details

  • Expected commitment: 35 hours per week.
  • Start: Immediate.

Compensation

$75–$105 USD per hour, depending on relevant skills and experience.

Interview Process

Shortlisted candidates will be invited to a 15-minute AI interview. The hiring team will then review your profile and interview responses to determine next steps, with no additional interview rounds required.

✦ Sick of applying to 40 jobs a month?
I rewrite your resume for ATS by hand first. Once you sign off on it, AutoApply applies to up to 20 roles like this a month, cover letter in your own voice each time. From $99/mo, cancel anytime.
Get AutoApply
Apply now