Senior Python Engineer - AI Code Evaluation (Codex / Claude Code, up to $200/hr)

Accountabilities:

• • Evaluate AI coding interactions: Review AI-generated coding sessions end to end and assess whether responses are useful, correct at a high level, and aligned with how an experienced engineer would approach the problem.
  • Assess engineering judgment: Evaluate whether AI agents demonstrate sound reasoning, appropriate trade-offs, clear decision-making, and practical software-engineering judgment rather than simply producing syntactically correct code.
  • Review explanations and reasoning: Judge the quality of model preambles, explanations, reasoning, and guidance, identifying whether they genuinely help developers or simply generate technically accurate but unhelpful output.
  • Provide structured feedback: Clearly and directly explain what worked, what did not, and what felt misleading, ineffective, or inconsistent with strong engineering practice.
  • Differentiate response quality: Apply consistent standards to distinguish varying levels of AI performance and articulate what separates an average response from an excellent one.
  • Help define evaluation standards: Contribute to the evolving understanding of what high-quality AI-assisted software development should look like, including interactions with AI-first development environments.
  • Apply independent judgment: Make subjective but rigorous assessments based on extensive engineering experience, without needing to execute or deeply inspect every line of generated code.

  Requirements:

  • Senior engineering experience: Staff-, Principal-, or similarly experienced software engineering background, with a strong track record of making technical decisions and maintaining a high standard for software quality.
  • Strong programming expertise: Deep hands-on experience with Python and/or TypeScript/JavaScript, with the ability to quickly understand and assess modern software implementations.
  • AI coding experience: Practical experience using AI-assisted development tools such as OpenAI Codex, Claude Code, Cursor, or comparable coding agents.
  • Modern development knowledge: Strong familiarity with contemporary AI-assisted software-development workflows and how developers interact with coding agents.
  • Engineering judgment: Ability to recognize whether an AI response reflects strong engineering thinking, including useful explanations, sound trade-offs, appropriate guidance, and trustworthy recommendations.
  • Critical evaluation skills: Comfortable evaluating code and technical reasoning at a high level without needing to execute or inspect every implementation detail.
  • Communication skills: Able to provide concise, direct, and opinionated feedback that clearly explains why an interaction succeeds or falls short.
  • High quality bar: Strong personal standards for what constitutes effective, maintainable, and professional engineering.
  • Preferred experience: Exposure to AI-first IDEs, prompt design, model evaluation, evaluation workflows, or engineering-standard development is a plus.
  • Additional advantage: Experience mentoring senior engineers or helping establish engineering standards and best practices is beneficial.

  Benefits:

  • Compensation: $100–$200 per hour, depending on experience and fit.
  • Flexible engagement: Approximately 10–20 hours per week.
  • Remote work: Fully remote contract engagement with flexibility across eligible locations.
  • Short-term focused engagement: Initial engagement runs through early May, with potential for extension.
  • Fast start: Opportunity to begin as soon as possible.
  • Meaningful AI impact: Directly contribute to evaluating and improving how AI coding agents support professional software engineers.
  • Senior-level scope: Apply your engineering expertise to challenging questions around AI reasoning, developer experience, and software-engineering quality.
  • Streamlined selection: Hiring process consists of a take-home evaluation exercise followed by one behavioral interview.