Business Domain Expert — AI Model Evaluation Pilot (Admin, Marketing, HR, Accounting)

Accountabilities:

• • Sign a confidentiality agreement before receiving project materials and ensure all study information is handled appropriately.
  • Complete one realistic, domain-specific professional task using the source materials provided.
  • Review five AI-generated responses to the same task and rank them from 1 to 5 with no ties.
  • Assign each model response a score from 0–100 based on the quality of the work and your professional judgment.
  • Write clear, specific, evidence-based rationales that reference relevant portions of each response to support your evaluation.
  • Assess the accuracy, reasoning, practical applicability, and overall quality of the model-generated work within your professional domain.
  • Provide structured feedback on the task itself and the overall study design to help improve future evaluation processes.
  • Complete all work independently without using LLMs or other AI assistants during task completion, evaluation, scoring, or rationale writing.

  Requirements

  • 5+ years of hands-on professional experience in Human Resources or Accounting.
  • Current or recent professional practice in your field, with sufficient experience to evaluate work according to real-world professional standards.
  • Strong ability to assess professional work critically and distinguish high-quality reasoning from flawed, incomplete, or unsupported conclusions.
  • Excellent written communication skills, with the ability to produce concise, specific, and evidence-grounded explanations for your judgments.
  • Strong attention to detail, analytical thinking, and professional judgment.
  • Ability to work reliably within a short turnaround period and complete a structured evaluation independently.
  • Comfortable working without a predefined rubric or “golden answer,” using your own professional expertise as the primary basis for evaluation.
  • Must be legally able to participate as an independent contractor in Canada.
  • Must not have previously worked on Project Alchemy in any capacity or accessed its source-world data, as prior exposure would invalidate the study.
  • Candidates requiring H-1B or STEM OPT sponsorship are not eligible for this engagement.

  Benefits

  • Fully remote, independent-contractor engagement that can be completed on your own schedule.
  • Estimated commitment of 8–12 hours for the one-time evaluation study.
  • Opportunity to contribute directly to the development and evaluation of advanced AI systems.
  • Apply your professional expertise to realistic business scenarios without a long-term production commitment.
  • Competitive payment for completed work, with weekly payments via Stripe or Wise.
  • Flexible project structure; engagements may be extended, shortened, or concluded early depending on project needs and performance.
  • No access to confidential or proprietary information belonging to your current or former employers, clients, or institutions.
  • Opportunity to earn up to $400 per successful referral, subject to applicable restrictions.