Evaluation Scenario Writer - QA

Overview

At Mindrift, we connect domain experts with cutting-edge AI projects. We are seeking an Evaluation Scenario Writer - QA for a project focused on ensuring the quality and correctness of evaluation scenarios created for LLM agents. This is a flexible, project-based opportunity that blends manual scenario validation, automated test thinking, and collaboration with writers and engineers.

Responsibilities

• Reviewing and validating test scenarios from Evaluation Writers
• Spotting logical inconsistencies, ambiguities, or missing checks
• Suggesting improvements to structure, edge cases, or scoring logic
• Collaborating with infrastructure and tool developers to automate parts of the review
• Creating clean and testable examples for others to follow

Qualifications

• Strong QA background (manual or automation), preferably in complex testing environments
• Understanding of test design, regression testing, and edge case detection
• Ability to evaluate logic and structure of test scenarios (even if written by others)
• Experience reviewing and debugging structured test case formats (JSON, YAML)
• Familiarity with Python and JS scripting for test automation or validation
• Clear communication and documentation skills
• Willingness to occasionally write or refactor test scenarios
• Experience testing AI-based systems or NLP applications (valued)
• Familiarity with scoring systems and behavioral evaluation
• Git/GitHub workflow familiarity (PR review, versioning of test cases)
• Experience using test management systems or tracking tools

Benefits

• Get paid for your expertise, with rates that can go up to $55/hour depending on your skills, experience, and project needs
• Take part in a flexible, remote, freelance project that fits around your commitments
• Participate in an advanced AI project and gain valuable experience for your portfolio
• Influence how future AI models understand and communicate in your field of expertise

How To Get Started

Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule.

Employment details

• Seniority level: Internship
• Employment type: Part-time