Senior QA Automation Engineer
About the role
We are looking for an experienced Lead QA Engineer to own quality for the Net2Phone billing platform. We are building a billing system where every invoice is provably correctly rated, taxed, prorated, reconciled and where the team can defend that claim with documented coverage rather than assumptions. You will focus on testing billing correctness end-to-end: how a call becomes a rated record and then an invoice line, where usage gets lost or double-counted between the switch and the bill, how proration behaves when a customer changes plans mid-cycle, and how a third-party tax engine's output is verified when you can't inspect it. You are equally skilled in leading a team: setting the documentation standard, acting as the intake gatekeeper with Product, and deciding where automation effort yields true value. This is a manual-testing-heavy position: automation leadership here means planning, prioritization, and coverage governance, not day-to-day framework development. You will not be the primary implementer, but you will be expected to read, review, and debug the team's suites. PLEASE SUBMIT CV IN ENGLISH. ONLY ACCEPTING CANDIDATES FROM LATAM. ADVANCED SPOKEN ENGLISH IS REQUIRED.
Responsibilities
Embed fully within a Scrum team, participating in all ceremonies and maintaining associated artifacts.
Own the end-to-end test strategy for the AI voice virtual agent across the LLM, speech, and telephony layers.
Design test scenarios for complex conversational behavior, including multi-turn dialogues, interruptions and barge-in, ambiguous or incomplete input, out-of-scope requests, human escalation/transfer, and failure recovery.
Build and maintain an evaluation framework for non-deterministic output, leveraging golden datasets, semantic/rubric-based assertions, and automated regression detection for prompt, model, or retrieval changes.
Adversarially test the agent against hallucinations, incorrect grounding, guardrail bypasses, prompt injections, and unauthorized disclosure of customer data.
Prepare and provision SIP devices, endpoints, and call scenarios to automate call origination end-to-end.
Diagnose failures across the entire stack, isolating issues down to the LLM, ASR (speech recognition), TTS (text-to-speech), media path, or call routing layer.
Extend automated coverage and integrate test suites into the CI/CD pipeline to serve as release gates.
Define and track core quality metrics, including task completion, containment, intent accuracy, transcription accuracy, response latency, and escalation correctness.
Maintain documented test coverage in the TMS with full traceability to business requirements.
Communicate and escalate quality risks to stakeholders backed by data and evidence.
Requirements:
5+ years in QA, with recent hands-on ownership of a complex, integrated product.
Practical experience testing SIP/VoIP systems, including call flows, SIP signaling, media handling, softphones, or SIP trunking, with the ability to troubleshoot using SIP traces or packet captures.
Demonstrated creativity in test design (live scenario design will be a core part of the interview process and heavily weighted).
Proficiency in Python, sufficient to independently build and maintain automation frameworks.
Experience testing non-deterministic or probabilistic systems, with a clear approach to designing assertions for variable outputs.
Strong analytical skills, including the ability to define quality metrics, justify their calculation, and use data to back release decisions.
Hands-on experience with formal TMS tools and Jira.
Professional fluency in English (spoken and written), with reliable overlap with US Eastern business hours.
We use the following technologies:
Python with pytest for test automation.
Playwright for browser and WebRTC-based call driving.
JsSIP / WebRTC for programmatic call origination and media control.
SIP softphones and devices for manual and exploratory testing.
Allure TestOps as our TMS, linking automated runs to documented test cases.
Jira for project management and issue tracking.
Jenkins & GitHub Actions for CI/CD pipelines.
Grafana for quality and delivery metrics.
LLM APIs and evaluation tooling — currently in active development, and we are open to any effective approach.
Pluses:
Direct experience testing LLM-based or conversational AI products, including evaluation frameworks and LLM-as-a-judge approaches.
Familiarity with speech technologies, including ASR/transcription accuracy measurement (WER), TTS quality assessment, endpointing, and silence detection.
Experience with RAG or retrieval-grounded systems, specifically in evaluating grounding correctness and context retrieval.
WebRTC and media-layer testing experience, including codec behavior and performance degradation under jitter or packet loss.
Background in testing IVR, contact center, or conversational IVR platforms.
Multilingual testing capabilities, with emphasis on Spanish/English code-switching.
Literacy in prompt engineering, sufficient to analyze system prompts and identify potential failure modes
We are pleased to offer:
Remote work opportunity!
B2B Employment ($, gross).
Stable job with long-term growth perspective.
Competitive salary with annual performance review.
Really good hardware.
An exciting and challenging job with talented people around.
Continuous learning and career growth opportunities.
Compensation for professional training, seminars, and conferences.
Referral program: get rewarded for helping us grow the team with talented people.
Company-supported English classes to enhance your professional growth.