Description
LLM AQA Engineer
Requirements
We are looking for a highly skilled LLM AQA Engineer to spearhead the quality, reliability, and security of our Generative AI platforms. In this role, you won’t just be testing software; you will be designing advanced automation frameworks to evaluate stochastic AI behaviors, validate complex data pipelines, and ensure our language models are safe, performant, and cost-effective.
- 6+ years in software test automation, with at least 1–2 years dedicated specifically to testing AI/ML or LLM-powered applications.
- A data-driven approach to solving the ambiguous, non-deterministic challenges of testing Generative AI.
- Strong communication skills to bridge the gap between AI Engineers, Data Engineers, Product Managers, and Stakeholders.
- Expert-level Python programming skills with deep experience in pytest, asynchronous programming, and building modular automation frameworks.
- Practical understanding of Large Language Models, vector databases (e.g., Pinecone, Milvus, Chroma), and orchestration tools (e.g., LangChain, LlamaIndex).
- Strong experience with API automation tools and validating protocols across both REST and gRPC.
- Familiarity with tracing and monitoring tools for LLM debugging.
- Understanding of OWASP Top 10 for LLMs, PII masking, and data leakage prevention mechanisms.
Job responsibilities
- Design and implement automated evaluation strategies to assess model determinism, hallucinations, and alignment using core parameters like Temperature and Top_P.
- Audit the integrity of AI data pipelines, focusing on chunking quality, embedding drift, vector database integrity, and retrieval consistency.
- Validate fine-tuned models against baseline metrics to ensure regression-free improvements in domain-specific tasks.
- Implement robust versioning and reproducibility checks for prompts, system instructions, datasets, and model configuration variants.
- Build, scale, and maintain scalable, robust test automation frameworks from scratch using Python and pytest.
- Conduct comprehensive testing of RESTful and gRPC APIs, implementing contract validation to ensure seamless service integration.
- Integrate automated AI evaluation suites into the DevOps pipeline for continuous model and software deployment.
- Utilize tracing tools and centralized logging to troubleshoot LLM chains, agents, and complex RAG workflows.
- Define and monitor production-level evaluation metrics
- Design and execute load and stress tests focused on throughput, latency, and token consumption efficiency.
- Conduct automated security testing targeting PII leakage, data privacy compliance, prompt injection vulnerabilities, and RBAC/access controls.
What we offer
Exciting Projects: We focus on industries like High-Tech, communication, media, healthcare, retail and telecom. Our customer list is full of fantastic global brands and leaders who love what we build for them.
Collaborative Environment: You Can expand your skills by collaborating with a diverse team of highly talented people in an open, laidback environment — or even abroad in one of our global centers or client facilities!
Work-Life Balance: GlobalLogic prioritizes work-life balance, which is why we offer flexible work schedules, opportunities to work from home, and paid time off and holidays.
Professional Development: Our dedicated Learning & Development team regularly organizes Communication skills training(GL Vantage, Toast Master),Stress Management program, professional certifications, and technical and soft skill trainings.
Excellent Benefits: We provide our employees with competitive salaries, family medical insurance, Group Term Life Insurance, Group Personal Accident Insurance , NPS(National Pension Scheme ), Periodic health awareness program, extended maternity leave, annual performance bonuses, and referral bonuses.
Fun Perks: We want you to love where you work, which is why we host sports events, cultural activities, offer food on subsidies rates, Corporate parties. Our vibrant offices also include dedicated GL Zones, rooftop decks and GL Club where you can drink coffee or tea with your colleagues over a game of table and offer discounts for popular stores and restaurants!
About GlobalLogic
GlobalLogic, a Hitachi Group Company, is a trusted digital engineering partner to the world’s largest and most forward-thinking companies. Since 2000, we’ve been at the forefront of the digital revolution – helping create some of the most innovative and widely used digital products and experiences. Today we continue to collaborate with clients in transforming businesses and redefining industries through intelligent products, platforms, and services.
