Senior Architect, Quality Engineering
Icertis
Icertis is the global leader in AI-powered contract intelligence. The Icertis platform revolutionizes contract management, equipping customers with powerful insights and automation to grow revenue, control costs, mitigate risk, and ensure compliance - the pillars of business success. Today, more than one third of the Fortune 100 trust Icertis to realize the full intent of millions of commercial agreements in 90+ countries.
Who we are: Icertis is the only contract intelligence platform companies trust to keep them out in front, now and in the future. Our unwavering commitment to contract intelligence is grounded in our FORTE values—Fairness, Openness, Respect, Teamwork and Execution—which guide all our interactions with employees, customers, partners, and stakeholders. Because in our mission to be the contract intelligence platform of the world, we believe how we get there is as important as the destination.
Icertis, Inc. provides Equal Employment Opportunity to all employees and applicants for employment without regard to race, color, religion, gender identity or expression, sex, sexual orientation, national origin, age, disability, genetic information, marital status, amnesty, or status as a covered veteran in accordance with applicable federal, state and local laws. Icertis, Inc. complies with applicable state and local laws governing non-discrimination in employment in every location in which the company has facilities. If you are in need of accommodation or special assistance to navigate our website or to complete your application, please send an e-mail with your request to careers@icertis.com or get in touch with your recruiter.
We are looking for an experienced QA AI Architect to design and lead the architecture of AI-driven Quality Engineering systems for next-generation AI-powered products. This role will focus on building scalable frameworks for testing and validating AI systems built on Large Language Models (LLMs), embeddings, vector databases, and modern AI pipelines. The architect will work closely with product, engineering, and AI teams to design intelligent QA solutions including AI-assisted test generation, intelligent automation, document understanding, and AI-driven validation systems. The ideal candidate combines deep QA architecture experience with strong AI/ML system knowledge, and is passionate about redefining quality engineering for AI products.
Nice to Have
- Experience building AI agents or autonomous systems
- should add Architect testing frameworks for agentic AI workflows and multi-agent orchestration
- Knowledge of LLM evaluation frameworks
- Understanding of AI safety and guardrails
- Experience architecting AI-powered SaaS products
What Makes This Role Exciting
You will play a key role in shaping the future of AI-powered Quality Engineering, building intelligent platforms that enable automated test generation, AI-driven validation, and scalable QA systems for AI products.
Key Responsibilities
AI Architecture & LLM Systems
- Architect and build AI-powered quality engineering platforms using LLMs and SLMs
- Evaluate and integrate open-source models such as Llama and Mistral
- Design prompt engineering workflows and Retrieval-Augmented Generation (RAG) pipelines
- Architect systems for LLM evaluation, guardrails, and validation frameworks
- Optimize architectures for token efficiency, cost, and performance
Embeddings & Vector Search
- Design and implement embedding pipelines for semantic retrieval
- Implement vectorization strategies for documents, test artifacts, and product knowledge bases
- Architect vector storage using technologies such as
- Pinecone
- Weaviate
- Milvus
- FAISS
- Optimize chunking and indexing strategies for large knowledge repositories
Model Optimization & Fine-Tuning
- Implement model fine-tuning pipelines using LoRA / PEFT techniques
- Evaluate model performance, reliability, and hallucination mitigation
- Manage training datasets and model evaluation metrics
Backend & System Architecture
- Design microservice-based AI system architectures
- Build scalable backend services and APIs for model inference and evaluation
- Develop data pipelines supporting AI workflows and large-scale inference systems
MLOps & AI Lifecycle
- Build MLOps pipelines for model deployment, monitoring, and retraining
- Implement CI/CD pipelines for AI systems
- Enable model observability, performance monitoring, and version control
OCR & Document Intelligence
- Design systems for OCR-based document processing
- Integrate OCR pipelines with embedding and AI workflows
- Build solutions for document understanding and QA automation
Required Skills
AI & LLM Systems
- Strong understanding of LLM and SLM architectures
- Experience working with open-source LLMs
- Prompt engineering and RAG architectures
- Model evaluation frameworks and hallucination control
- Token optimization strategies
Embeddings & Vector Databases
- Experience working with embedding models and semantic search
- Hands-on experience with vector databases such as
- FAISS
- Pinecone
- Weaviate
- Milvus
- Infuse AI with
- Automated Test Failure analysis
- Crawl Product/Test code to figure out whether issue is probable Product/Test/Env issue
Programming & AI Frameworks
- Strong Python programming skills
- Experience with AI frameworks such as
- LangChain
- LlamaIndex
- Backend development using FastAPI / Flask / Node.js
Backend & Cloud Architecture
- Microservices architecture and REST APIs
- Containerization using Docker
- Experience working on AWS / Azure / GCP
MLOps
- Model deployment pipelines
- CI/CD for machine learning systems
- Monitoring and observability of AI models
OCR & Document Processing
- Experience with OCR frameworks such as
- Tesseract OCR
- PaddleOCR
System Foundations
- Strong understanding of system design, operating systems, networking, and distributed systems