Evaluation Testing and Optimization Questions
Practice questions for Evaluation Testing and Optimization topic in Claude Certified Architect – Professional. 31 questions covering this domain.
An architect must design an evaluation framework for a Claude-powered contract review system before production deployment. The system analyzes contrac...
An architect must design A/B testing for a prompt optimization that claims to improve customer satisfaction scores. Which evaluation design provides t...
Which metric is most appropriate for evaluating a Claude system that must retrieve all relevant items from a knowledge base?
A production Claude system passed all pre-deployment evaluations with 96% accuracy but production monitoring shows 84% accuracy after 2 weeks. Investi...
A production Claude system's accuracy has degraded from 94% to 89% over 3 months with no code changes. An architect must diagnose and resolve the issu...
An architect needs to evaluate a Claude-based email classification system that routes emails to 5 different departments. The current evaluation shows ...
What is the primary purpose of LLM-as-judge evaluation pattern in Claude system testing?
An architect is optimizing a Claude document summarization system. The current implementation uses Claude Sonnet 5 and costs $5,000/month for 1 millio...
An architect needs to measure whether a Claude system successfully retrieves all relevant safety warnings from a technical documentation database when...
A production Claude system for medical coding (assigning ICD-10 diagnostic codes to clinical notes) has been operating for 6 months. Initial evaluatio...
An architect is evaluating two prompt variants for a Claude-powered customer service chatbot: Prompt A (current production) and Prompt B (proposed opt...
An architect is designing an evaluation framework for a Claude-powered contract negotiation assistant that suggests contract clause modifications to p...
A Claude-powered document classification system assigns incoming support tickets to 8 different department queues (billing, technical, sales, shipping...
An architect must design a pre-production evaluation framework for a Claude-powered medical literature research system that answers physician question...
A Claude-powered resume screening system for a recruitment firm has been in production for 8 months, initially evaluated at 91% accuracy. Recent hirin...
A development team deployed a Claude-powered customer service assistant directly to production after testing with 5 sample conversations that 'seemed ...
A retail company deployed a Claude-powered product recommendation system 6 months ago. Offline evaluation at deployment showed 91% recommendation rele...
A law firm deploys a Claude-powered agentic legal research assistant. Attorneys submit queries and the agent autonomously searches legal databases usi...
A healthcare company operates a production Claude system for clinical documentation assistance that was accepted at 96% accuracy on their internal ben...
An architect has developed a new system prompt for a production Claude-powered e-commerce search assistant. Offline evaluation on a 500-question test ...
Sign in to see all 31 questions
Create a free account to browse the questions included with the free plan.