A production Claude system passed all pre-deployment evaluations with 96% accuracy but production monitoring shows 84% accuracy after 2 weeks. Investigation reveals the gold standard evaluation dataset contained only 50 examples from a single source. Which evaluation framework weakness does this expose, and how should the architect remediate?
This question requires Pro
Unlock all 31 questions in this certification
Written by certified professionals · Aligned to official exam objectives
View all Pro featuresMore Evaluation Testing and Optimization Questions
31 questions
Full Claude Certified Architect – Professional Practice Test
All topics covered
All Claude Certified Architect – Professional Questions
Browse by topic
Related Questions
An architect must design an evaluation framework for a Claude-powered contract review system before ...
A production Claude system's accuracy has degraded from 94% to 89% over 3 months with no code change...
An architect needs to evaluate a Claude-based email classification system that routes emails to 5 di...
What is the primary purpose of LLM-as-judge evaluation pattern in Claude system testing?...
An architect is optimizing a Claude document summarization system. The current implementation uses C...
Educational Content — CertQnA practice questions are written against official exam objectives, covering the same domains tested on the real exam. All content is original and independent — not actual exam questions, not affiliated with any certification vendor. Learn more about our content policy