An analyst manages a Genie Space that has been deployed to a business team. Over time, users are asking new types of questions and some responses are inaccurate. What is the recommended approach for systematically evaluating and improving Genie's accuracy at scale?

Question

Accepted Answer

B. Use benchmarks to run a collection of test questions and score Genie's accuracy. Databricks documentation describes benchmarks as the mechanism to scale up testing and evaluation of responses in a Genie Space. Authors can run a collection of test questions and compare Genie's responses to expected SQL results, scoring accuracy. This systematic approach is preferred over manual log review. Deleting and recreating the space loses historical context. Granting MANAGE privilege would give users admin-level access, which is a security risk.

An analyst manages a Genie Space that has been deployed to a business team. Over time, users are asking new types of questions and some responses are inaccurate. What is the recommended approach for systematically evaluating and improving Genie's accuracy at scale?

Related Questions

Discussion