A Model Serving endpoint has two external models listed: Model A (60% traffic, listed first) and Model B (40% traffic, listed second). Unity AI Gateway fallbacks are enabled. A request is routed to Model A and returns a 429 error. What happens?

Question

Accepted Answer

C. The request falls back to Model B, which is the next served entity in the endpoint's listing order.. Unity AI Gateway documentation states that when fallbacks are enabled and a request returns a 429 or 5XX error, it falls back to the next served entity in the order they were listed during endpoint creation. Traffic percentage does not influence fallback order — listing order determines the fallback sequence.

A Model Serving endpoint has two external models listed: Model A (60% traffic, listed first) and Model B (40% traffic, listed second). Unity AI Gateway fallbacks are enabled. A request is routed to Model A and returns a 429 error. What happens?

Related Questions

Discussion