Why Validate.QA uses many AI models instead of one
Each step of a test run uses the model that does that job best. The lab keeps testing new models and moves a step when one wins, so one plan covers the work of many AI subscriptions.
A single AI subscription gives you one model, and one model is a compromise. The model that is quick at walking an app is not always the model that writes the sturdiest test, or the one that can tell a moved button from a real bug. Validate.QA uses many models, and puts each one on the step it does best.
Our lab is continuously testing the new models that come out every day or week. When one does a step better, we swap it in. The suite keeps catching your application's bugs without you changing a setting, and without you paying for a separate subscription for each model.
What each step is for
These are the steps of a run, and the kind of model that holds each one. The names of the models change. The jobs do not.
How a model earns the step
The lab runs models head to head on real apps. A full run is scored on the quality of the tests it writes and on whether it catches real bugs. A replay of a recorded lab call counts for less, and a step stays put when there is no evidence for a change.
We keep more than one configuration. One spends for the strongest result on each step. One is the everyday setup. One stays on the cheaper model except where that model clearly loses. A custom configuration sets each step on its own. Promoting a winner is an update to the active configuration, so the next run uses it.
When a provider cannot take the call
Where a model can be reached through more than one host, the call tries those hosts in order. The model does not change on the way down. A host with no key is skipped. A spent plan, a rate limit, or a refused key sends the call to the next host. A server error or a timeout does the same only when that fallback is switched on.
If every host fails, the run switches to a different model on another provider when the active configuration has one. Where it does not, the run pauses and resumes when the provider can serve again. The details, including which steps a workspace can be given a different model for, are in how we pick the model for each step .
Topics: Models, Product, Testing.
Read the full article · Get Started Free