How Validate.QA picks the model for each step
Each step runs on the model that does it best. The lab tests new models continuously and moves a step when one wins. One plan covers them.
Validate.QA does not run the whole pipeline on one model. Each step has its own assignment, and the model on that step is the one that does the job best. Our lab is continuously testing the new models that come out every day or week. When one does a step better, we swap it in, so the suite keeps catching bugs in your application.
One Validate.QA plan covers the models we route. You do not keep a separate subscription for each of them, and you do not need your own AI provider key for those steps.
What each step runs on
The product keeps a few configurations. One spends for the strongest result on each step. One is the everyday setup. One stays on the cheaper model except where that model clearly loses. A custom configuration sets each step on its own. One configuration is active for the product, and we can point a workspace at a different model for the steps that allow an override.
Mapping the app — A fast model walks the running app, including signed-in pages, and groups what it finds into features and flows. A separate pass can then explore one feature in depth. That deeper pass follows the product configuration for every workspace.
Planning — A reasoning model turns the map into the scenarios worth testing. A second pass can check the plan and drop or strengthen a weak expected outcome. The survey's own planner can be set per workspace. Scenario planning follows the product configuration.
Writing UI tests — A model that writes sturdy code turns each flow into Playwright you can read. A workspace can be given a different model for this step.
Writing API tests — A model that follows request chains writes API tests from the calls the app actually made, and can repair an API test that fails. A workspace can be given a different model for this step.
Repairing broken tests — A model that works in a live browser finds the moved step and repairs the test, then the flow is run again. A workspace can be given a different model for this step.
Judging a failure — A careful model separates a broken test, noise, and a real bug in the app. Drift is repaired. A real bug is flagged and quarantined, not rewritten into a passing test. After a discovery run, a separate pass sorts console errors and failed requests the same way: real problems, expected failures, and third-party noise.
Visual review — Layout checks run on every surveyed page without a model: cut-off text, covered controls, broken images, and low contrast. They are findings, not test failures. An optional screenshot review uses a vision model, and a second model can confirm what it saw. That review stays off until it is switched on, and it follows the product configuration for every workspace.
How a model earns a step
The lab runs models head to head on real apps. A full run is scored on the quality of the tests it writes and on whether it catches real bugs. A replay of a recorded lab call is weaker evidence, and it is treated as weaker evidence. A step stays on its current model when there is no evidence for a change.
When a model wins, we promote it: the active configuration is updated and that step moves. The same comparison decides the everyday configuration and the lowest-cost one, which keep a cheaper model on a step unless another model is clearly better there.
When a provider cannot serve the call
Where a model can be reached through more than one host, the call tries those hosts in order. The model does not change on the way down the chain. Inside one host, its accounts are tried in order before the call moves on. A host with no key is skipped. A model with a single host has nowhere else to go.
The call moves to the next host when the current one cannot take it because its plan or credits are spent, it rate-limits the call, or it refuses the key. A server error or a timeout moves to the next host only when that fallback is switched on. Until then it is recorded and the call stays on the host that failed.
The last host in a chain is used on the production service. Local runs do not use it unless that is switched on too.
If every host in the chain fails, the run switches to a different model on another provider when one is configured. The configuration aimed at the strongest result sets one. The everyday and lowest-cost configurations do not yet, so a spent plan pauses those runs and resumes them. A custom configuration uses a fallback model only when one has been stored.
Your own AI provider key
You do not need your own AI provider key for the models we route. Those calls use Validate.QA's keys. Solo Dev includes that AI, and Enterprise does too.
MCP Army is the plan where you bring your own AI. You drive Validate.QA through the MCP server and the local runner, and you pay your own AI provider. The steps that use our models stay locked until you move to Solo Dev: survey, discovery, healing, test generation, visual review, and the rest of that list. There is no setting where you paste a provider key to replace a routed step.
Open How Validate.QA picks the model for each step · Get Started Free