#329 · 3 AI services with more demand than supply (2026)
I start with the most capable model to verify feasibility, then run evaluations to see how often it succeeds under guardrails. If a smaller, cheaper model performs almost as well (e.g., Astra 6 medium vs high), I'll use it to save tokens. I test across a set of models, averaging results over 10‑20 runs rather than 100, and pick the most efficient that meets the required accuracy.
Yes, I have a framework. First, set your accuracy constraints, determine how accurate you need to be, for example 95%. Second, build your flow. Third, choose the best model, starting with the most expensive option. Run an evaluation to see if the model's accuracy meets or exceeds your goal. If it does, reduce model quality (choose a cheaper model) and rerun the evaluation. For instance, GPT-4.5 might give 97% accuracy, but if your cutoff is 95%, you can try GPT-4.1, which gives about 95% accuracy, saving cost. Repeat this process, testing cheaper models until you find the right balance between cost and ability to perform the desired task.
Yes, utilizing multiple similar AI models is worth it when they hold different tasks. You want to match the model to the task rather than relying on a single model for everything.
I don’t sell AI agents, but running one is cheap: unless you’re using the most expensive model (GPT‑4.5) and paying for everything yourself, the cost is marginal. With cheaper models like GPT‑4.1 mini, you won’t notice the expense unless you’re being spammed constantly.
He suggests two ways to gauge AI‑agent performance. The more thorough method is to create a standardized evaluation set of factual claims, run every agent conversation through it, use a cheaper model to pull out claims, attribute them back to the original conversation via unique IDs, then verify the claims with a stronger model. This gives you an automated hallucination check. A simpler, business‑focused approach is to compare financial results before and after replacing human work with the AI agent: if revenue stays the same or increases, the agent is delivering value; if revenue drops, the AI is costing you more than it saves. Ultimately, AI should be a revenue benefit, not just a cost‑saving metric.