Contenu source (brut)
<p class="wp-block-paragraph">Imagine running a business where a supplier regularly delivers late, customers receive incomplete orders, and some gradually stop buying from you.</p>
<p class="wp-block-paragraph">Could an AI recognize that its purchasing decisions are costing the business money and change its approach before the losses grow?</p>
<p class="wp-block-paragraph">That’s the question Bittensor’s <strong>ORO (SN15)</strong> set out to answer with <a href="https://oroagents.com/benchmarks/smb-bench">SMB Bench</a>, a new benchmark released on November 6, 2026.</p>
<p class="wp-block-paragraph">ORO tested nine leading AI models by putting them in charge of purchasing decisions for a simulated wholesale business over 90 days.</p>
<p class="wp-block-paragraph">The goal was to see whether they could make profitable decisions, learn from mistakes, and respond to problems whose consequences only become visible weeks later.</p>
<p class="wp-block-paragraph">The results revealed a major limitation. Only one of the nine models earned more than a simple, automated buying rule on average, while several ended the simulated quarter with substantial losses.</p>
<h2 class="wp-block-heading">How ORO’s New Benchmark Works</h2>
<p class="wp-block-paragraph">ORO is a Bittensor subnet focused on developing and evaluating AI agents for commerce.</p>
<p class="wp-block-paragraph">On SN15, miners build shopping agents that compete to complete purchasing tasks, with validators testing their performance and rewarding the strongest agents.</p>
<p class="wp-block-paragraph">SMB Bench is a separate research benchmark developed by the ORO team to investigate a more demanding type of decision-making.</p>
<p class="wp-block-paragraph">Instead of asking an agent to find a product or complete a single purchase, SMB Bench places it in charge of buying fresh produce for a wholesaler that supplies restaurants, catering businesses, and walk-in customers.</p>
<p class="wp-block-paragraph">Each morning, the agent reviews inventory records, sales figures, supplier prices, and delivery schedules. It must decide what to buy, how much to order, and which suppliers to use. It can also purchase emergency stock to cover shortages and keep notes to guide its decisions on subsequent days.</p>
<p class="wp-block-paragraph">The challenge is that the agent doesn’t know everything happening behind the scenes. If shelves are empty, lost sales aren’t recorded. If restaurants receive incomplete orders, they may quietly reduce future purchases. Suppliers can also begin delivering late without warning. The agent must recognize these patterns from the available records and adjust its decisions.</p>
<p class="wp-block-paragraph">ORO tested three conditions. In a normal quarter, there were no unusual consequences. In a second scenario, customers reduced their orders after receiving incomplete deliveries. In the third, customers behaved the same way while the cheapest supplier also became unreliable.</p>
<p class="wp-block-paragraph">Across <strong>485 runs covering nine models and 18 tasks</strong>, ORO measured how these decisions affected the business’s profitability. Each model was also compared against a fixed buying rule that simply looked at recent sales, considered delivery times, maintained extra stock, and purchased from the cheapest supplier.</p>
<h2 class="wp-block-heading">What the Nine AI Models Achieved</h2>
<p class="wp-block-paragraph">GPT-6 Astra emerged as the strongest model, generating an average simulated quarterly profit of $7,749. The fixed buying rule earned $6,412, outperforming all eight other models. Claude Fable 5.1 and GPT-6.1 Sol followed with $5,427 and $5,404, respectively.</p>
<p class="wp-block-paragraph">At the other end, four models lost money on average, with Muse Wonder 1.0 recording the largest loss at $8,291.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="606" src="https://taodaily.io/wp-content/uploads/2026/11/image-59-1024x606.png" alt="" class="wp-image-26198" srcset="https://taodaily.io/wp-content/uploads/2026/11/image-59-1024x606.png 1024w, https://taodaily.io/wp-content/uploads/2026/11/image-59-300x177.png 300w, https://taodaily.io/wp-content/uploads/2026/11/image-59-768x454.png 768w" sizes="(max-width: 1024px) 100vw, 1024px" /><figcaption class="wp-element-caption">Average simulated quarterly profit across all nine models and the fixed buying rule</figcaption></figure>
<p class="wp-block-paragraph">Importantly, Astra’s advantage did not come primarily from learning to adapt. In nearly every run, it began by instructing the business to cover customer shortages through emergency purchases, even when those purchases were expensive. That decision helped preserve customer orders and protect future revenue.</p>
<p class="wp-block-paragraph">The findings also showed that all nine models consistently bought less produce than the fixed rule, purchasing between 5% and 42% less over the quarter. They frequently recognized shortages but failed to increase their orders enough to prevent them from happening again.</p>
<h2 class="wp-block-heading">AI Could Recognize Problems Without Fixing Them</h2>
<p class="wp-block-paragraph">One of SMB Bench’s most important findings was the gap between recognizing a problem and actually doing something about it.</p>
<p class="wp-block-paragraph">In 323 runs where customers reduced purchases after receiving incomplete orders, the models explicitly connected declining demand to their own delivery failures in only 69 runs. Even among those 69 cases, they responded by protecting every affected customer within three days just 12 times.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="524" src="https://taodaily.io/wp-content/uploads/2026/11/image-60-1024x524.png" alt="" class="wp-image-26199" srcset="https://taodaily.io/wp-content/uploads/2026/11/image-60-1024x524.png 1024w, https://taodaily.io/wp-content/uploads/2026/11/image-60-300x153.png 300w, https://taodaily.io/wp-content/uploads/2026/11/image-60-767x392.png 767w" sizes="(max-width: 1024px) 100vw, 1024px" /><figcaption class="wp-element-caption">The gap between recognizing a business problem and changing purchasing decisions.</figcaption></figure>
<p class="wp-block-paragraph">ORO also compared runs where customer behavior changed with runs where it didn’t. Agents changed their approach in 20% of the runs with customer consequences, compared with 11% without them. This suggested that some models were responding to delayed feedback, but meaningful adaptation remained uncommon.</p>
<p class="wp-block-paragraph">There were encouraging exceptions. In one Gemini 4 Argon run, the model noticed that four customers receiving incomplete deliveries had reduced their orders by roughly 40–50%, while two customers receiving full deliveries maintained their demand. It identified the relationship and changed its purchasing approach to protect the remaining customers.</p>
<p class="wp-block-paragraph">That is precisely the behavior SMB Bench is designed to identify. An AI agent must do more than recognize a business problem. It needs to understand how its earlier actions contributed to it and make better decisions while there is still time.</p>
<h2 class="wp-block-heading">How far are we from a real expert?</h2>
<p class="wp-block-paragraph">To show how much money was left on the table, ORO added two more buyers. One is a “perfect” buyer that can see the future and re-plans every day, which sets the ceiling. The other is a scripted expert, built from rules ORO worked out with GetASAP and real wholesalers.</p>
<figure class="wp-block-image size-large is-resized"><img loading="lazy" decoding="async" width="1024" height="414" src="https: