This is a promising direction! Unfortunately, I think the benchmark result here is essentially meaningless.
I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mistake. The benchmark was saturated!
Now, looking at the result here, I see a similar pattern. Fable is not better than Opus, and the score is ~95%. Notably, this post omits which subagent is being used. Why? An intellectually honest way to tell if this thing really works would be to run that agent and report its score and cost as well.
Going back to my GPQA Diamond lesson, you can see here how a saturated leaderboard behaves https://artificialanalysis.ai/evaluations/gpqa-diamond. Fable gets 92.6% for $0.22 per task while several models score higher for $0.01. I could easily publish a router that “enhances Fable on GPQA Diamond” showing improved score for lower cost, just by implementing a router that picks the model at random!
The breakdown with which model, per-task cost, and methodology is in the "Full results and methodology" link in the post, not omitted. Definitely check it out if you haven't.
On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publish results on these hard benchmarks shortly). Apart from current user experiences, that will show the value of our harness.
I did read "Full results and methodology". It doesn't seem to show which agents you are routing to. Am I missing something? And how do those agents score on SWE-Bench Verified?
The benchmark wasn't evaluated with the router, we wanted an apples-to-apples comparison of our harness against other harnesses like mini-swe-agent on the same models to see if we added speed + cost value beyond routing.
Okay, that makes more sense, but then this post is very confusing. Why mention the router at all? Or does your live system have a router and the eval doesn't? In which case why not publish an eval with the router?
Fair point, yes, the router is only in the live system (we'll try to make that more clear). We thought that comparing the router to other harnesses on the benchmark wouldn't be a fair evaluation because the speed/cost gains from our harness would mostly be from using cheaper and simpler models rather than actually having a better harness.
The same thing holds for speed. I could build a system that speeds up Fable on GPQA Diamond ~50%, while improving score, by literally randomly selecting between Fable and Gemini 3.7 Flash. (Solve time for Flash is 0.1min and 0.8min for Fable, with Flash having a better score.)
Hell, I could publish better score at 87.5% time reduction by having the router always pick Flash!
Like I mentioned, we omitted routing from the benchmark and evaluated with our strongest model modality. Also, randomly selecting between Fable and Gemini 3.7 Flash wouldn't preserve quality.
Mentioned in my comment above. Sorry, I thought you were talking about Gemini 3.6 Flash, not their new one. Yes, that's the reason why we didn't include the router in the benchmark to have a fair comparison of the harness itself.
Hey just some feedback. I'm giving it a try on a new project I'm spinning up. It does seem faster although I haven't timed it against Claude or anything. I like the integrated browser to immediately see the changes (or see what the agent is doing). Thanks for having Linux support.
Thanks for trying it! If you want to time it against Claude/Codex in app, we have a race lab you can use for that. Good to hear that the Linux support and integrated browser help.
I use codex cli. It doens't have a UI for linux. I use opencode for that but it doens't work that well with the new 5.6 model suite from openai. Codex is both faster and better.
I would have loved if there was a way to use a GUI that worked like codex. Gave this a try. I think this might be it.
Please consider atleast adding MCP support if you can. That'll help
Can someone explain how this actually can go faster than other harnesses?
1. Model routing. Do you regret giving a task to Fable when it could have literally been done by Sonnet?
This is the one thing that I can understand.
2. Targeted code + context search. We think embedding the whole repo is dumb. We also think sticking the whole context (or compressed context) in chat is dumb. So we do faster and better greps over both.
Is this just a different system prompt? Why would other harnesses not be able to do it this way given instructions to do so?
3. Aggressive context hygiene. Tool output is bounded, stale screenshots disappear, we don’t re-read files…the garbage never floods the model.
Existing harnesses already don't re-read files that have not changed. And content that is already in the context window is cached tokens and doesn't add significant latency, what am I missing? Removing context from history invalidates KV cache.
4. Efficient turns. Batch independent investigation, make one surgical edit, then perform one focused verification. Internal measurement showed 16% fewer round trips and 27% lower cost.
Isn't this just a specific sub-agent strategy, also addressable with prompting?
I'm genuinely confused about what is mechanically different in this harness that could not be accomplished with a prompt/skill in another harness.
Thanks for bringing this up and trying it out! We've disabled Cmd+Option+I in the new version and also have a message during signup about sharing chats with Bullet, let us know if you run into any other issues!
OP was providing a userful tips to users, and they weren't giving you a bug report to remove it. If anything, they were suggesting you remove a login-wall.
But now that you blocked the inspector tool, you'll find it harder for users to report bugs. Unless of course those are automatically shared too? /s
We're actively working on implementing a guest mode feature right now for users that don't want to make an account. For reporting bugs, we have an in-app feedback form. Bugs are not automatically shared.
harnesses are becoming the new tech stack, I created my own and will compare it to how bullet operates, I’ve also built and deployed iOS and Android apps with my tool. When you have built something with the tool, other than the tool, it’s stronger proof.
Agree, building real apps with the harness is the strongest proof. Would love to hear where Bullet fell short vs your setup, especially on speed for your builds.
Congrats on the launch.
I can't find which model providers are supported?
Are you calling the claude-code CLI directly and make bullet usable with anthropic subscriptions like orca or herdr?
I guess however this is a harness and it needs to connect to the API?
I think this adds no value. I would stick to OpenCode.
Things I would value: high-fidelity visualization (bonus points for Figma-like visual edits), good tool use (don't force me to tell the model), token efficiency, resource efficiency. Things that are not problems:
* Routing: OpenCode predefines subagents that you can set to appropriate models.
* Search: there are AST and embedding-based search MCPs. I use https://github.com/DeusData/codebase-memory-mcp
Thanks for the feedback, we do focus on good tool management, context efficiency, etc. As for routing, ideally it isn't the user who has to route themselves, but a layer that routes for them.
Thanks for the link to the repo! We have looked at things like https://github.com/Graphify-Labs/graphify but haven't had a chance to rigorously evaluate their speed boosts on our harness.
No, we don’t support ACP today. Bullet uses its own internal harness and CLI integration. That said, we can definitely see it being something we work on in the future!
Are you freaking kidding me with YC throwing money at something like this? I guess I can fund raise just by having built https://maki.sh, and months ahead of other founders too...
I think that if there are many players in the space, it's a problem worth working on! Like I said in the post, our main problem is speed and that's what we're trying to fix. It's been a problem, and it still is a problem. We have tried tools like OpenCode, Pi, etc. but for our speed issue, it did not fix it. If we go down dying on this boat, so be it, at least we went down dying on a problem we care about!
Meanwhile, Cerebras running Sol at 750 tokens per second, the harness "speed" becomes irrelevant compared to making better use of results from insta-Sol.
Yes, we saw this and it's really exciting, can’t wait to try it in Bullet. But faster inference only speeds up generation. In our own experience, testing, building, searching, and other tool work still dominate plenty of real tasks. We’ll have to test it in practice, but it’s a great development for everyone building agents.
People were saying that about YC in 2009, when Airbnb and Stripe were funded. YC’s philosophy has always been to invest in a great many companies, accepting that you can’t know in advance which will succeed, most will fail, but the ones that succeed will be successful enough to pay for all the losses.
I recently switched over to using primarily Bullet for my projects and the speed of it makes it very nice to get projects off the ground quickly and work properly. I also found that Adi and Alex make updates very regularly based on some of the feedback i've submitted to their feedback tab. Good work on this product guys! I'm excited to see how it develops in the future.
I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mistake. The benchmark was saturated!
Now, looking at the result here, I see a similar pattern. Fable is not better than Opus, and the score is ~95%. Notably, this post omits which subagent is being used. Why? An intellectually honest way to tell if this thing really works would be to run that agent and report its score and cost as well.
Going back to my GPQA Diamond lesson, you can see here how a saturated leaderboard behaves https://artificialanalysis.ai/evaluations/gpqa-diamond. Fable gets 92.6% for $0.22 per task while several models score higher for $0.01. I could easily publish a router that “enhances Fable on GPQA Diamond” showing improved score for lower cost, just by implementing a router that picks the model at random!
On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publish results on these hard benchmarks shortly). Apart from current user experiences, that will show the value of our harness.
Hell, I could publish better score at 87.5% time reduction by having the router always pick Flash!
What do you mean it wouldn't preserve quality? I demonstrate in the comments here how it would on a saturated benchmark.
Please consider atleast adding MCP support if you can. That'll help
1. Model routing. Do you regret giving a task to Fable when it could have literally been done by Sonnet?
This is the one thing that I can understand.
2. Targeted code + context search. We think embedding the whole repo is dumb. We also think sticking the whole context (or compressed context) in chat is dumb. So we do faster and better greps over both.
Is this just a different system prompt? Why would other harnesses not be able to do it this way given instructions to do so?
3. Aggressive context hygiene. Tool output is bounded, stale screenshots disappear, we don’t re-read files…the garbage never floods the model.
Existing harnesses already don't re-read files that have not changed. And content that is already in the context window is cached tokens and doesn't add significant latency, what am I missing? Removing context from history invalidates KV cache.
4. Efficient turns. Batch independent investigation, make one surgical edit, then perform one focused verification. Internal measurement showed 16% fewer round trips and 27% lower cost.
Isn't this just a specific sub-agent strategy, also addressable with prompting?
I'm genuinely confused about what is mechanically different in this harness that could not be accomplished with a prompt/skill in another harness.
Cmd+Option+I > Console > 'allow pasting'
const onboarding = document.querySelector('#onboarding'); const app = document.querySelector('#app'); onboarding.style.setProperty('display', 'none', 'important'); app.inert = false; app.removeAttribute('aria-hidden'); document.querySelector('#prompt')?.focus();
Also, warning:
# Share chats with Bullet — helps us improve model routing and answer quality
Enabled by default.
Bulletproof.
OP was providing a userful tips to users, and they weren't giving you a bug report to remove it. If anything, they were suggesting you remove a login-wall.
But now that you blocked the inspector tool, you'll find it harder for users to report bugs. Unless of course those are automatically shared too? /s
I guess however this is a harness and it needs to connect to the API?
Is it even hidden when your AI ends up tagging it with `aria-label="Hidden secret code"`? Lol. Fun mini game.
Things I would value: high-fidelity visualization (bonus points for Figma-like visual edits), good tool use (don't force me to tell the model), token efficiency, resource efficiency. Things that are not problems:
* Routing: OpenCode predefines subagents that you can set to appropriate models. * Search: there are AST and embedding-based search MCPs. I use https://github.com/DeusData/codebase-memory-mcp
Thanks for the link to the repo! We have looked at things like https://github.com/Graphify-Labs/graphify but haven't had a chance to rigorously evaluate their speed boosts on our harness.
That should be an option, not forced.
If you’re up for giving it another shot, would love to hear other feedback on the app you have.
"The bullets" are not any different than other new agent projects. New agent projects pop up and die constantly and it's not from a lack of funding.
Dear YC, please force the startups to dedicate some of the 500k funding for standard web design.
Looks like "Techno music design of the 90s" or similar :-D