The performance costs of thinking and provider defaults
While building Tomo and several prototypes using LLMs, I've experimented with several popular language models. It's generally easier to prototype using the OpenAI chat responses API because most providers support this early API spec (mostly). This approach makes it pretty simple to switch between...
Model Context Protocol servers have become a popular way to expose software services for LLMs. Prior to this post, I'd not spent nearly enough time with them to have a strong opinion of their pros and cons, so I set out to change that.
It started because I was using the OpenAI completion API to try several different models while building Tomo.
I was working with Amazon Bedrock to run LLM inference. AWS has its fair share of complexity -- VPCs, subnets, security groups, etc.
I was reading your blog and had a question about this: "I noticed that my coworker was prompting for specific technical implementations, and Claude was struggling, pulling in too much context and taking an unfocused approach, whereas I would have been much more vague and general to start and...
If you've read any of my writing in the past year, you're probably aware I've heavily adopted agents to build much of the software I write now. What I've done less of is write about the strategies I've used to do this.
Who is finding LLMs useful and who is not? And why is this the case?
Lots of language model providers implement the OpenAI API spec. These look similar in shape but often behave differently in subtle ways. Anthropic's prefill sequences are one such example.
Today, I ran into an issue where I wanted to use repomix to pack a large codebase into a single file to pass to an LLM, but I couldn't paste the output into any of the UIs I typically use. The React apps all became sluggish as I waited for ~500,000 tokens to paste.
These days I use agents that write code often. When I am trying to build a new feature, I first write a markdown spec, then point the agent at it and send it on its way.