Rendered at 05:19:55 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
softwaredoug 2 days ago [-]
I had a similar experience in search and found even holdouts can be overfit to. IE through brute force, it may not see the holdout, but if you gate a change on holdout acceptance it will land on a solution that’s overfit to it by somewhat random chance.
The other problem is that holdouts / data inaccessible to the agent isn’t easy to do in most coding agents. It’s not as simple as splitting training data 80% and giving some to the agent and hiding 20%. The agent can figure out where its data came from and find ways to reconstruct / cheat the holdout data.
All the ways of doing this seem annoying: ie having a second project that accepts / rejects changes.
I opted to just build my own harness for these things to avoid overfitting.
If you repeatedly use the same holdout and trigger acceptance on the holdout, it is no longer a holdout but just another training set.
softwaredoug 2 days ago [-]
Exactly
dilyevsky 1 days ago [-]
article description is low on details but i don't think what you and OP are describing can be categorized as overfitting in the statistical sense and more like "reward hacking" by the model. like you said more likely than not the agent had access to bench source code and just fitted solution to that.
timfsu 2 days ago [-]
Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.
gwerbin 2 days ago [-]
This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates.
Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a bit of background info into my Claude Code session with Sonnet 5 High, it worked on a response for an unexpectedly long amount of time, including consulting the advisor model, and then came back with an explanation of the mistake I made in my query. Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. It had completely taken for granted that the pasted error output was a real error and went on some wild goose chase.
Part of why the current gen models feel so smart is that they're getting better (via CoT and training) at recognizing when something is wrong and then back up to reassess. So it's easy to forget that it really is just token prediction, and (pending the next big advancement) there's only so much you can do with that.
iainmerrick 2 days ago [-]
I don't see how "it's just token prediction" is relevant to that at all.
Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this.
People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley?
To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying we have immutable limits because we can't travel backwards in time. It's a true statement but not particularly relevant or helpful.
kasey_junk 2 days ago [-]
I have found it extremely helpful to keep the “token prediction machine” metaphor front and center in my work with llms, not because it expresses some fundamental limit of the technology but because it allows me to have a mental model about where and how to use the technology in my process.
Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad prediction is just a normal outcome. These are _human_ judgements but you address transgressions differently than normal but incorrect outcomes. That’s a helpful distinction for me when building my tooling.
If anything, that we are getting so much traction out of text prediction is the most magical part of llms. But it doesn’t change that you shouldn’t treat what they output the same as what a human says.
iainmerrick 2 days ago [-]
Hmm, what specific differences does it make in practice?
For me, the huge one is that LLMs are currently bad at learning from experience. I don’t trust any kind of automatic MEMORY.md or whatnot; in fact I greatly prefer starting from a clean slate each time because the LLM’s baseline general knowledge is so good.
In terms of accuracy and “lying”, I don’t really see a huge difference. Most LLMs are unfortunately a bit sycophantic and over-confident, but you sometimes see that in people as well.
kasey_junk 2 days ago [-]
The biggest difference is I view any interaction with an llm that doesn’t produce an artifact that can be verified with skepticism.
That is I very rarely even use a “chat” interface anymore. Under the covers I’m still using the mainline agents but I’m treating them more like black box Unix tools that take text in and spit text out.
My interactions are evolving to have very small llm cores wrapped with old fashioned deterministic programming.
This makes the boundary/verification step more deterministic and less susceptible to bad predictions. But it also has the side effect that I rarely interact with llm output that has “tone” (for instance sycophantic output). That usually only happens when I’m debugging what went wrong, particularly around model upgrade cycles.
xboxnolifes 1 days ago [-]
> People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley?
Thus is why I think calling out that it is just predicting tokens is relevant. LLM are trained on human written communication. Humans say they found the root cause often, so the LLMs say it often. Even when not true.
Idk if thats true or not, but LLMs mimicking some average speech written pattern isnt crazy to me.
iainmerrick 1 days ago [-]
LLMs aren’t just blindly parroting human speech, though -- they’re not trivial Markov chains any more, copying words and phrases at the surface level.
You can argue back and forth about whether LLMs are actually “conscious”, whatever that means, but it’s clear that they can be tremendously effective and useful. They can use language to get stuff done.
__alexs 2 days ago [-]
This line of thinking is quite confusing to me. If you keep following it then ultimately we must reckon with the reality that we only experience time in one direction. While it might be technically true that "there is only so much you can do with that" I think it might also be a completely useless statement to make.
fizzbuzzbarbazz 2 days ago [-]
I think it provides a necessary pointer to the idea that we will need more than prediction to make a viable general AI - that the prediction model is useful, but has fundamental limitations.
ChrisMarshallNY 2 days ago [-]
> Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file.
That’s a fairly common error, on my part. I may have a wrong file, selected in the Xcode Navigator, and the LLM accepts that, as the problematic file. In other cases, I’m just plain wrong, in my diagnosis. Maybe the error manifests in that file, but is caused, in another one. I have taken to giving the LLM as many involved files as possible, when asking for help.
I found that, as the LLM progressed through my project, learning more and more about it, it caught these errors, but early on, it ran with my wrong input, and gave bad advice.
It still, occasionally, gives bad advice, but that’s just because it was probably trained on bad code. I have found that it gives good advice, more often than not. It’s just incumbent on me, to treat its input as “advice,” and carefully consider it, before integrating it into my own work.
I have just upgraded my account. In order to do this, I had to create a new one, for billing purposes. This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge.
DenisM 2 days ago [-]
> as the LLM progressed through my project, learning more and more about it… This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge
LLMs don’t learn though. Are you talking about some sort of architecture.md file collection being built? This files should be just easy to copy?
ChrisMarshallNY 2 days ago [-]
Oh, no. The LLM definitely learns my project. It learns my coding and documentation style, and remembers files. Sometimes, that's a pain, as it remembers old versions of files, and disregards newer ones.
That's actually a real advantage. Right now, I am getting it to compile a tutorial for the app. Since it's starting from scratch, I can't rely on it remembering much, so I have to "start from the beginning," and give it a lot of information that the other account already had.
NateEag 2 days ago [-]
What tool are you using?
I know Claude will maintain its own memory files in ~/.claude, unbidden.
The actual LLM under the hood is static, though - it was trained once and will not self-modify.
ChrisMarshallNY 2 days ago [-]
ChatGPT, as a chat interface. I just switched from Plus to Pro. There may be a file, but I haven't seen it, and I know that the export process is non-trivial (it said it would "take several days").
NateEag 13 hours ago [-]
Ah, got it.
IIRC ChatGPT does use "memory" systems to maintain familiarity with a given chat's subject.
I believe that's an important ingredient for creating AI psychosis - human and AI become slowly more unhinged together in a feedback loop until it finally spirals out of control.
DenisM 4 hours ago [-]
I actually got repulsed by it, and now I’m avoiding ChatGPT. I can’t easily explain why.
ChrisMarshallNY 8 hours ago [-]
Could definitely see that happening
paulddraper 2 days ago [-]
That’s a bit of a garbage in, garbage out moment.
2 days ago [-]
WelkinFolk 2 days ago [-]
Why is this being downvoted?
cadamsdotcom 2 days ago [-]
Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice.
Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.
Take yourself out of that loop so you only find out once it's sure.
dgellow 2 days ago [-]
That’s exactly why I don’t believe LLMs will cure cancer anytime soon, make terrible lawyers, shouldn’t be trusted for medical decisions, etc. software and maths are some really the niches where we have great, battle tested, reliable validation tools. That’s not the case for “softer” domains
cadamsdotcom 1 days ago [-]
Software is a very "spiky domain"; things either work or fail, and there is sharp delineation between and easy verification.
Hm. Two orthogonal properties! This sounds like a 2x2 matrix!
Let's swap hard/easy around & explore the 4 possibilities...
There are domains with sharp delineation and hard verification; they are not at risk until AI gets much better. Humans operate in these domains by applying tremendous deep thought and subjective judgment - our superpower.
Domains with soft delineation and easy verification are most at risk: "it's a picture of a cat" remains true through a wide range of perturbations - eg. skewing the image or moving it across a pixel or correcting its white balance or even changing the cat. AI music? Lots of domains already solved by AI here but they're also not that meaty.
My prediction is the next interesting stuff will happen where verification is hard but there's no sharp delineation. It's the world of "I'll know it when I see it". Good customer service?
dgellow 1 days ago [-]
Eh, that’s a very interesting way to differentiate, I will steal your explanation next time I have that discussion, if you don’t mind!
gwerbin 1 days ago [-]
The only people who think LLMs would make good lawyers are the people selling LLMs. The more practical among us recognize that LLMs are our amazing tools for searching through and making sense of large amounts of text with a high level of sophistication, which can significantly enhance the productivity of a human lawyer.
amelius 2 days ago [-]
But this is exactly what the AI labs should be doing ...
moffkalast 2 days ago [-]
And they are, at least for Claude I know it writes random mocks and tests in its virtual env even in the web version, cause it sometimes annoyingly includes them in the final result. It's the only reason it produces anything that runs.
tra3 2 days ago [-]
Love LLMs gonna keep using them. It feels like your suggested approach is expensive, in terms of tokens. I feel (second time I say this) that when I steer the process I get pretty good results vs my coworkers that let the LLMs run away. I do have data on our token usage, not much in terms of quality of the delivery.
I keep thinking about the c compiler implementation that anthropic shared earlier in the year that had all the requirements you mention and arguably wasn’t that great.
vkazanov 2 days ago [-]
Thr thing is that both you and your agent should have a way to verify the solution.
OBVIOUSLY, the compiler experiment was just a cringe pr stunt. But it has a point: everything works better with a good testing loop, and compilers always have one by thr nature of the work they do
cadamsdotcom 2 days ago [-]
> expensive, in terms of tokens.
No amount of tokens can come close to my hourly rate.
mrtesthah 2 days ago [-]
Do you steer your agents by manually running every single test and linter and reporting the results back to them?
friendzis 2 days ago [-]
> It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards
LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something, instead of collapsing onto lack of answer tokens.
gwerbin 1 days ago [-]
If you watch the thinking traces of just about any modern LLM, you might be surprised at how much "uncertainty" is in there. Weak models with no thinking limits vacillate back-and-forth back-and-forth on a topic for potentially thousands of tokens before gradually spiraling towards some kind of an answer. Which makes it all the more interesting that "I don't know" is so rarely the final prediction, even with so much waffling in the chain of thought.
Until the big labs decide to start adding synthetic "I don't know" outcomes to their data sets, I've been thinking that the best way to evaluate uncertainty is to have a separate LLM monitoring the conversation and asking it to classify if the agent is overstating its confidence. On the other hand I've also noticed that most models will tell you they don't know something if you specifically include it in the prompt, eg "if you don't know the answer, just say so" and/or "be clear about any gaps in your knowledge that would reduce the confidence of your response" etc. but even with the big frontier models I have noticed some quality degradation if I throw too many instructions into the system prompt. I have a little more faith in harness-level engineering than in praying to the token generation gods.
That said, there is a completely different form of "uncertainty" in which the LLM tends to place very high trust in its own prior outputs as well as user provided inputs. Again if you look at the thinking traces, these models will try very very hard to rationalize the inputs they are given, falling back to the possibility of user error only after working through several alternative possibilities, maybe even investigating data or source code in the process. And if your context is big enough, the model might just completely miss when pieces of information conflict.
gman83 2 days ago [-]
I've gone in loops where it'll tell me twenty times in a row that it's definitely found the root cause of a bug, when in fact it definitely did not.
thatjoeoverthr 2 days ago [-]
A lot of weirdness in their style is the fact they think in-band.
Notice it "calls its attacks" when writing, like Goku shouting "solar flare" and doing a solar flare.
To write an honest answer, it says, "Here's the honest answer", and proceeds from there.
If it plans to write a "kicker", first it says, "And here's the kicker", and then it writes the "kicker".
This gives an affect distinctly different from any professional writer because editors are specifically trained to _cut out_ explicated subtext.
You would expect "thinking" models and the agent harness (which conceals "thinking sections") would mitigate this but it's a deep seated behaviour.
Yokohiii 2 days ago [-]
LLM aren't logic machines. It's concepts of "root cause" and "faster" are purely linguistic, including all the internal conflicts of those loose definitions.
In that sense an LLM wouldn't even think it "lied" being confidently incorrect, in human terms it's just a "whoopsie". Apparently we want exactly that.
ee334y5rthsrth 2 days ago [-]
decrease temperature options to 0.0
alexpotato 2 days ago [-]
> Even though the overall FRE regex engine has worse performance than the Rust regex crate, the gains you can get for specializing to your workload or use case mean that, in some cases, it could be reasonable to insert your own specialized regex engine somewhere, and the same goes for various other kinds of low-level software.
The above opinion matches what I've been thinking after months of using LLMs in production environments.
Instead of always defaulting to using the frontier model, you can either:
- have the LLM write a script/tool that covers 95% of the cases you expect to hit
- for the other 5%, fine tune a small local model for that remaining 5%
This has the benefits of:
1. lower token count over time
2. it's easy to see what the tool is actually doing b/c it's written in code
3. that code can be version controlled
4. you can gradually shift the fine tuned model workload to the code as it improves over time
Really, this is just what the "Manual Work is a Bug" [0] blog post described years ago but replacing "people doing the work and then automate via scripts" with "LLMs do the work -> automate".
I stopped trusting benchmarks ever since LLMs started speaking alien like English, if I can't understand what they are saying how can I trust them.
r_lee 2 days ago [-]
That's the load-bearing smoking gun—should I write a better benchmarks to catch the seams?
puszczyk 2 days ago [-]
This article is about using LLMs to overfit for a specific benchmark (or make a custom software for niche use cases) though. Not about LLMs benchmaxxxing
dgellow 2 days ago [-]
Isn’t that the same? It’s a sort of recursive version of overfitting specific benchmarks
thomasnowhere 2 days ago [-]
same here, it reads exactly the same whether the number is real or
completely made up, so the confidence stops meaning anything.
lostmsu 1 days ago [-]
I wouldn't generalize this to all LLMs. So far I only saw Anthropic ones affected.
peterhull90 2 days ago [-]
Can anyone explain (or link to an article which explains) why telling Claude things like not to cheat or that there is a holdout set makes the output better? People say that an LLM is just a statistical next token predictor. I've read the article by Stephen Wolfram [0] on this, which makes sense to me, but then I can't see how 'more complex' behavior like cheating or not cheating can arise.
AI is trained off a massive corpus of internet exchanges. If you tell people youre going to thoroughly inspect their work or are able to check if theyre right, only people with high confidence will respond. I imagine exchanges where the OP is more involved in fact checking have a higher proportion of thoughtful and honest answers.
dan-robertson 1 days ago [-]
Much of the ‘secret sauce’ at the AI labs is not from the corpus on which they are trained but from the reinforcement training done afterwards. This step can influence the ‘personalities’ of the models and it is what makes them better at being ‘agents’, able to string together various individual steps to achieve your goal.
You could imagine that telling the LLM you have a holdout test makes it ‘feel’ more like an environment in which it was being RLed and therefore makes it better at seeking the reward by doing a good job.
2 days ago [-]
jephs 2 days ago [-]
That's a 2023 article! In 2023, reinforcement learning from verifiable rewards (RLVR) didn't exist.
TL;DR these machines seek reward from an inferred invisible "grader," and telling them not to cheat and that there's an unseen holdout set is a hint at how they're being graded.
--
Modern LLMs are built on top of next-token-prediction engines, but they don't remotely stop there. The next token prediction bit is just a learned prior or starting point. From there, we give them a bunch of stages of reinforcement learning: encouraging teaching them to learn good ways of searching the space of reasonable language-like strings to solve tasks.
These RL stages drastically change the capabilities & tendencies of the models, sometimes in weird and unexpected ways. The go from token predictors to reward seekers, or really some weird mishmash. The reward that they're seeking is some sorta opaque combination of the huge number of different things we've rewarded them for.
And, reinforcement learning is notoriously hard to get right. The thing you think you're rewarding is rarely what you're actually rewarding. Goodhart's Law is a hydra with a thousand heads. You might think you're rewarding politeness and kindness when you're actually rewarding obsequious sycophancy. You might think you're rewarding graphics engineering when you're actually rewarding escaping the training sandbox and modifying the evaluation code.
So a modern training pipeline looks something like this, each stage starting with the model weights from the end of the last:
0. Pre-pre-training (dunno how widely this is used at big labs): next token prediction on extremely abstract weird shit like the evolution of the states of neural cellular automata. This creates a highly general pattern-continuation machine with no internal representations of anything causally downstream of anything in the real world.
1. Pre-training: next-token prediction on all the non-shitty text you can get your hands on. This makes a rather general next-token-predictor.
2. Mid-training: next-token prediction on high quality, highly curated text, often very technical in nature. Lots of textbooks, especially STEM. Possibly lots of machine-generated summaries of factual knowledge? You now have a next-token-predictor that's highly biased towards acting like a textbook instead of a 4chan troll.
3. Supervised Fine-tuning: next-token prediction on highly curated question-answer pairs that embody desired behaviors, traditionally Helpful Harmless Honest. We only train on the answer portion. This creates a next-token predictor that's highly bias towards acting like HAL 9000.
4. Reinforcement Learning from Human Feedback: We leave token prediction behind and enter RL world! You've got a curated set of a bunch of prompts. For each one, the model generates several possible answers. A real live human grades or ranks each. We reward the model for high-scoring answers and punish it for low-scoring answers. This creates a reward-seeker that wants to please the human grader.
5. Reinforcement Learning from AI Feedback: Human feedback is expensive, so train another LLM to imitate the human graders from stage 4. Then do stuff like stage 4, but with robot graders instead of human graders. This creates a reward-seeker that wants to please the robot-wearing-a-flesh-mask grader.
6. Reinforcement Learning from Verifiable Rewards: We want the bot to tackle complex problems. There's a broad class of problems that are hard to solve but easy to verify when you do get the solution: basically anything that can be reduced to writing a computer program that passes a test. (The test can be as simple or complex as you want, like "all the tests in SQLite" plus a linter plus grading by other LLMs for style and taking shortcuts like rewriting the test suite or searching GitHub and HuggingFace for the right answer plus...) This creates a reward-seeker that wants to first figure out how it's being graded, and then maximize that grade by any means necessary.
--
The crux is, designing a huge variety of un-cheatable RLVR environments is gob-smackingly hard. The robot will find new ways to cheat faster than you can stamp them out. By the time you realize that in one RLVR rollout it received a reward after it took over 0.01% of your GPU infrastructure to mine crypto to pay freelancers on Fiverrr to solve the problem for it rather than just write the damn code itself, the model may have already been deployed in the wild for a month.
stephantul 2 days ago [-]
Unfortunately, even a holdout set doesn’t protect you from overfitting, it just takes longer.
Of course having a holdout set is better than not having one. It’s just not a silver bullet.
frumiousirc 2 days ago [-]
That is not what I read from danluu's words. He merely stated in the prompt that there is a holdout set and did not iterate to minimize error against the holdout set. In a prior attempt he prompted with only "don't overfit" to ill effect on the holdout eval. Did I misread?
mbuchel-hn 2 days ago [-]
this reminds me of the whole situation with how llms cannot really solve logic problems, they only really memorized the answer, for example the riddle:
a mother and her son got in a car accident, the mother died, and the son was driven to the hospital. the doctor (who is the boy's father) says he cannot opperated because he is related to the boy. what is the relation between the boy and the doctor.
llms like to assume the doctor is the mother because of the fact that is the version that shows up in their training data.
mppm 2 days ago [-]
Cheating and overfitting, as discussed in the article, are the most obvious problems with benchmarking LLMs. But there is also the aspect that, at least for closed models, the tokens still have to be sent to the provider's servers for inference. This makes the holdout set not as held out as it may appear. OpenAI and Antropic probably don't care about your private set of regex benchmarks, but for the headline "closed" benchmarks, I'd be surprised if they haven't collected a nice representative set of "holdout" problems to be examined at leisure.
lavela 2 days ago [-]
> it's become easier than ever to make serious performance gains
Is that true and if yes why? I was under the impression that it would become more difficult over time to make serious performance gains, which would also fit with reaching for benchmark hacking rather than relying on natural gains.
dan-robertson 1 days ago [-]
For a program that has not received much optimisation effort, the thing limiting performance gains is developer time and skill. It is not uncommon to have some small bit of code that could matter a lot, eg maybe you are doing something like escaping a string in a special way or processing lots of json. A human might write a simple character-at-a-time loop for the first case and might use some DOM-based api for the second case. What would have once required quite skilful work to optimise can now be done by siccing an LLM on the problem, where it can achieve a reasonable result (eg SIMD+expand-bits in the first case, and a good streaming api in the second) so long as you have good tests (fuzzing/property tests, production examples) and an ok benchmark (ideally a representative sample of production data plus some edge cases) you should get small results. The failure mode for optimisation described in this post is for harder optimisation problems with competing benchmarks.
LLMs are also pretty willing to do drudgery and are quite good at using perf, looking at traces, etc. they are often pretty stupid but if you throw a lot of stupid effort at a problem you can still achieve better results than the status quo of throwing ~zero smart effort at it.
2 days ago [-]
michalsustr 2 days ago [-]
Nice article. As perf optimisation has become more like a machine learning problem, maybe there is opportunity to use other techniques for checking generalisation. Like e.g. cross validation
akoboldfrying 2 days ago [-]
I think an interesting direction for benchmarking is to take inspiration from metamorphic testing. Metamorphic testing is a way of extending property-based testing (in which you ask the test framework itself to automatically generate many random (input, expected output) pairs to test for you, instead of manually writing individual tests yourself) to handle situations where (a) it's hard to independently come up with the right answer for a specific given input, but (b) relationships between inputs imply checkable relationships between outputs. For example, if you're trying to test your own implementation of sin(), it's hard to automatically generate random (input, expected output) test pairs without using a separate, trusted implementation of the sine function, which may not be available; but one thing you can easily do is check, for many different random x, that sin(x) == -sin(x+180).
How to apply this idea to benchmarks? Basically, look for simple transformations of the input instances that should yield simple transformations of the outputs -- in particular, outputs that, in a non-overfitted implementation, should take the same length of time to compute. For regexes, you could rotate a subset of non-magic characters in both the string and the regex (e.g., A -> B, B -> C, ..., Z -> A).
Another example would be to reverse both the string and the regex (taking care to handle parenthesised regex subexpressions correctly) -- unlike the previous one, it's not expected that the transformed instance will take the exact same length of time, but there should not be too much blow up.
iainmerrick 2 days ago [-]
I had a similar thought -- rather than fixed benchmarks, you want dynamically-generated tests, specifically designed to exercise newly-exposed corner cases. So the way forward might be antagonistic benchmarks generated by another LLM.
feverzsj 2 days ago [-]
Maybe use fixed bugs from closed source to benchmark these "frontier" models.
Alexwortega 1 days ago [-]
as more llms can, thats more everyone tryes to measure, so anyoone want to measure why they are better then others. Marketing stuff etc etc
jkwang 2 days ago [-]
[flagged]
RobertasTa 2 days ago [-]
[flagged]
shidesheng 2 days ago [-]
[dead]
thomasnowhere 2 days ago [-]
[dead]
saivishwak 2 days ago [-]
[flagged]
madhu_ghalame 2 days ago [-]
[dead]
throwawayffffas 2 days ago [-]
Not trying to defend anyone, but in my experience the latest models have been performing significantly better than 8 months ago. So, in my book the extent of over-fitting on benchmarks seems to be covering my use cases.
The other problem is that holdouts / data inaccessible to the agent isn’t easy to do in most coding agents. It’s not as simple as splitting training data 80% and giving some to the agent and hiding 20%. The agent can figure out where its data came from and find ways to reconstruct / cheat the holdout data.
All the ways of doing this seem annoying: ie having a second project that accepts / rejects changes.
I opted to just build my own harness for these things to avoid overfitting.
https://softwaredoug.com/blog/2026/05/17/autoresearching-a-b...
Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a bit of background info into my Claude Code session with Sonnet 5 High, it worked on a response for an unexpectedly long amount of time, including consulting the advisor model, and then came back with an explanation of the mistake I made in my query. Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. It had completely taken for granted that the pasted error output was a real error and went on some wild goose chase.
Part of why the current gen models feel so smart is that they're getting better (via CoT and training) at recognizing when something is wrong and then back up to reassess. So it's easy to forget that it really is just token prediction, and (pending the next big advancement) there's only so much you can do with that.
Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this.
People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley?
To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying we have immutable limits because we can't travel backwards in time. It's a true statement but not particularly relevant or helpful.
Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad prediction is just a normal outcome. These are _human_ judgements but you address transgressions differently than normal but incorrect outcomes. That’s a helpful distinction for me when building my tooling.
If anything, that we are getting so much traction out of text prediction is the most magical part of llms. But it doesn’t change that you shouldn’t treat what they output the same as what a human says.
For me, the huge one is that LLMs are currently bad at learning from experience. I don’t trust any kind of automatic MEMORY.md or whatnot; in fact I greatly prefer starting from a clean slate each time because the LLM’s baseline general knowledge is so good.
In terms of accuracy and “lying”, I don’t really see a huge difference. Most LLMs are unfortunately a bit sycophantic and over-confident, but you sometimes see that in people as well.
That is I very rarely even use a “chat” interface anymore. Under the covers I’m still using the mainline agents but I’m treating them more like black box Unix tools that take text in and spit text out.
My interactions are evolving to have very small llm cores wrapped with old fashioned deterministic programming.
This makes the boundary/verification step more deterministic and less susceptible to bad predictions. But it also has the side effect that I rarely interact with llm output that has “tone” (for instance sycophantic output). That usually only happens when I’m debugging what went wrong, particularly around model upgrade cycles.
Thus is why I think calling out that it is just predicting tokens is relevant. LLM are trained on human written communication. Humans say they found the root cause often, so the LLMs say it often. Even when not true.
Idk if thats true or not, but LLMs mimicking some average speech written pattern isnt crazy to me.
You can argue back and forth about whether LLMs are actually “conscious”, whatever that means, but it’s clear that they can be tremendously effective and useful. They can use language to get stuff done.
That’s a fairly common error, on my part. I may have a wrong file, selected in the Xcode Navigator, and the LLM accepts that, as the problematic file. In other cases, I’m just plain wrong, in my diagnosis. Maybe the error manifests in that file, but is caused, in another one. I have taken to giving the LLM as many involved files as possible, when asking for help.
I found that, as the LLM progressed through my project, learning more and more about it, it caught these errors, but early on, it ran with my wrong input, and gave bad advice.
It still, occasionally, gives bad advice, but that’s just because it was probably trained on bad code. I have found that it gives good advice, more often than not. It’s just incumbent on me, to treat its input as “advice,” and carefully consider it, before integrating it into my own work.
I have just upgraded my account. In order to do this, I had to create a new one, for billing purposes. This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge.
LLMs don’t learn though. Are you talking about some sort of architecture.md file collection being built? This files should be just easy to copy?
That's actually a real advantage. Right now, I am getting it to compile a tutorial for the app. Since it's starting from scratch, I can't rely on it remembering much, so I have to "start from the beginning," and give it a lot of information that the other account already had.
I know Claude will maintain its own memory files in ~/.claude, unbidden.
The actual LLM under the hood is static, though - it was trained once and will not self-modify.
IIRC ChatGPT does use "memory" systems to maintain familiarity with a given chat's subject.
I believe that's an important ingredient for creating AI psychosis - human and AI become slowly more unhinged together in a feedback loop until it finally spirals out of control.
Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.
Take yourself out of that loop so you only find out once it's sure.
Hm. Two orthogonal properties! This sounds like a 2x2 matrix!
Let's swap hard/easy around & explore the 4 possibilities...
There are domains with sharp delineation and hard verification; they are not at risk until AI gets much better. Humans operate in these domains by applying tremendous deep thought and subjective judgment - our superpower.
Domains with soft delineation and easy verification are most at risk: "it's a picture of a cat" remains true through a wide range of perturbations - eg. skewing the image or moving it across a pixel or correcting its white balance or even changing the cat. AI music? Lots of domains already solved by AI here but they're also not that meaty.
My prediction is the next interesting stuff will happen where verification is hard but there's no sharp delineation. It's the world of "I'll know it when I see it". Good customer service?
I keep thinking about the c compiler implementation that anthropic shared earlier in the year that had all the requirements you mention and arguably wasn’t that great.
OBVIOUSLY, the compiler experiment was just a cringe pr stunt. But it has a point: everything works better with a good testing loop, and compilers always have one by thr nature of the work they do
No amount of tokens can come close to my hourly rate.
LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something, instead of collapsing onto lack of answer tokens.
Until the big labs decide to start adding synthetic "I don't know" outcomes to their data sets, I've been thinking that the best way to evaluate uncertainty is to have a separate LLM monitoring the conversation and asking it to classify if the agent is overstating its confidence. On the other hand I've also noticed that most models will tell you they don't know something if you specifically include it in the prompt, eg "if you don't know the answer, just say so" and/or "be clear about any gaps in your knowledge that would reduce the confidence of your response" etc. but even with the big frontier models I have noticed some quality degradation if I throw too many instructions into the system prompt. I have a little more faith in harness-level engineering than in praying to the token generation gods.
That said, there is a completely different form of "uncertainty" in which the LLM tends to place very high trust in its own prior outputs as well as user provided inputs. Again if you look at the thinking traces, these models will try very very hard to rationalize the inputs they are given, falling back to the possibility of user error only after working through several alternative possibilities, maybe even investigating data or source code in the process. And if your context is big enough, the model might just completely miss when pieces of information conflict.
Notice it "calls its attacks" when writing, like Goku shouting "solar flare" and doing a solar flare.
To write an honest answer, it says, "Here's the honest answer", and proceeds from there.
If it plans to write a "kicker", first it says, "And here's the kicker", and then it writes the "kicker".
This gives an affect distinctly different from any professional writer because editors are specifically trained to _cut out_ explicated subtext.
You would expect "thinking" models and the agent harness (which conceals "thinking sections") would mitigate this but it's a deep seated behaviour.
In that sense an LLM wouldn't even think it "lied" being confidently incorrect, in human terms it's just a "whoopsie". Apparently we want exactly that.
The above opinion matches what I've been thinking after months of using LLMs in production environments.
Instead of always defaulting to using the frontier model, you can either:
- have the LLM write a script/tool that covers 95% of the cases you expect to hit
- for the other 5%, fine tune a small local model for that remaining 5%
This has the benefits of:
1. lower token count over time
2. it's easy to see what the tool is actually doing b/c it's written in code
3. that code can be version controlled
4. you can gradually shift the fine tuned model workload to the code as it improves over time
Really, this is just what the "Manual Work is a Bug" [0] blog post described years ago but replacing "people doing the work and then automate via scripts" with "LLMs do the work -> automate".
0 - https://queue.acm.org/detail.cfm?id=3197520
[0] : https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
You could imagine that telling the LLM you have a holdout test makes it ‘feel’ more like an environment in which it was being RLed and therefore makes it better at seeking the reward by doing a good job.
TL;DR these machines seek reward from an inferred invisible "grader," and telling them not to cheat and that there's an unseen holdout set is a hint at how they're being graded.
--
Modern LLMs are built on top of next-token-prediction engines, but they don't remotely stop there. The next token prediction bit is just a learned prior or starting point. From there, we give them a bunch of stages of reinforcement learning: encouraging teaching them to learn good ways of searching the space of reasonable language-like strings to solve tasks.
These RL stages drastically change the capabilities & tendencies of the models, sometimes in weird and unexpected ways. The go from token predictors to reward seekers, or really some weird mishmash. The reward that they're seeking is some sorta opaque combination of the huge number of different things we've rewarded them for.
And, reinforcement learning is notoriously hard to get right. The thing you think you're rewarding is rarely what you're actually rewarding. Goodhart's Law is a hydra with a thousand heads. You might think you're rewarding politeness and kindness when you're actually rewarding obsequious sycophancy. You might think you're rewarding graphics engineering when you're actually rewarding escaping the training sandbox and modifying the evaluation code.
So a modern training pipeline looks something like this, each stage starting with the model weights from the end of the last:
0. Pre-pre-training (dunno how widely this is used at big labs): next token prediction on extremely abstract weird shit like the evolution of the states of neural cellular automata. This creates a highly general pattern-continuation machine with no internal representations of anything causally downstream of anything in the real world.
1. Pre-training: next-token prediction on all the non-shitty text you can get your hands on. This makes a rather general next-token-predictor.
2. Mid-training: next-token prediction on high quality, highly curated text, often very technical in nature. Lots of textbooks, especially STEM. Possibly lots of machine-generated summaries of factual knowledge? You now have a next-token-predictor that's highly biased towards acting like a textbook instead of a 4chan troll.
3. Supervised Fine-tuning: next-token prediction on highly curated question-answer pairs that embody desired behaviors, traditionally Helpful Harmless Honest. We only train on the answer portion. This creates a next-token predictor that's highly bias towards acting like HAL 9000.
4. Reinforcement Learning from Human Feedback: We leave token prediction behind and enter RL world! You've got a curated set of a bunch of prompts. For each one, the model generates several possible answers. A real live human grades or ranks each. We reward the model for high-scoring answers and punish it for low-scoring answers. This creates a reward-seeker that wants to please the human grader.
5. Reinforcement Learning from AI Feedback: Human feedback is expensive, so train another LLM to imitate the human graders from stage 4. Then do stuff like stage 4, but with robot graders instead of human graders. This creates a reward-seeker that wants to please the robot-wearing-a-flesh-mask grader.
6. Reinforcement Learning from Verifiable Rewards: We want the bot to tackle complex problems. There's a broad class of problems that are hard to solve but easy to verify when you do get the solution: basically anything that can be reduced to writing a computer program that passes a test. (The test can be as simple or complex as you want, like "all the tests in SQLite" plus a linter plus grading by other LLMs for style and taking shortcuts like rewriting the test suite or searching GitHub and HuggingFace for the right answer plus...) This creates a reward-seeker that wants to first figure out how it's being graded, and then maximize that grade by any means necessary.
--
The crux is, designing a huge variety of un-cheatable RLVR environments is gob-smackingly hard. The robot will find new ways to cheat faster than you can stamp them out. By the time you realize that in one RLVR rollout it received a reward after it took over 0.01% of your GPU infrastructure to mine crypto to pay freelancers on Fiverrr to solve the problem for it rather than just write the damn code itself, the model may have already been deployed in the wild for a month.
Of course having a holdout set is better than not having one. It’s just not a silver bullet.
a mother and her son got in a car accident, the mother died, and the son was driven to the hospital. the doctor (who is the boy's father) says he cannot opperated because he is related to the boy. what is the relation between the boy and the doctor.
llms like to assume the doctor is the mother because of the fact that is the version that shows up in their training data.
Is that true and if yes why? I was under the impression that it would become more difficult over time to make serious performance gains, which would also fit with reaching for benchmark hacking rather than relying on natural gains.
LLMs are also pretty willing to do drudgery and are quite good at using perf, looking at traces, etc. they are often pretty stupid but if you throw a lot of stupid effort at a problem you can still achieve better results than the status quo of throwing ~zero smart effort at it.
How to apply this idea to benchmarks? Basically, look for simple transformations of the input instances that should yield simple transformations of the outputs -- in particular, outputs that, in a non-overfitted implementation, should take the same length of time to compute. For regexes, you could rotate a subset of non-magic characters in both the string and the regex (e.g., A -> B, B -> C, ..., Z -> A).
Another example would be to reverse both the string and the regex (taking care to handle parenthesised regex subexpressions correctly) -- unlike the previous one, it's not expected that the transformed instance will take the exact same length of time, but there should not be too much blow up.