leodb's Shortform

This is the kind of thing I don't trust LLMs about, too much hallucinating.

As an example, once an LLM insisted that "infinity plus one" is a finite number... because the sequence (infinity, infinity plus one) has a length two, and of course a finite sequence cannot have an infinite limit. I guess it's because it read many statements like this, but missed the detail that those were finite sequences of finite numbers, not finite sequences already starting at infinity.

So, in my opinions, the current LLMs are unable to do math, because math requires precision, and they are just putting together words that statistically go well together... but there is a huge difference between being 99% correct and 100% correct; one bad step invalidates the entire chain.

I suspect that something like this (words that sound correct, but are taken out of context which makes them completely invalid mathematically) is likely to be in the text you posted, too.

By the way, I am not saying that LLMs couldn't do this in principle, but their context probably needs to be much larger that it is typically today. Basically, a math textbook can define a thing in chapter 1, and then refer to it in chapter 15, by which point a typical LLM already forgot which definition was used in this specific book (did "finite sequence" refer to a sequence of finite length, or a sequence containing finite numbers, or both?).

I think the only reasonable way to do math with LLMs is to have them generate proofs that you afterwards check using some other system, such as Lean. Perhaps someone will build an AI that will do this automatically, i.e. will check its own reasoning in Lean before it tells you the conclusion.