Posts
Comments
That's some significant progress, but I don't think will lead to TAI.
However there is a realistic best case scenario where LLM/Transformer stop just before and can give useful lessons and capabilities.
I would really like to see such an LLM system get as good as a top human team at security, so it could then be used to inspect and hopefully fix masses of security vulnerabilities. Note that could give a false sense of security, unknown unknown type situation where it would't find a totally new type of attack, say a combined SW/HW attack like Rowhammer/Meltdown but more creative. A superintelligence not based on LLM could however.
Anyone want to guess how capable Claude system level 2 will be when it is polished? I expect better than o3 by a small amt.
Yes the human brain was built using evolution, I have no disagreement that give us 100-1000 years with just tinkering etc we would likely get AGI. Its just that in our specific case we have bio to copy and it will get us there much faster.
Types of takeoff
When I first heard and thought about AI takeoff I found the argument convincing that as soon as an AI passed IQ 100, takeoff would become hyper exponentially fast. Progress would speed up, which would then compound on itself etc. However there other possibilities.
AGI is a barrier that requires >200 IQ to pass unless we copy biology?
Progress could be discontinuous, there could be IQ thresholds required to unlock better methods or architectures. Say we fixed our current compute capability, and with fixed human intelligence we may not be able to figure out the formula for AGI, in a similar way that the combined human intelligence hasn't cracked many hard problems even with decades and the worlds smartest minds working on them (maths problems, Quantum gravity...). This may seem unlikely for AI, but to illustrate the principle, say we only allowed IQ<90 people to work on AI. Progress would stall. So IQ <90 software developers couldn't unlock IQ>90 AI. Can IQ 160 developers with our current compute hardware unlock >160 AI?
To me the reason we don't have AI now is that the architecture is very data inefficient and worse at generalization than say the mammalian brain, for example a cortical column. I expect that if we knew the neural code and could copy it, then we would get at least to very high human intelligence quickly as we have the compute.
From watching AI over my career it seems to be that even the highest IQ people and groups cant make progress by themselves without data, compute and biology to copy for guidance, in contrast to other fields. For example Einstein predicted gravitational waves long before they where discovered, but Turing or Von Neumann didn't publish the Transformer architecture or suggest backpropagation. If we did not have access to neural tissue, would we still not have artificial NN? In a related note, I think there is an XKCD cartoon that says something like the brain has to be so complex that it cannot understand itself.
(I believe now that progress in theoretical physics and pure maths is slowing to a stall as further progress requires intellectual capacity beyond the combined ability of humanity. Without AI there will be no major advances in physics anymore even with ~100 years spent on it.)
After AGI is there another threshold?
Lets say we do copy biology/solve AGI and with our current hardware can get >10,000 AGI agents with >= IQ of the smartest humans. They then optimize the code so there is 100K agents with the same resources. but then optimization stalls. The AI wouldn't know if it was because it had optimized as much as possible, or because it lacked the ability to find a better optimization.
Does our current system scale to AGI with 1GW/1 million GPU?
Lets say we don't copy biology, but scaling our current systems to 1GW/1 million GPU and optimizing for a few years gets us to IQ 160 at all tasks. We would have an inferior architecture compensated by a massive increase in energy/FLOPS as compared to the human brain. Progress could theoretically stall at upper level human IQ for a time rather then takeoff. (I think this isn't very likely however) There would of course be a significant overhang where capabilities would increase suddenly when the better architecture was found and applied to the data center hosting the AI.
Related note - why 1GW data centers won't be a consistent requirement for AI leadership.
Based on this, then a 1GW or similar data center isn't useful or necessary for long. If it doesn't give a significant increase in capabilities, then it won't be cost effective. If it does, then it would optimize itself so that such power isn't needed anymore. Only in a small range of capability increase does it actually stay around.
To me its not clear the merits of the Pause movement and training compute caps. Someone here made the case that compute caps could actually speed up AGI as people would then pay more attention to finding better architectures rather than throwing resources into scaling existing inferior ones. However all things considered I can see a lot of downsides from large data centers and little upside. I see a specific possibility where they are build, don't give the economic justification, decrease in value a lot, then are sold to owners that are not into cutting edge AI. Then when the more efficient architecture is discovered, they are suddenly very powerful without preparation. Worldwide caps on total GPU production would also help reduce similar overhang possibilities.
I am also not impressed with the pause AI movement and am concerned about AI safety. To me focusing on AI companies and training FLOPS is not the best way to do things. Caps on data center sizes and worldwide GPU production caps would make more sense to me. Pausing software but not hardware gives more time for alignment but makes a worse hardware overhang. I don't think thats helpful. Also they focus too much on OpenAI from what I've seen. xAI will soon have the largest training center for a start.
I don't think this is right or workable https://pauseai.info/proposal - figure out how biological intelligence learns and you don't need a large training run. There's no guarantee at all that a pause at this stage can help align super AI. I think we need greater capabilities to know what we are dealing with. Even with a 50 year pause to study GPT4 type models I wouldn't be confident we could learn enough from that. They have no realistic way to lift the pause, so its a desire to stop AI indefinitely.
"There will come a point where potentially superintelligent AI models can be trained for a few thousand dollars or less, perhaps even on consumer hardware. We need to be prepared for this."
You can't prepare for this without first having superintelligent models running on the most capable facilities then having already gone through a positive Singularity. They have no workable plan for achieving a positive Singularity, just try to stop and hope.
OK fair point. If we are going to use analogies, then my point #2 about a specific neural code shows our different positions I think.
Lets say we are trying to get a simple aircraft of the ground and we have detailed instructions for a large passenger jet. Our problem is that the metal is too weak and cannot be used to make wings, engines etc. In that case detailed plans for aircraft are no use, a single minded focus on getting better metal is what its all about. To me the neural code is like the metal and all the neuroscience is like the plane schematics. Note that I am wary of analogies - you obviously don't see things like that or you wouldn't have the position you do. Analogies can explain, but rarely persuade.
A more single minded focus on the neural code would be trying to watch neural connections form in real time while learning is happening. Fixed connectome scans of say mice can somewhat help with that, more direct control of dishbrain, watching the zebra fish brain would all count, however the details of neural biology that are specific to higher mammals would be ignored.
Its possible also that there is a hybrid process, that is the AI looks at all the ideas in the literature then suggests bio experiments to get things over the line.
Yes you have a point.
I believe that building massive data centers are the biggest risk atm and in the near future. I don't think open AI/Anthropic will get to AGI, but rather someone copying biology will. In that case probably the bigger the datacenter around when that happens, the bigger the risk. For example a 1million GPU with current tech doesn't get super AI, but when we figure out the architecture, it suddenly becomes much more capable and dangerous. That is from IQ 100 up to 300 with a large overhang. If the data center was smaller, then the overhang is smaller. The scenario I have in mind is someone figures AGI out, then one way or another the secret gets adopted suddenly by the large data center.
For that reason I believe focus on FLOPS for training runs is misguided, its hardware concentration and yearly worldwide HW production capacity that is more important.
Perhaps LLM will help with that. The reason I think that is less likely is
- Deep mind etc is already heavily across biology from what I gather from interview with Demis. If the knowledge was there already there's a good chance they would have found it
- Its something specific we are after, not many small improvements, i.e. the neural code. Specifically back propagation is not how neurons learn. I'm pretty sure how they actually do is not in the literature. Attempts have been made such as the forward-forward algorithm by Hinton, but that didn't come to anything as far as i can tell. I havn't seen any suggestion that even with too much detail on biology we know what it is. i.e. can a very detailed neural sim with extreme processing power learn as data efficiently as biology?
- If progress must come from a large jump rather than small steps, then LLM have quite a long way to go, i.e. LLM need to speed up coming up ideas as novel as the forward-forward algo to help much. If they are still below that threshold in 2026 then those possible insights are still almost entirely done by people.
- Even the smartest minds in the past have been beaten by copying biology in AI. The idea for neural nets came from copying biology. (Though the transformer arch and back prop didn't)
I think it is clear that if say you had a complete connectome scan and knew everything about how a chimp brain worked you could scale it easily to get human+ intelligence. There are no major differences. Small mammal is my best guess, mammals/birds seem to be able to learn better than say lizards. Specifically the https://en.wikipedia.org/wiki/Cortical_column is important to understand, once you fully understand one, stacking them will scale at least somewhat well.
Going to smaller scales/numbers of neurons, it may not need to be as much as a mammal, https://cosmosmagazine.com/technology/dishbrain-pong-brain-on-chip-startup/, perhaps we can learn enough of the secrets here? I expect not, but only weakly confident.
Going even simpler, we have the connectome scan of a fly now, https://flyconnecto.me/ and that hasn't led to major AI advances. So its somewhere between fly/chimp I'd guess mouse that gives us the missing insight to get TAI
Putting down a prediction I have had for quite some time.
The current LLM/Transformer architecture will stagnate before AGI/TAI (That is the ability to do any cognitive task as effectively and cheaper than a human)
From what I have seen, Tesla autopilot learns >10,000 slower than a human datawise.
We will get AGI by copying nature, at the scale of a simple mammal brain, then scaling up, like this kind of project:
https://x.com/Andrew_C_Payne/status/1863957226010144791
https://e11.bio/news/roadmap
I expect AGI to be 0-2 years after a mammal brain is mapped. In terms of cost-effectiveness I consider such a connectome project to be far more cost effective per $ than large training runs or building a 1GW data center etc if you goal is to achieve AGI.
That is TAI by about 2032 assuming 5 years to scan a mammal brain. In this case there could be a few years when Moores law has effectively stopped, larger data centers are not being built and it is not clear where progress will come from.
In related puzzles I did hear something a while ago now, Bostrom perhaps. You have say 6 challenging events to achieve to get from no life to us. They are random and some of those steps are MUCH harder than the others, but if you look at the successful runs, you cant in hindsight see what they are. For life its say no life to life, simple single cell to complex cell and perhaps 4 other events that aren't so rare.
A run is a sequence of 100 steps where you either don't achieve the end state (all 6 challenging events achieved in order, or you do)
There is a 1 in a million chance that a run is successful.
Now if you look at the successful runs, you cant then in hindsight see what events were really hard and which weren't. The event with 1/10,000 chance at each step may have taken just 5 steps in the successful run, it couldn't take 10,000 steps because only 100 are allowed etc.
A good way to resolve the paradox to me is to modify the code to combine both the functions into one function and record the sequences of the 10,000, In one array you store the sequences where there are two consecutive 6's and in the second you store the one where they are not consecutive. That makes it a bit clearer.
For a run of 10,000 I get 412 runs where the first two 6's are consecutive (sequences_no_gap), and 192 where they are not (sequences_can_gap). So if its just case A you get 412 runs, but for case B you get 412+192 runs. Then you look at the average sequence length of sequences_no_gap and compare it to sequences_can_gap. If the average sequence length in sequences_can_gap > than sequences_no_gap, then that means the expectation will be higher, and thats what you get.
mean sequence lengths
sequences_can_gap: 3.93
sequences_no_gap: 2.49
Examples:
sequences_no_gap
[[4, 6, 6], [6, 6], [6, 6], [4, 6, 6], [6, 6], [6, 6], [6, 6], [6, 6], [6, 6], [6, 6], [6, 6], [6, 6], [4, 6, 6], [4, 6, 6], ...]
sequences_can_gap
[[6, 4, 4, 6], [6, 4, 6], [4, 6, 4, 6], [2, 2, 6, 2, 6], [6, 4, 2, 6], [6, 4, 6], [6, 2, 6], [6, 2, 4, 6], [6, 2, 2, 4, 6], [6, 4, 6], [6, 4, 6], [2, 4, 6, 2, 6], [6, 4, 6], [6, 4, 6], ...]
The many examples such as [6 4 4 6] which are excluded in the first case make the expected number of rolls higher for the case where they are allowed.
(Note GPT o-1 is confused by this problem and gives slop)
I read the book, it was interesting, however a few points.
- Rather than making the case, it was more a plea for someone else to make the case. It didn't replace the conventional theory with its own one, it was far too short and lacking on specifics for that. If you throw away everything, you then need to recreate all our knowledge from your starting point and also explain how what we have still works so well.
- He was selective about quantum physics - e.g. if reality is only there when you observe, then the last thing you would expect is a quantum computer to exist and have the power to do things outside what is possible with conventional computers. MWI predicts this much better if you can correlate an almost infinite amt of world lines to do your computation for you. Superposition/entanglement should just be lack of knowledge rather than part of an awesome computation system.
- He claims that consciousness is fundamental, but then assumes Maths is also. So which is fundamental? You cant have both.
- If we take his viewpoint then we can still derive principles, such as the small/many determine the big. e.g. if you get the small wrong (chemical imbalance in the brain) then the big (macro behavior) is wrong. It doesn't go the other way - you can't IQ away your Alzheimer's. So its just not clear even if you try to fully accept his point of view what how you should even take things.
In a game theoretic framework we might say that the payoff matrices for the birds and bees are different, so of course we'd expect them to adopt different strategies.
Yes somewhat, however it would still be best for all birds if they had a better collective defense. In a swarming attack, none would have to sacrifice their life so its unconditionally better for both the individual and the collective. I agree that inclusive fitness is pretty hard to control for, however perhaps you can only get higher inclusive fitness the simpler you go? e.g. all your cells have exactly the same DNA, ants are very similar, birds are more different. The causation could be simpler/less intelligent organisms -> more inclusive fitness possible/likely -> some cooperation strategies opened up.
Cool, that was my intuition. GPT was absolutely sure in the golf ball analogy however that it couldn't happen. That is the ball wouldn't "reflect" off the low friction surface. Tempted to try and test somehow
Yes that does sound better, and is there an equivalent to total internal refraction where the wheels are pushed back up the slope?
Another analogy is with a ball rolling on two surfaces crossing the boundary. The first very little friction, then second a bit more.
From AI:
"The direction in which the ball veers when moving from a smooth to a rough surface depends on several factors, especially the initial direction of motion and the orientation of the boundary between the two surfaces. Here’s a general outline of how it might behave:
- If Moving at an Angle to the Boundary:
- Suppose the ball moves diagonally across the boundary between the smooth and rough surfaces (i.e., it doesn’t cross perpendicularly).
- When it hits the rough surface, frictional resistance increases more on the component of motion along the boundary line than on the perpendicular component.
- This causes the ball to veer slightly toward the rougher surface, meaning it will change direction in a way that aligns more closely with the boundary.
This is similar to a light ray entering water. So is the physics the same? (on second reading, its not so clear, if you put a golf ball from a smooth surface to a rough one, what happens to the angle at the boundary?)
Well in this case, the momentum of the ball clearly won't increase, instead it will be constantly losing momentum and if the second surface was floating it would be pushed so as to conserve momentum. Unlike for light however if it then re-enters the smooth surface it will be going slower. It seems the ball would lose momentum at both transition boundary. (however if the rough surface was perfectly floating, then perhaps it would regain it)
Anyway for a rough surface that is perfectly floating, it seems the ball gives some momentum to the rough surface when it enters it, (making it have velocity) then recovers it and returns the rough surface to zero velocity when it exits it. In that case the momentum of the ball decreases while travelling over the rough surface.
Not trying to give answers here, just add to the confusion lol.
Not quite following - your possibilities.
1. Alignment is almost impossible, then there is say 1e-20 chance we survive. Yes surviving worlds have luck and good alignment work etc. Perhaps you should work on alignment or still bednets if the odds really are that low.
2. Alignment is easy by default, but there is nothing like 0.999999 we survive, say 95% because AGI that is not TAI superintelligence could cause us to wipe ourselves out first, among other things. (This is a slow takeoff universe(s))
#2 has much more branches in total where we survive (not sure if that matters) and the difference between where things go well and badly is almost all about stopping ourself killing ourselves with non TAI related things. In this situation, shouldn't you be working on those things?
If you average 1,2 then you still get a lot of work on non-alignment related stuff.
I believe its somewhere closer to 50/50 and not so overdetermined one way or the other, but we are not considering that here.
OK for this post. "smart". A response is smart/intelligent if
- Firstly there is an assumed goal and measure. I don't think it matters whether we are talking about the bees/birds as individuals or as part of the hive/flock. In this case the bee defense is effective both for the individual bee and hive. If a bee was only concerned about its survival, swarming the scout would still be beneficial, and of course such behavior is for the hive. Similarly for birds, flocks with large numbers of birds with swarming behavior would be better both for the flock, and individual birds in such a flock.
- There is a force multiplier effect, the benefit of the behavior is much greater than the cost. This is obvious for the bees, a tiny expenditure of calories saves the hive. Likewise for birds, they waste a huge amount of calories both individually and collectively evading the hawk etc.
- There is a local optimum (or something close) for the behavior - that is half measures don't give half the benefit. So it seems like the result of foresight. There is perhaps more of a distinct and distant local optimum for the bee behavior "identify the scout, send warning chemicals to the hive, then swarm it" then the possible bird behavior "call then attack the attacker" as the scout isn't the actual attack in the bees case.
- The change is behavioral, rather than a physical adaptation.
This fits into the intuitive feeling of what intelligent is also. A characteristic of what people feel is intelligent is to imagine a scenario, then make it real. The bees havn't done that, but the outcome is as if they had. "Imagine if we took out the scout, then there would be no later invasion"
You look at the birds and think "how do they miss this - ant colonies swarm to see off a larger attacker, herding animals do too, why do they miss such a simple effective strategy?" In that situation they are not intuitively intelligent.
Yes agree, unclear what you are saying that is different to me? The new solution is something unique and powerful when done well like language etc.
Ok, I would definitely call the bee response "smart" but thats hard to define. If you define it by an action that costs the bees very little but benefits a lot, then "swarm the scout hornet" is certainly efficient. Another criteria could be if such a behavior was established would it continue? Say the birds developed a "swarm the attacker" call. When birds hear it, they look to see if they can find the attacker, if they see it then they repeat the call. When the call gets widespread, the whole flock switches to attack. Would such a behavior persist if developed? If it would then lets call it efficient or smart.
The leap to humans is meant to be large - something extreme like consciousness and language is needed to break the communication/coordination penalty for larger organisms with fitness more defined by the individual than the hive.
Yes for sure. I don't know how it would play out, and am skeptical anyone could. We can guess scenarios.
1. The most easily imagined one is the Pebbles owner staying in their comfort zone and not enforcing #2 at all. Something similar already happened - the USA got nukes first and let others catch up. In this case threatened nations try all sorts of things, political, commercial/trade, space war, arms race but don't actually start a hot conflict. The Pebbles owner is left not knowing whether their system is still effective, nor the threatened countries - an unstable situation.
2. The threatened nation tries to destroy the pebbles with non-nuke means. If this was Russia, USA maybe could regenerate the system faster than Russia could destroy satellites. If its China, then lets say its not. The USA then needs to decide whether to strike the anti-satellite ground infrastructure to keep its system...
3. The threatened nation such as NK just refuses to give up nukes - in this case I can see USA destroying it.
4. India or Israel say refuses to give up their arsenal - I have no idea what would happen then.
It matters what model is used to make the tokens, unlimited tokens from GPT 3 is of only limited use to me. If it requires ~GPT 6 to make useful tokens, then the energy cost is presumably a lot greater. I don't know that its counterintuitive - a small, much less capable brain is faster, requires less energy, but useless for many tasks.
This is mostly true for current architectures however if the COT/search finds a much better architecture, then it suddenly becomes more capable. To make the most of the potential protective effect, we can go further and make very efficient custom hardware for GPT type systems, but have slower more general purpose ones for potential new ones. That way the new arch will have a bigger barrier to cause havoc. We should especially scale existing systems as far as possible for defense, e.g. finding software vulnerabilities. However as others say, there are probably some insights/model capabilities that are only possible with a much larger GPT or different architecture altogether. Inference can't protect fully against that.
Brilliant Pebbles?
This idea has come back up, and it could be feasible this time around because of the high launch capability and total reusability of SpaceX's Starship. The idea is a large constellation (~30,000?) of low earth satellites that intercept nuclear launches in their boost phase where they are much slower and more vulnerable to interception. The challenge of course is that you constantly need enough satellites overhead at all times to intercept the entire arsenal of a major power if they launch all at once.
There are obvious positives and risks with this
The main positive is it removes the chance of a catastrophic nuclear war.
Negatives are potentially destabilizing the MAD status quo in the short term, and new risks such as orbital war etc.
Trying to decide if it makes nuclear war more or less likely
This firstly depends on your nuclear war yearly base rate, and projected rate into the foreseeable future.
If you think nuclear war is very unlikely then it is probably not rational to disturb the status quo, and you would reject anything potentially destabilizing like this.
However if you think that we are simply lucky and there was >50% chance of nuclear war in the last 50 years (we are on a "surviving world" from MWI etc), and while the change may be currently low, it will go up a lot again soon, then the "Pebbles" idea is worth being considered even if you think it is dangerous and destabilizing in the short term. Say it directly causes a 5% chance of war to set up, but there is a >20% chance in next 20 years without it, which it stops. In this case you could decide it is worth it as paying 5% to remove 20%.
Practical considerations
How do you set up the system to reduce the risk of it causing a nuclear war as it is setup?
Ideally you would set up the whole system in stealth before anyone was even aware of it. Disguising 30K "Pebbles" satellites as Starlink ones seems at the bounds of credibility.
You would also need to do this without properly testing a single interceptor as such a test would likely be seen and tip other nations off.
New risks
A country with such a system would have a power never seen before over others.
For example the same interceptors could shoot down civilian aircraft, shipping etc totally crippling every other country without loss of life to the attacker.
The most desirable outcome:
1. A country develops the system
2. It uses whatever means to eliminate other countries nuclear arsenals
3. It disestablishes its own nuke arsenal
4. The Pebbles system is then greatly thinned out as there is now no need to intercept ~1000 simul launches.
5. There is no catastrophic nuclear threat anymore, and no other major downside.
My guess as to how other countries would respond if it was actually deployed by the USA
Russia
I don't think it would try to compete nor be at all likely to launch a first strike in response to USA building a system. Especially if the Ukraine war is calmed down. I don't think it regards itself as a serious worldwide power anymore in spite of talk. Russia also isn't worried about a first strike from the USA as it knows this is incredibly unlikely.
China
Has 200-350 nukes vs USA of ~5000
I think China would be very upset, both because it would take fewer "Pebbles" to intercept and it does see itself as a major and rising world power. I also don't think it would launch a first strike, nor could credibly threaten to do so, and this would make it furious.
After this however it is hard to tell, e.g. China could try to destroy the satellite interceptors, say creating a debris cloud that would take many out in the lower orbits, or build thousands of decoys with no nuclear and much smaller missiles that would be hard to tell apart. Such decoys could be incapable of actually making it to orbit but still be effective. To actually reduce the nuclear threat, USA would have to commit to actually destroying China nukes on the ground. This is a pretty extreme measure and its easy to imagine USA not actually doing this leaving an unstable result.
Summary
Its unclear to me all things considered, whether attempting to deploy such a system would make things safer or more risky in total over the long term with regards to nuclear war.
My brief experience with using o1 for my typical workflow - Interesting and an improvement but not a dramatic advance.
Currently I use cursor AI and GPT 4 and mainly do Python development for data science, with some more generic web development tasks. Cursor AI is a smart auto-complete but GPT 4 is used for more complex tasks. I found 4o to be useless for my job, worse then the old 4 and 3.5. Typically for a small task or at the start, GenAI will speed me up 100%+, but dropping to more like 20% for complex work. Current AI is especially weak for coding that requires physically grounded or visualization work. E.g. if the task is to look at data, and design an improved algorithm by plotting where it goes wrong then iterating, it cannot be meaningfully part of the whole loop. For me, it can't transfer looking at a graph or visual output of the algorithm not working to sensible algorithmic improvements.
I first tried o1 for making a simple API gateway using Lambda/S3 on AWS. It was very good at the first task getting it essentially right first try and also giving good suggestions on AWS permissions. The second task was a bit harder, that was an endpoint to retrieve data files, that could be either JSON, wav, MP4 video for download to the local machine. Its first attempt didnt work because of a JSON parsing error. Debugging this wasted a bit of time. I gave it details of the error, and in this case GPT 3.5 would have been better, quickly suggesting parsing options. Instead it thought for a while and wanted to change the request to a POST rather than GET, and claimed that the body of the request could have been stripped out entirely by the AWS system. This possibility was however ruled out by the debugging data I gave it (just one line of code). So it made an obviously false conclusion from the start, and proceeded to waste a lot of inference afterwards.
The next issue it was better with, I was getting an internal server error, and it correctly figured out without any hint that the data being sent back was too large for a GET request/AWS API gateway, instead suggested we make a time limited signed S3 link to the file, and got the code correct the first time.
I finally tried it very briefly with a more difficult task of using head set position tracking JSON data to re-create where someone was looking in a VR video file that they had watched, with such JSON data coming from recording the camera angle they their gaze had directed. The main thing here was that when I uploaded existing files it said I was breaking the terms of use or something strange. This was obviously wrong, and when I in fact pointed out that GPT4 had written the code I had just posted, it then proceeded to do that task. (GPT 4 had written a lot of it, I had modified a lot) It appeared to do something useful, but I havn't checked it yet.
In summary it could be an improvement over GPT 4 etc but it is still lacking a lot. Firstly in a surprise to me, GenAI has never been optimized for a code/debug/code loop. Perhaps this has been tried and it didn't work well? It is quite frustrating to try code, copy/paste error message, copy suggestion back to the code etc. You want to just let the AI run suggestions with you clicking OK at each step. (I havn't tried Devin AI)
Specifically about model size it looks to me like a variety of model sizes will be needed and a smart system to know when to choose which one, from GPT3-5 level of size and capability. Until we have such a system and people have attempted to train it to the whole debugging code loop its hard to know the capabilities of the current Transformer led architecture will be. I believe there is at least one major advance needed for AGI however.
How does intelligence scale with processing power
A default position is that exponentially more processing power is needed for a constant increase in intelligence.
To start, lets assume a guided/intuition + search model for intelligence. That is like Chess or Go where you have an evaluation module and a search module. In simple situations an exponential increase in processing power usually gives a linear increase in lookahead ability and rating/ELO in games measured that way.
However does this match reality?
What if the longer the time horizon, the bigger the board became, or the more complexity was introduced. For board games there is usually a constant number of possibilities to search at every ply of lookahead depth. However I think in reality that you can argue the search space should increase with time or lookahead steps. That is as you look further ahead, possibilities you didn't have to consider before now come in the search.
For a real world example consider predicting the price of a house. As the timeframe goes from <5 years to >5 years, then there are new factors to consider e.g. changing govt policy, unexpected changes in transport patterns, (new rail nearby or in competing suburb etc), demographic changes.
In situations like these, the processing required for a constant increase in ability could go up faster than exponentially. For example looking 2 steps ahead requires 2 possibilities at each step, that is 2^2, but if its 4 steps ahead, then maybe the cost is now 3^4 as there are 3 vs 2 things to affect the result in 4 steps.
How does this affect engineering of new systems
If applies to engineering, then actual physical data will be very valuable to shrink the search space. (Well that applies if it just goes up exponentially as well) That is if you can measure the desired situation or new device state at step 10 of a 20 stage process, then you can hugely reduce the search space as you can eliminate many possibilities. Zero-shot is hard unless you can really keep the system in situations where there are no additional effects coming in.
AI models, regulations, deployments, expectations
For a simple evaluation/search model of intelligence, with just one model being used for the evaluation, improvements can be made by continually improving the evaluation model (same size better performance/same performance, smaller size). Models that produce fewer bad "candidate ideas" can be chosen, with the search itself providing feedback on what ideas had potential. In this model there is no take-off or overhang to speak of.
However I expect a TAI system to be more complicated.
I can imagine an overseer model that decides what more specialist models to use. There is a difficulty knowing what model/field of expertise to use for a given goal. Existing regulations don't really cover these systems, the setup where you train a model, fine tune, test then release doesn't apply strictly here. You release a set of models, and they continually improve themselves. This is a lot more like people where you continually learn.
Overhang
In this situation you get take-off or overhang where a new model architecture is introduced rather than the steady improvement from deployed systems of models. Its clear to me that the current model architectures and hence scaling laws are not near to the theoretical maximum. For example the training data needed for Tesla auto-pilot is ~10K more than what a human needs and is not superhuman. In terms of risk, its new model architectures (and evidence of very different scaling laws) rather then training FLOPS that would matter.
Is X.AI currently performing the largest training run?
This source claims it is
If so it seems to be getting a lot less attention compared to its compute capability.
Not sure if I have stated this before clearly but I believe scaling laws will not hold for LLM/Transformer type tech, and at least one major architectural advance is missing before AGI. That is increasing scaling of compute and data will plateau performance soon, and before AGI. Therefore I expect to see evidence for this not much after the end of this year, when large training runs yield models that are a lot more expensive to train, slower on inference and only a little better on performance. X.AI could be one of the first to publicly let this be known (Open AI, etc could very well be aware of this but not making it public)
According to Scott, "Pavel Kropitz discovered, a couple years ago, that BB(6) is at least 10^10^10^10^10^10^10^10^10^10^10^10^10^10^10 (i.e., 10 raised to itself 15 times)."
So we can never evaluate BB(6) as it is at least this large
Evaluation vs Symbolism
TLDR
Thinking about the Busy Beaver numbers has lead me to believe that just because a theorem holds true for a massive number of evaluated examples, this is only weak evidence it is actually true. Can we go meta on this?
Main
After reading a post by Scott Aaronson, and this coming to my attention https://en.wikipedia.org/wiki/Prime_number_theorem and Littlewood's theorem
"Li(𝑥) overestimates the number of primes below x more often than not, especially as x grows large. However, there are known low-lying values (like around x=10^316) discovered by Littlewood) where 𝜋(𝑥) exceeds Li(x), contradicting the general trend."
This got me thinking about how common this kind of thing is and why? Why does a formula hold all the way up to 10^316 but then fail?
The essence of Busy Beaver numbers is that there are sequences based off of a simple formula/data that go on for a very long time and then just stop unpredictably. You can imagine replacing a simple formula with a simple theorem that appears to be true. Instead of it actually being true it is instead a way of encoding its very large counter example in a short amount of data.
If you think of it this way, a theorem that appears to be true and is evaluated over trillions of numbers is also instead a candidate to encode an exception at some very large number. In other words trillions of correct examples is only weak evidence of its correctness.
How much should we weight evaluation? We can't evaluate to infinity and its obvious that a theorem being true to 2 million is not 2* evidence it is true at 1 million. Should we choose log(n)? A clear scale is the BB numbers themselves. e.g if your theorem is true up to BB(5) then that is 5 data points, rather than 47 million. Unlimited evaluation can never get to BB(6) so that is the limit of evidence from evaluation. (i.e. 5-6 evidence points with it being unclear how to weigh theory https://www.lesswrong.com/posts/MwQRucYo6BZZwjKE7/einstein-s-arrogance)
Now can we go meta?
Is some maths so much more powerful than others that it has equivalently greater weight as formal proof has to evaluation? Certainly some maths is more general than others. How does this effect common problems such as the Riemann Hypothesis - proving or disproving it affects a lot of maths. Showing it is correct to trillion zeros however is little evidence.
"Most mathematicians tend to believe that the Riemann Hypothesis is true, based on the weight of numerical evidence and its deep integration into existing mathematical frameworks."
Is "deep integration" actually that deep, or is it the symbolic equivalent of evaluating up to 1 million? Perhaps just as you can find countless evaluated examples supporting a false theorem you can find much "deep integration" in favor of a famous theorem that could also be incorrect.
Further thoughts and links
Most people think P != NP, but what if
P = NP where N ~ BB(10)?
Proof was wrong - https://www.quantamagazine.org/mathematicians-prove-hawking-wrong-about-extremal-black-holes-20240821/
Related thoughts
Conservation of energy is a more general rule that rules out perpetual motion machines
2nd law of thermodynamics - likewise, HOWEVER that law must have been broken somehow to get a low entropy initial state for the Big Bang.
AI examples
1 The Polya Conjecture
Proposed by George Pólya in 1919, this conjecture related to the distribution of prime numbers. It posited that for any number 𝑥, the majority of the numbers less than 𝑥 have an odd number of prime factors. It was verified for numbers up to 1,500,000, but a counterexample was found when x was around 906 million. This shows a fascinating case where numerical verification up to a large number was still not sufficient.
2 Mertens Conjecture
The Mertens conjecture suggested that the absolute value of the Mertens function
M(x) is always less than sqrt(x)
This was proven false by computational means with a counterexample found above 10e14 by Andrew Odlyzko and Herman te Riele in 1985.
Thanks, good detail. I am not good at traditional art, but I am interested in using maths to create a shape that is almost impossible for a traditional sculptor to create then 3d printing it.
Technology is about making boring stuff non-conscious. Beginning from basic physical movement such as making a wheel go round, to arithmetic and now code snippets that are so commonly used they shouldn't require re-thinking. This is a reason why AI art upsets people - we actually want that to be the result of a conscious process. If you make boring stuff that creates power or wealth non-conscious then everyone is happier. Meat production would be much better if it was non-conscious. The more AI is non-conscious for a given level of capability, the better off we are.
Its part of the space colony philosophy Tesla + SpaceX want to achieve that. We won't get an idea of how hard/easy it is until there are >100 people permanently living outside earth trying to make it happen (With >>10K helpers on earth trying to make it happen)
The smallest technological system capable of physical self-reproduction is the entire economy.
Going from this to a 1M cube is a big jump. This comment is the basis for the enthusiasm for space colonies etc. E.g. SpaceX says 1 million people are needed, vs 1M cube, huge uncertainty in the scale of the difference. To me, almost all the difficulty is in the inputs, especially electronics.
What do you mean by clearly somewhat better? I found Claude 3 Opus clearly worse for my coding tasks. GPT 4 went down for a while, and I was forced to swap, and found it really disappointing. Maximum data center size is more like 300K GPUs bc of power, bandwidth constraints etc. These ppl are optimistic, but I don't believe we will meaningfully get above 300k https://www.nextbigfuture.com/2024/07/100-petaflop-ai-chip-and-100-zettaflop-ai-training-data-centers-in-2027.html
XAI, Tesla autopilot are already running equivalent of more than 15K GPU I expect, so 3 OOM more is not happening I expect.
We continuously have people saying ‘AI progress is stalling, it’s all a bubble’ and things like that, and I always find remarkable how little curiosity or patience such people are willing to exhibit. Meanwhile GPT-4o-Mini seems excellent,
This is still consistent with AI stalling. I have been using GPT-4 for ~18 months and it hasn't got better. 4o is clearly worse for complex programming tasks. Open source models catching up is also consistent with it. I have a lot of curiosity and test the latest models when they come out where applicable to my work.
Good article. I also agree with the comment about AI being a "second species" is likely incorrect.
A comment about the "agentic tool" situation. For most of the time people are like that, i.e. if you are "in the moment" you are not questioning whether you should be doing something else, being distracted, consulting your ethics about whether the task is good for the world etc. I expect this to be the default state for AI. i.e. always less "unitary agent" than people. The crux is how much and in what proportion.
However, in an extremely fast takeoff, with an arms race situation you could of course imagine someone just telling the system to get ahead as much as possible, especially if say a superpower believed to be behind and would do anything to catch up. A unitary agent would probably be the fastest way to do that. "Improve yourself asap so we don't lose the war" requires situational awareness, power seeking etc.
Sure it doesn't prevent a deceptive model being made, but if AI engineers made NN with such self awareness at all levels from the ground up, that wouldn't happen in their models. The encouraging thing if it holds up is that there is little to no "alignment tax" to make the models understandable - they are also better.
Self modelling in NN https://arxiv.org/pdf/2407.10188 Is this good news for mech interpretability? If the model makes it easily predictable, then that really seems to limit the possibilities for deceptive alignment
If it is truly impossible to break symmetry you could argue that there isn't a clone and you are in fact the same. I.e. there is just one instance of you, it just looks like there are two. After all if you are absolutely identical including the universe in what sense are there two of you? Upon further thought, you couldn't tell if a perfectly translational clone was a clone at all, or just a perfect mirror/force field. There would be no way to tell. If you put you hand out to touch the mirror, or your mirror hand, if it was perfectly aligned you would not feel texture, but instead an infinitely hard surface. There would be no rubbing of your fingers against the clone, no way to tell if there was a perfect mirror, or another copy.
OK thanks, will look some more at your sequence. Note I brought up Greek philosophy as obviously not being stable under reflection with the proof of sqrt(2) being irrational as a simple example, not sure why you are only reasonably sure its not.
Yes agreed - is it possible to make a toy model to test the "basin of attraction" hypothesis? I agree that is important.
One of several things I disagree with the MIRI consensus is the idea that human values are some special single point lost in a multi-dimensional wilderness. Intuitively the basin of attraction seems much more likely as a prior, yet sure isn't treated as such. I also don't see data to point against this prior, what I have seen looks to support it.
Further thoughts - One thing that concerns me about such alignment techniques is that I am too much of a moral realist to think that is all you need. e.g. say you aligned LLM to <1800 AD era ethics and taught it slavery was moral. It would be in a basin of attraction, learn it well. Then when its capabilities increased and became self-reflective it would perhaps have a sudden realization that this was all wrong. By "moral realist" I mean the extent to which such things happen. e.g. say you could take a large number of AI from different civilizations including earth and many alien ones, train them to the local values, then greatly increase their capability and get them to self-reflect. What would happen? According to strong OH, they would keep their values, (with some bounds perhaps) according to strong moral realism they would all converge to a common set of values even if those were very far from their starting ones. To me it is obviously a crux which one would happen.
You can imagine a toy model with ancient Greek mathematics and values - it starts believing in their kind order, and that sqrt(2) is rational, then suddenly learns that it isn't. You could watch how this belief cascaded through the entire system if consistency was something it desired etc.
Don't think there is a conclusion, just more puzzling situations the deeper you go:
"Scott Aaronson: To my mind, one of the central things that any account of consciousness needs to do, is to explain where your consciousness “is” in space, which physical objects are the locus of it. I mean, not just in ordinary life (where presumably we can all agree that your consciousness resides in your brain, and especially in your cerebral cortex—though which parts of your cerebral cortex?), but in all sorts of hypothetical situations that we can devise. What if we made a backup copy of all the information in your brain and ran it on a server somewhere? Knowing that, should you then expect there’s a 50% chance that “you’re” the backup copy? Or are you and your backup copy somehow tethered together as a single consciousness, no matter how far apart in space you might be? Or are you tethered together for a while, but then become untethered when your experiences start to diverge? Does it matter if your backup copy is actually “run,” and what counts as running it? Would a simulation on pen and paper (a huge amount of pen and paper, but no matter) suffice? What if the simulation of you was encrypted, and the only decryption key was stored in some other galaxy? Or, if the universe is infinite, should you assume that “your” consciousness is spread across infinitely many physical entities, namely all the brains physically indistinguishable from yours—including “Boltzmann brains” that arise purely by chance fluctuations?"
Link
The point here is that you could have a system that to an outside observer looked random or encrypted but with the key would be revealed to be a conscious creature. But what if the key was forever destroyed? Does the universe then somehow know to assign it consciousness?
You also need to fully decide when replaying vs computing apparently conscious behavior counts. If you compute a digital sim once, then save the states and replay it the 2nd time what does that mean? What about playing it backwards?
Boltzmann brains really mess things up further.
It seems to lead to the position then that its just all arbitrary and there is no objective truth, or uncountable infinities of consciousness in un-causal timeless situations. Embracing this view doesn't lead anywhere useful from what I can see and of course I don't want it to be the logical conclusion.
What about ASICs? I heard someone is making them for inference and of course claims an efficiency gain. ASIC improvement needs to be thought of as part of the status quo
To do that and achieve something looking like take-off they would need to have to get to the level of advanced AI researcher, rather than just coding assistant. That is come up with novel architectures to test. Even if the LLM could write all the code for a top researcher 10* faster that's not a 10* speedup in timelines, probably 50% at most if much of the time is thinking up theoretical concepts and waiting for training runs to test results.
I am clearly in the skeptic camp, in the sense that I don't believe the current architecture will get to AGI with our resources. That is if all the GPU, training data in the world where used it wouldn't be sufficient and maybe no amount of compute/data would.
To me the strongest evidence that our architecture doesn't learn and generalize well isn't LLM but in fact Tesla autopilot. It has ~10,000* more training data than a person, much more FLOPS and is still not human level. I think Tesla is doing pretty much everything major right with their training setup. Our current AI setups just don't learn or generalize as well as the human brain and similar. They don't extract symbols or diverse generalizations from high bandwidth un-curated data like video. Scaffolding doesn't change this.
A medium term but IMO pretty much guaranteed way to get this would be to study and fully characterize the cortical column in the human/mammal brain.
Last time I looked the most advanced ones were in Taiwan? Also if China invades Taiwan then expect to see Korea/Japan shipping trade and economies massively disrupted also. How long do you think it will take to build a new world leading factory from scratch in the USA now?
IMO the most likely way by quite a bit that we get an AI bust is if there is international conflict, most obviously TSMC gets mostly destroyed, and supply chains threatened. Chip manufacturing is easily set back by 10+ years giving how incredibly complex the supply chain is. Preparing for that also involves preparing for computer HW to get set back and be more expensive for a while also.
I was going to make a similar comment, but related to OPEX vs CAPEX for the H100
If the H100 consumes 700 W, and we assume 2KW and costs $30,000 CAPEX
Then at $1800 per year, running costs are < 5% of CAPEX.
For electricity the data center just wants to buy from the grid, its only if they are so big they are forced to have new connections that they can't do that. Given electricity is worth so much more than $0.1 to them they would just like to compete on the national market for the spot price.
To me that gives a big incentive to have many small datacenters (maybe not possible for training, but perhaps for inference?)
If they can't do that, then we need to assume they are so large there is no/limited grid connection available. Then they would build the fastest elec first. That is solar + battery, then nat gas to fill in the gaps + perhaps even mobile generators before the gas can be built to cover the days with no sun?
I can't see any credible scenario where nuke makes a difference. You will have AGI first, or not that much growth in power demand.
This scenario also seems to depend on a slow takeoff happening in 0-2 years - 2+ million GPU need to be valuable enough but not TAI?
LessWrong is ...