then that's just irrelevant. You don't need to evaluate millions of positions to backtrack (unless you think humans don't backtrack) or play chess. 

Humans are not transformers. The "context window" for a human is literally their entire life.

Setting up the architecture that would allow a pretrained LLM to trial and error whatever you want is relatively trivial.

I agree.  Or at least, I don't see any reason why not.

My point was not that "a relatively simple architecture that contains a Transformer as the core" cannot solve problems via trial and error (in fact I think it's likely such an architecture exists).  My point was that transformers alone cannot do so.

You can call it a "gut claim" if that makes you feel better.  But the actual reason is I did some very simple math (about the window size required and given quadratic scaling for transformers) and concluded that practically speaking it was impossible.

Also, importantly, we don't know what that "relatively simple" architecture looks like.  If you look at the various efforts to "extend" transformers to general learning machines, there are a bunch of different approaches: alpha-geometry, diffusion transformers, baby-agi, voyager, dreamer, chain-of-thought, RAG, continuous fine-tuning, V-JEPA.  Practically speaking, we have no idea which of these techniques is the "correct" one (if any of them are).

In my opinion saying "Transformers are AGI" is a bit like saying "Deep learning is AGI".  While it is extremely possible that an architecture that heavily relies on Transformers and is AGI exists, we don't actually know what that architecture is.

Personally, my bet is either on a sort of generalized alpha-geometry approach (where the transformer generates hypothesis and then GOFAI is used to evaluate them) or Diffusion Transformers (where we iteratively de-noise a solution to a problem).  But I wouldn't be at all surprised if a few years from now it is universally agreed that some key insight we're currently missing marks the dividing line between Transformers and AGI.

Ok? That's how you teach anybody anything. 

Have you never figured out something by yourself?  The way I learned to do Sudoku was: I was given a book of Sudoku puzzles and told "have fun".

you said it would be impossible to train a chess playing model this century.

I didn't say it was impossible to train an LLM to play Chess. I said it was impossible for an LLM to teach itself to play a game of similar difficulty to chess if that game is not in it's training data.

These are two wildly different things.

Obviously LLMs can learn things that are in their training data.  That's what they do.  Obviously if you give LLMs detailed step-by-step instructions for a procedure that is small enough to fit in its attention window, LLMs can follow that procedure.  Again, that is what LLMs do.

What they do not do is teach themselves things that aren't in their training data via trial-and-error.  Which is the primary way humans learn things.

sure.  4000 words (~8000 tokens) to do a 9-state 9-turn game with the entire strategy written out by a human.  Now extrapolate that to chess, go, or any serious game.

And this doesn't address at all my actual point, which is that Transformers cannot teach themselves to play a game.

Absolutely.  I don't think it's impossible to build such a system.  In fact, I think a transformer is probably about 90% there.   Need to add trial and error, some kind of long-term memory/fine-tuning and a handful of default heuristics.  Scale will help too, but no amount of scale alone will get us there.

It certainly wouldn't generalize to e.g Hidouku

In the technical sense that you can implement arbitrary programs by prompting an LLM (they are turning complete), sure.

In a practical sense, no.

GPT-4 can't even play tic-tac-toe.  Manifold spent a year getting GPT-4 to implement (much less discover) the algorithm for Sudoku and failed.

Now imagine trying to implement a serious backtracking algorithm.  Stockfish checks millions of positions per turn of play.  The attention window for your "backtracking transformer" is going to have to be at lease {size of chess board state}*{number of positions evaluated}.

And because of quadratic attention, training it is going to take on the order of {number or parameters}*({chess board state size}*{number of positions evaluated})^2

Even with very generous assumptions for {number of parameters} and {chess board state}, there's simply no way we could train such a model this century (and that's assuming Moore's law somehow continues that long).

Obvious bait is obvious bait, but here goes.

Transformers are not AGI because they will never be able to "figure something out" the way humans can.

If a human is given the rules for Sudoku, they first try filling in the square randomly.  After a while, they notice that certain things work and certain things don't work.  They begin to define heuristics for things that work (for example, if all but one number appears in the same row or column as a box, that number goes in the box).  Eventually they work out a complete algorithm for solving Sudoku.

A transformer will never do this (pretending Sudoku wasn't in its training data).  Because they are next-token predictors, they are fundamentally incapable of reasoning about things not in their training set.  They are incapable of "noticing when they made a mistake" and then backtracking they way a human would.

Now it's entirely possible that a very small wrapper around a Transformer could solve Sudoku.  You could have the transformer suggest moves and then add a reasoning/planning layer around it to handle the back-tracking.  This is effectively what Alpha-Geometry does.

But a Transformer BY ITSELF will never be AGI.

Lots of people think that.  Occasionally because they are in a situation where the assumptions of Kelly don't hold (for example, you are not playing a repeated game), but more often because they confuse single bets (where Kelly maximizes log dollars) with repeated play (where Kelly maximizes total dollars).

In the long run, Kelly is optimal so long as your utility function is monotonic in dollars since it maximizes your total dollars.

From wikipedia

No matter what the reward function, isn't a rational maximizer engaging in repeated play going to "end up" concave due to e.g Kelly Criterion?  I would think convex agents would be sub-optimal/quickly self-destruct if you tried to create one and nature will basically never do so.

The confusion arises because the terms Inner/Outer alignment were invented by people doing philosophy and not people doing actual machine learning.  In practice inner misalignment is basically not a thing.  See, for example, Nora Belrose's comments about the "counting argument".

but for sufficiently large function approximators, the trend reverses

Transformers/deep learning work because of built-in regularization methods (like dropout layers) and not because "the trend reverses".  If you did naive "best fit polynomial" with a 7 billion parameter polynomial you would not get a good result.

I do not know who is right about Mamba here and it doesn’t seem easy to check?

Nora is definitely right in this case (Mamba isn't doing dramatically more serial operations than Transformers).  If it was, it would be slower running in not-batch-mode, whereas the whole point of Mamba is it's faster (due to sub-quadratic attention).

and I am confident that no, Google did this on their own

What makes you so confident?  We know that the gov't has its fingers in a lot of pies (remember twitter?) and this seems like exactly the sort of thing the Biden executive order wants to be about: telling industry how to safely build AI.

Same thing with image generation. When I want something specific, I expect to be frustrated and disappointed. When I want anything at all within a vibe zone, when variations are welcomed, often the results are great.

I routinely start with a specific image in mind and use AI art to generate it.  Mostly this means not just using text-to-image and instead using advanced techniques like controlnet, ipadapter, img2img, regional prompter, etc.  

Yes, this is a skill that has to be learned, but it is still 100x faster than I could achieve the same thing before AI art.  When it comes to movies, the speedup will be even greater.  The $500m budget marvel movies of today will be something that a team of 5-10 people like Corridor Crew can put together in six months on a budget of <$1m two years from now.

There are also important technical limitations of existing (Clip-based) models that go away entirely when we switch to a transformer architecture.  This image would be basically impossible to get (using only text-to-image) using existing models.

"For this invention will produce forgetfulness in the minds of those who learn to use it, because they will not practice their memory. Their trust in writing, produced by external characters which are no part of themselves, will discourage the use of their own memory within them. You have invented an elixir not of memory, but of reminding; and you offer your pupils the appearance of wisdom, not true wisdom, for they will read many things without instruction and will therefore seem to know many things, when they are for the most part ignorant and hard to get along with, since they are not wise, but only appear wise."

idk.

Maybe I got carried away with the whole "everything overhang" idea.

While I do think fast vs slow takeoff is an important variable that determines how safe a singularity is, it's far from the only thing that matters.

If you were looking at our world today and asking "what obvious inefficiencies will an AGI exploit?" there are probably a lot of lower-hanging fruits (nuclear power, genetic engineering, zoning) that you would point to before getting to "we're not building chip fabs as fast as physically possible".

My actual views are probably closest to d/acc which is that there are a wide variety of directions we can chose when researching new technology and we ought to focus on the ones that make the world safer.

I do think that creating new obvious inefficiencies is a bad idea.  For example, if we were to sustain a cap of 10**26 FLOPs on training runs for a decade or longer, that would make it really easy for a rouge actor/ai to suddenly build a much more powerful AI than anyone else in the world has.

As to the specific case of Sam/$7T, I think that it's largely aspiration and to the extent that it happens it was going to happen anyway.  I guess if I was given a specific counterfactual, like: TSMC is going to build 100 new fabs in the next 10 years, is it better that they be built in the USA or Taiwan? I would prefer they be built in the USA.  If, on the other hand, the counterfactual was: the USA is going to invest $7T in AI in the next 10 years, would you prefer it be all spent on semiconductor fabs or half on semiconductor fabs and half on researching controllable AI algorithms, I would prefer the latter.

Basically, my views are "don't be an idiot", but it's possible to be an idiot both by arbitrarily banning things and by focusing on a single line of research to the exclusion of all others.

compared to what?

My understanding is that the fundamental disagreement is over whether there will be a "sharp discontinuity" at the development of AGI.  

In Paul's model, there is no sharp discontinuity.  So, since we expect AGI to have a large economic impact, we expect "almost AGI" to have an "almost large" economic impact (which he describes as being trillions of dollars).  

One way to think of this is to ask: will economic growth suddenly jump on the day AGI is invented?  Paul think's 'no' and EY thinks 'yes'.

Since sudden discontinuities are generally dangerous, a slow (continuous) takeoff is generally thought of as safer, even though the rapid economic growth prior to AGI results in AGI happen sooner.

This also effects the "kind of world" that AGI enters.  In a world where pre-AGI is not widely deployed, the first AGI has a large "compute advantage" versus the surrounding environment.  But in a world where pre-AGI is already quite powerful (imagine everyone has a team of AI agent that handles their financial transactions, protects them from cyber threats, is researching the cutting edge of physics/biology/nanotechnology, etc), there is less free-energy so to speak for the first AGI to take adavantage of.

Most AI Foom stories involve the first AGI rapidly acquiring power (via nanomachines or making computers out of dna or some other new technology path).  But if pre-AGI AIs are already exploring these pathways, there are fewer "exploits" for the AGI to discover and rapidly gain power relative to what already exists.

edit:

I feel like I didn't sufficiently address the question of compute overhang.  Just as a "compute overhang" is obviously dangerous, so is an "advanced fab" overhang or a "nanotechnology" overhang", so pushing all of the tech-tree ahead enhances our safety.

I would not use Manifold as any data point in assessing the potential danger of future AI.

What would you use instead?  

In particular, I'd be interested in knowing what probability you assign to the chance that GPT-5 will destroy the world and how you arrived at that probability.

I think that Sam's actions increase the likelihood of a slow takeoff.

Consider Paul's description of a slow takeoff from the original takeoff speed debate

right now I think hardware R&D is on the order of $100B/year, AI R&D is more like $10B/year, I guess I'm betting on something more like trillions? (limited from going higher because of accounting problems and not that much smart money)

I do actually think $7T is enough that it would materially accelerate Moore's law, since "production gets more efficient over time" style laws tend to be be functions of "quantity produced" not of time.

In a world where we're currently spending ~$600B/year on semiconductors, spending a few billion (current largest AI training runs) is insignificant, but if Sam really does manage to spend $7T/5 years, that would be basically tripling our semiconductor capacity.

There might also be negative feedback loops, because when you try to spend a large amount of money quickly you tend to do so less efficiently, so I doubt Moore's law would literally triple.  But if you thought (as Kurtzweil predicts) AGI will arrive circa 2035 based on Moore's law alone, an investment of this (frankly ridiculous) scale reducing that time from 10 years down to 5 is conceivable.

I think you’re confused. If you hold the “singularity date” / “takeoff end time” fixed—e.g., you say that we somehow know for certain that “the singularity” will happen on August 29, 2047—then the later takeoff starts, the faster it is. I think that’s what you have in mind, right? 

That's not what I have in mind at all.

I have picture like this in mind when I think of slow/fast takeoff:



There are 3 types of ways we can accelerate AGI progress:
1. Moore's law (I consider this one more or less inevitable)

 As we make progress in the material sciences, the cost of compute goes down

2. Spending

If we spend more money on compute, we can build AGI sooner

3. Algorithmic progress

The more AI we build, the better we get at building AI


Assuming we don't plan to ban science altogether (and hence do not change Moore's law), then pausing AI research (by banning large training runs) inevitably leads to a period of catch-up growth when the pause ends.

I think that the line marked "slow takeoff" is safer because:
1. the period of rapid catch up growth seems the most dangerous
2. we spend more time near the top of the curve where AI safety research is the most productive

I suppose that if you could pause exactly below the red line marked "dangerous capabilities threshold", that would be even safer.  But since we don't know where that line is I don't really believe that to be possible.  The closest approximation is Anthropic's RSP or Open AI's early warning system which says "if we notice we've already crossed the red line, then we should definitely pause".

No, I will not agree that GPT5 will not destroy the world, cause I have no idea what it will be capable of.

Great, this appears to be an empirical question that we disagree on!

I (and the Manifold prediction market I linked) think there is a tiny chance that GPT-5 will destroy the world.

You appear to disagree.  I hope you are buying "yes" shares on Manifold and that one of us can "update our priors" a year from now when GPT-5 has/has-not destroyed the world.

all take offs are equally dangerous.

I think that a slow (earlier) takeoff is safer than a fast (later) takeoff.

Let us agree for a moment that GPT-5 is not going to destroy the world.

Suppose aliens were going to arrive at some unknown point in the future.  Humanity would obviously be in a better position  to defend themselves if everyone on Earth had access to GPT-5 than if they didn't.

Similarly, for AGI.  If AGI arrives and finds itself in a world where most humans already have  access to powerful (but not dangerous) AIs, then it is less likely to destroy us all.

As an extreme, consider a world in which all technology was legally frozen at an Amish level of development for several thousand years but nonetheless some small group of people eventually broke the restriction and secretly developed AGI.  Such a world would be much more doomed than our own.

Rootclaim's value as a source of truth to me will remain at "essentially zero" so long as they continue to have this on their homepage.

The faster the takeoff, the more dangerous, the thinking goes. 

You seem to be fundamentally confused about what the words "fast takeoff" and "slow takeoff" mean.  A fast takeoff occurs later in time than a slow takeoff because a slow takeoff involves gradual acceleration over a longer period of time, whereas a slow takeoff involves sudden takeoff over a shorter period of time.

Slow takeoffs are considered safer not because of when they take, but because a gradual ramp-up gives us more opportunities to learn about AI alignment (we spend more time near the part of the curve were AI is powerful but not yet dangerous).

And yes, I am quickly getting tired of doing this close reading over and over again every time anyone introduces a draft bill, dealing with the same kind of legal theoretical maximalism combined with assuming no one fixes the language.

Is there a specific name for the fallacy of claiming "no would be so stupid as to.." particularly when applied to Congress which has repeatedly demonstrated an ability to be so stupid.

Iteration + RLHF: RLHF actively rewards the system for hiding problems, which makes iteration less effective; we’d be better off just iterating on a raw predictive model.

I don't think is is actually true.  instruct-tuned models are much better at following instructions on real-world tasks than a "raw predictive model".

If we're imagining a chain of the form: human -> slightly smarter than Human AGI -> much smarter than human AGI -> ... -> SAI, we almost certainly want the first AGI to be RLHF'd/DPO's/whatever the state of the art is at the time.

Because the economy seems to have the property that when you change a bunch of things at once it takes a while to reach a new equilibrium.

As you've noted, the people reducing their consumption (due to higher taxes) and different from the people increasing their consumption (due to buying fewer bonds).  I expect the people lowering their consumption will react more quickly due to loss aversion leading to a short-term drop in aggregate demand.

The question is, can we find a mechanism to produce this adjustment instantly and painlessly.

As an example of the sort of thing, I'm thinking off: normally deflation causes unemployment in an economy due to phillips-curve effects.  But if a country simply "crosses off zeros" from their currency, this has basically no effect on the real economy despite technically causing massive deflation.

The question is, might there be a "crosses off zeros" equivalent for lowering the national deficit.

I predict step 3 causes a lot of unemployment.

We also seem to have different opinions about whether the ZLB is a real thing.  Even at the ZLB I think the Fed can still stimulate demand with QE.

The new taxes would decrease the amount of money people have to spend, but this would be exactly balanced by an increase in money available to spend due to people no longer using their money to buy government bonds. 

This is exactly my point.  The only difference (ideally) between the current world and 4. is the on-paper accounting.

I mean, but our universe is not Conway's Game of Life. 

Setting aside for now the problems with our universe being continuous/quantum weirdness/etc, the bigger issue has to do with the nature of the initial state of the board.

Whether or not math would be unreasonably effective in a universe made out of Conway's Game of Life depends supremely on the initial state of the board.  

If the board was initialized randomly, then it would already be in a maximum-entropy distribution, hence "minds" would have no predictive power and math would not be unreasonably effective.  Any minds that did come into existence would be similar to Boltzmann Brains in the sense that they would come into existence for one brief moment and then be destroyed the next.

The initial board would have to be special for minds like ours  to exist in Conway's Game of Life.  The initial setup of the board would have to be in a specific configuration that allowed minds to exist for long durations of time and predict things.  And in order for that to be the case, there would have to be some universe wide set of rules governing how the board was set up.  This is analogous to how the number "2" is a thing mathematicians think is useful no matter where you go in our universe.

Math isn't about some local deterministic property that depends on the interaction of simple parts but about the global patterns.

You can't have e.g. Lorenz Invariance on a quantized spacetime.  Of course, you can always argue that "the quantization is really small giving us approximate Lorenz Invariance", but this would be a claim not only without evidence, but actually against all of the evidence we have.

Similarly, the Schrodinger Wave Equation is defined mathematically as an operator evolving in continuous spacetime.  Obviously you can approximate it using a digital world (as we do on our computers), but the real world doesn't show any evidence of using such an approximation.

Okay, but if actual infinities are allowed, then what defines small in the "made up of small parts"? Like, would tiny ghosts be okay because they're "small"?

The Unreasonable Effectiveness of Math makes a predictable claim: models which can be represented using concise mathematical notation are more likely to be true, but this includes the whole mathematical universe.  

What part of the mathematical universe do you reject exactly?

I'm still trying to understand this quote:

In general I have not found anything which a mathematical universe explained which did not either contradict reductionism, or which could not be explained at least as well using the physical view of the world. I conclude from this that it's unlikely that there exists a mathematical universe.

And so far it sounds like you're fine with literal infinity.  So what part of a mathematical universe do you find distasteful?  Is it all infinities larger than $2^{{\aleph }_0}$ or the idea that "2" exists as an abstract idea apart from any physical model, or something else?

Maybe I'm just confused because I recently had an argument with someone who didn't believe in infinity.

When I pointed out all of physics is based on the assumption that spacetime is continuous (an example of infinity) his response was essentially "we'll fix that someday".

So, given that you deny "the mathematical universe", does that mean you think spacetime isn't continuous?  Or are "small physical parts" allowed to be infinitely subdivided?

I'm afraid that doesn't make your position any more clear to me.

The tautological belief that everything is made of small physical parts itself not a "small physical part", it is one of the most broad claims about the universe possible.

It seems to me that you belief in at least 3 truths axiomatically:

1. The universe has the convenient property that the outcomes of large physical systems can be predicted by small systems such as human minds.
2. All systems are made up of small physical parts
3. There are no other axioms besides 1. 2. and this one

Even if I accept 1. then 2. and 3. neither follow from 1. nor do they have additional predictive power that would cause me to accept them.  

In general I have not found anything which a mathematical universe explained which did not either contradict reductionism, or which could not be explained at least as well using the physical view of the world. I conclude from this that it's unlikely that there exists a mathematical universe.

‘My Model of Beff Jezos's Position: I don't care about this prediction of yours enough to say that I disagree with it. I'm happy so long as entropy increases faster than it otherwise would.

This isn't what Beff believes at all.

Maximizing entropy in E/Acc takes approximately the same place as maximizing money takes in Objectivism.  It is not inherently good, but it is a strong signal that you are on the right track.

In Objectivism, if you see someone promoting "for the benefit of all mankind", you can probably assume they are a villain.  In E/Acc if you see someone promoting deceleration, likewise.

I consider that noticing this tautology and deciding to ground my philosophy on it to be in the top 3 best decisions I've ever made.

Interesting that in a post claiming that everything can be deduced from "bottom up" reasoning you justify your conclusions using tautological axioms rather than bottom-up reasoning.  It's almost as though doing rationality requires one to make assumptions about the thing being reasoned about rather than deducing everything from the bottom up.

"We reverse engineered every single circuit and can predict exactly what the model will do using our hand-crafted code" seems like it's setting the bar way too high for MI.

Instead, aim for stuff like the AI Lie Detector, which both 1) works and 2) is obviously useful.

To do a Side-Channel Attack on a system, you don't have to explain every detail of the system (or even know physics at all).  You just have to find something in the system that is correlated with the thing you care about.

I think $5k is still high. Fundamentally these things aren't any more complex than a Roomba, and those sell for $100-$1000. If you want to add a cheap robot arm to it, maybe double the price.

"But those don't have the accuracy and reliability required for a  commercial environment"

Yeah, AI fixes this.

>Now imagine that Musk gets in trouble with the government

Now image the same scenario but  Elon has not gotten in trouble with the government and multiple people (including those who fired him) have affirmed he did nothing wrong.

Does an "ordinary layman's understanding of actual history" include knowledge of how tanks are used in combined arms warfare to create breakthroughs in "blitzkrieg" style warfare?  Seems like "don't attack until you have tanks and a good idea of how to use them in coordination with infantry and artillery, and also don't antagonize America" is sufficient for near certain victory.

In contrast, suppose you have a strong and knowledgeable multimodal predictor trained on all data humanity has available to it that can output arbitrary strings. Then apply extreme optimization pressure for never losing at chess. Now, the boundaries of the space in which the AI operates are much broader, and the kinds of behaviorist "values" the AI can have are far less constrained. It has the ability to route through the world, and with extreme optimization, it seems likely that it will.

"If we build AI in this particular way, it will be dangerous"

Okay, so maybe don't do that then.

Well, I claim that these are more-or-less the same fact. It's no surprise that the AI falls down on various long-horizon tasks and that it doesn't seem all that well-modeled as having "wants/desires"; these are two sides of the same coin.

It's weird that this sentence immediately follows you talking about AI being able to play chess.  A chess playing AI doesn't "want to win" in the behaviorist sense.  If I flip over the board or swap pieces mid game or simply refuse to move the AI's pieces on it's turn, it's not going to do anything to stop me because it doesn't "want" to win the game.  It doesn't even realize that a game is happening in the real world.  And yet it is able to make excellent long term plans about "how" to win at chess.

Either:
a) A chess playing AI fits into your definition of "want", in which case who cares if AI wants things, this tells us nothing about their real-world behavior.
b) A chess playing AI doesn't "want" to win (my claim) in which case AI can make long term plans without wanting.

Construction of overhead electric lines would be much more expensive in America than other countries, making those ROI estimates inaccurate.

I think you might be seriously underestimating 1.  Rail projects cost 50% more in the US (vs e.g France).

"reality is large" is a bad objection.

It's possible in principle to build a simulation that is literally indistinguishable from reality.  Say we only run the AI in simulation for 100million years, and there's a simulation overhead of 10x.  That should cost (100e6 ly)**3*(100e6 years) * 10 of our future lightcone.  This is a minuscule fraction of our actual future lightcone (9.4e10 ly) * (10^15 y)

A few better objections:

Simulating a universe with a paperclip maximizer in it means simulating billions of people being murdered and turned into paperclips.  If we believe computation=existence, that's hugely morally objectionable.

The AGI's prior that it is in a simulation doesn't depend on anything we do, only on the universal prior.

Since human values are not solely about power acquisition, evolution is continuously pushing the world away from them and towards states that are all about it, and our values fight an uphill battle against that. They are just a coincidence of their time, after all.

Assuming an awful lot of our conclusion today, aren't we?