Posts
Comments
We never got around to write more unfortunately.
I recommend this paper for a good overview of compMecb https://arxiv.org/abs/cond-mat/9907176
Crypticity, Reverse Epsilon Machines and the Arrow of Time?
[see https://arxiv.org/abs/0902.1209 ]
Our subjective experience of the arrow of time is occasionally suggested to be an essentially entropic phenomenon.
This sounds cool and deep but crashes headlong into the issue that the entropy rate and the excess entropy of any stochastic process is time-symmetric. I find it amusing that despite hearing this idea often from physicists and the like apparently this rather elementary fact has not prevented their storycrafting.
Luckily, computational mechanics provides us with a measure that is not time symmetric: the stochastic complexity of the epsilon machine
For any stochastic process we may also consider the epsilon machine of the reverse process, in other words the machine that predicts the past based on the future. This can be a completely different machine whose reverse stochastic complexity is not equal to .
Some processes are easier to predict forward than backward. For example, there is considerable evidence that language is such a process. If the stochastic complexity and the reverse stochastic complexity differ we speak of a causally assymetric process.
Alec Boyd pointed out to me that the classic example of a glass falling of a table is naturally thought of in these terms. The forward process is easy to describe while the backward process is hard to describe where easy and hard are meant in the sense of stochastic complexity: bits needed to specify the states of perfect minimal predictor, respectively retrodictor.
rk. note that time assymmetry is a fundamentally stochastic phenomenon. THe underlyiing (let's say classicially deterministic) laws are still time symmetric.
The hypothesis is then: many, most macroscopic processes of interest to humans, including other agents are fundamentally such causally assymetric (and cryptic) processes.
The cases of the Dutch republic and English development after the Civil war are highly confounded.
I wouldn't put too much stock in this kind of correlation.
I think I speak for many when I ask you to please elaborate on this!
What did Yudkoswky get right?
- The central problem of AI alignment. I am not aware of anything in subsequent work that is not already implicit in Yudkowsky's writing.
- Short timelines avant le lettre. Yudkowsky was predicting AGI in his lifetime from the very start when most academics, observers, AI scientists, etc considered AGI a fairytale.
- Inherent and irreducible uncertainty of forecasting, foolishness of precise predictions.
- The importance of (Pearlian) causality, Solomonoff Induction as theory of formal epistemology, Bayesian statistics, (Shannon) information theory, decision theory [especially UDT-shaped things].
- (?nanotech, ?cryonics)
- if you had a timemachine to go back to 2010 you should buy bitcoin and write Harry Potter fanfiction
Preach, brother.
One hundred twenty percent agreed. Hubris is the downfall of the rationalist project.
I'm usually a skeptic of the usefulness of this kind of speculation but I found this a good read. I am particularly intrigued hy the suggestion of decomposability of goals.
Excited to see this go live, Nick!
Played around with Pantheon for a couple minutes. I wrote a couple of lines but I didn't get any daemons yet. How long should I have to wait for them to pop up?
On the word 'theory'.
The word 'theory' is oft used and abused.
there is two ways 'theory' is used that are different and often lead to confusion.
Theory in thescientific sense
the way a physicist would use: it's a model of the world that is either right or wrong. there might be competing theories and we neeed to have empirical evidence to figure out which one's right. Ideally, they agree with empirical evidence or at least are highly falsifiable. Importantly, if two theories are to conflict they need to actually speak about the same variables, the same set of measurable quantities.
Theory in the mathematician' sense; a formal framework
There is a related but different notion of theory that a mathematician would use: a theory of groups, of differential equations, of randomness, of complex systems, of etc etc. This is more like a formal framework for a certain phenomenon or domain.
It defines what the quantities, variables, features one is interested in even are.
One often hears the question whether this (mathematical) theory makes testable predictions. This sounds sensible but doesn't really makes sense. It is akin to asking whether arithmetic or calculus makes testable predictions.*
Theories in the mathematician's sense can't really be wrong or right since (at least in theory) everything is proven. Of course, theories in this sense can fail to say much about the real world, they might bake in unrealistic assumptions of course etc.
Other uses of 'Theory'
The world 'theory' is also used in other disciplines. For instance, in literature studies where it is a denotes free form vacuous verbiage; or in ML where 'theory' it is used for uninformed speculation.
*one could argue that the theory of Peano Arithmetic actually does make predictions about natural numbers in the scientific sense, and more generally theories in the mathematical sense in a deep sense really are theories in the scientific sense. I think there is something to this but 1. it hasn't been developed yet 2. mostly irrelevant in the present context.
Whats FPT!=W[1] ?
I'm a computational complexity noob so apologies if this question is elementary.
I thought if one could solve one NP-complete problem then one can solve all of them. But you say that the treewidth doesn't help at all with the Clique problem. Is the parametrized complexity filtration by treewidth not preserved by equivalence between different NP-complete problems somehow?
To me, if this is true it exposes a much more serious flaw in computational complexity classes and does not resolve the issue of whether P vs NP matters in practice at all. My takeaway is that the definitions are too coarse, aren't really capturing what's going on. But likely I'm confused.
Thanks for writing this up Dalcy!
My knowledge of computational complexity is unfortunately lacking. I too wonder to what degree the central definitions of the complexity classes truly reflect the deep algorithmic structures or perhaps are a rather coarse shadow of more complex invariants.
You mention treewidth - are there other quantities of similar importance?
I also wonder about treewidth. I looked up the definition on wikipedia but struggle to find a good intuitive story what it really measures. Do you have an intuition pump about treewidth you could share?
Current work on Markov blankets and Boundaries on LW is flawed and outdated. State of the art should factor through this paper on Causal Blankets; https://iwaiworkshop.github.io/papers/2020/IWAI_2020_paper_22.pdf
A key problem for accounts of blankets and boundaries I have seen on LW so far is the following elementary problem (from the paper):
"Therefore, the MB [Markov Blanket] formalism forbids interdependencies induced by past events that are kept in memory, but may not directly influence the present state of the blankets.
Thanks to Fernando Rosas telling me about this paper.
@Vanessa Kosoy knows more.
My question for you is: in the world we live in, the full causal history of any real event contains almost the whole history of Earth from the time of the event backwards, because the Earth is so small relative to the speed of light, and everything that could have interacted with the event is part of the history of that event. So in practice, won't all counterfactable events need to be a more-or-less a full specification of the whole state of the world at a certain point in time?
I would argue that this is not in fact the case.
Our world is highly modular, much less entangled than other possible worlds [possibly because of anthropic reasons].
The way I think about it: in practice as you zoom in on a false counterfactual you will need to pay more and more bits for conflicting 'coincidences'.
interesting. who do you think feature space is like a simplicial complex ?
Thanks. I'm looking forward to your paper!
The 'surprise accounting' framework sounds quite a lot like the Minimum Description Length principle (e.g. here). Do you have any takes on how surprise accounting compares and differs vis a vis MDL?
Do I understand correctly that the main issue is finding ~ a canonical prior on the set of circuits?
Why does the OR-gate cost only 1 bit?
One argument I can see is that for any binary gate you only consider OR and AND gates. If we make the (semi-reasonable) assumption that these are equally likely then an OR gate cost 1 -bit?
However, you also have to describe which gate is to be described which takes more bits. You;d need ~ worth of bits to describe an arbitrary gate.
You might be able to formalize this using algorithmic information theory /K-complexity.
Blocked and reported to the ichor-blooded mods. You will pay dearly for your hubris.
As suggested by @Mateusz Bagiński it is tempting to suggest that the proper reading of a coproof of should be a proof of the double negation of , i.e. is a proof .
The absence of evidence against the negation of is weaker than a proof however. The former allows for new evidence to still appear while the latter categorically rejects this possibility on pain of contradiction.
Coproofs as countermodels
How could absence of evidence be interpreted logically? One way is to interpret it as the provision of a countermodel. That is - a coproof of would be a model such that .
Currently, the countermodel prevents a proof of and the more countermodels we find the more we might hypothesize that in all models and therefore not . On the other hand, we may find new evidence in the future that expands our knowledge database and excludes the would-be countermodel . This would open the way of a proof of in our new context.
Coproofs as sets of countermodels
We can go further by defining some natural (probability) distribution on the space of models .
This is generically a tricky business but given a finite signature of propositional letters over a classical base logic the space of models is given by "ultrafilters/models/truth assignments" which assign or to the basic propsitional letters and are extended to compound propositions in the usual manner (i.e. etc).
A subset of models can now be interpreted as a coproof of if for all .
Probability distributions on propositions and distributions on models
We might want to generalize subsets of models to (generalized) distributions of models. Any distribution on the set of models now induces a distribution on the set of propositions.
In the simple case above we could also make an invocation of the principle of indifference to define a natural uniform distributions on . This would assigns a proposition the ratio .
Rk. Note that similar ideas appear in the van Horn-Cox theorem.
arbitrary reference parameters .
what is an 'arbitrary reference parameter'? This is not in my vocabulary.
(and why do we need it? can't we just take the log here).
Why do people like big houses in the countryside /suburbs?
Empirically people move out to the suburbs/countryside when they get children and/or gain wealth. Having a big house with a large yard is the quintessential American dream.
but why? Dense cities are economoically more productive, commuting is measurably one of the worst factors for happiness and productivity. Raising kids in small houses is totally possible and people have done so at far higher densities in the past.
Yet people will spend vast amounts of money on living in a large house with lots of space - even if they rarely use most rooms. Having a big house is almost synonymous with wealth and status.
Part of the reason may be an evolved disease response. In the past, the most common way to die was as a child dieing to a crowd-disease. There was no medicine that actually worked yet wealthier people had much longer lifespans and out reproduced the poor (see Gregory Clark). The best way to buy health was to move out of the city (which were population sinks until late modernity) and live in a large aired house.
It seems like an appealing model. On the other hand, there are some obvious predicted regularities that aren't observed to my knowledge.
Never ? That's quite a bold prediction. Seems more likely than not that AI companies will be effectively nationalized. I'm curious why you think it will never happen.
yes !! discovered this last week - seems very important the quantitative regret bounds for approximatiions is especially promising
You are absolutely right and I am of course absolutely and embarrasingly wrong.
The minimal optimal predictor as a Hidden Markov Model of the simple nonunfilar is indeed infinite. This implies that any other architecture must be capable of expressing infinitely many states - but this is quite a weak statement - it's very easy for a machine to dynamically express finitely many states with finite memory. In particular, a transformer should absolutely be able to learn the MSP of the epsilon machine of the simple nonunifilar source - indeed it can even be solved analytically.
This was an embarrasing mistake I should not have made. I regret my rash overconfidence - I should have taken a moment to think it through since the statement was obviously wrong. Thank you for pointing it out.
We intend to review end of the submit deadline June 30th but I wouldn't hold off on your application.
Behold
You may be positively surprised to know I agree with you. :)
For context, the dialogue feature just came out on LW. We gave it a try and this was the result. I think we mostly concluded that the dialogue feature wasn't quite worth the effort. Anyway
I like what you're suggesting and would be open to do a dialogue about it !
Compare also the central conceit of QM /Koopmania. Take a classical nonlinear finite-dimensional system X described by a say a PDE. This is a dynamical system with evolution operator X -> X. Now look at the space H(X) of C/R-valued functions on the phase space of X. After completion we obtain an Hilbert space H. Now the evolution operator on X induces a map on H= H(X). We have now turned a finite-dimensional nonlinear problem into an infinite-dimensional linear problem.
Probably within.
I mostly regard LLMs = [scaling a feedforward network on large numbers of GPUs and data] as a single innovation.
One result to mention in computational complexity is the PCP theorem which not only gives probabilistically checkable proofs but also gives approximation case hardness. Seems deep but I haven't understood the proof yet.
My mainline prediction scenario for the next decades.
My mainline prediction * :
- LLMs will not scale to AGI. They will not spawn evil gremlins or mesa-optimizers. BUT Scaling laws will continue to hold and future LLMs will be very impressive and make a sizable impact on the real economy and science over the next decade.
- there is a single innovation left to make AGI-in-the-alex sense work, i.e. coherent, long-term planning agents (LTPA) that are effective and efficient in data sparse domains over long horizons.
- that innovation will be found within the next 10-15 years
- It will be clear to the general public that these are dangerous
- governments will act quickly and (relativiely) decisively to bring these agents under state-control. national security concerns will dominate.
- power will reside mostly with governments AI safety institutes and national security agencies. In so far as divisions of tech companies are able to create LTPAs they will be effectively nationalized.
- International treaties will be made to constrain AI, outlawing the development of LTPAs by private companies. Great power competition will mean US and China will continue developing LTPAs, possibly largely boxed. Treaties will try to constrain this development with only partial succes (similar to nuclear treaties).
- LLMs will continue to exist and be used by the general public
- Conditional on AI ruin the closest analogy is probably something like the Cortez-Pizarro-Afonso takeovers. Unaligned AI will rely on human infrastructure and human allies for the earlier parts of takeover - but its inherent advantages in tech, coherence, decision-making and (artificial) plagues will be the deciding factor.
- The world may be mildly multi-polar.
- This will involve conflict between AIs.
- AIs very possible may be able to cooperate in ways humans can't.
- The arrival of AGI will immediately inaugurate a scientific revolution. Sci-fi sounding progress like advanced robotics, quantum magic, nanotech, life extension, laser weapons, large space engineering, cure of many/most remaining diseases will become possible within two decades of AGI, possibly much faster.
- Military power will shift to automated manufacturing of drones & weaponized artificial plagues. Drones, mostly flying will dominate the battlefield. Mass production of drones and their rapid and effective deployment in swarms will be key to victory.
Two points on which I differ with most commentators: (i) I believe AGI is a real (mostly discrete) thing , not a vibe, or a general increase of improved tools. I believe it is inherently agenctic. I don't think spontaneous emergence of agents is impossible but I think it is more plausible agents will be built rather than grown.
(ii) I believe in general the ea/ai safety community is way overrating the importance of individual tech companies vis a vis broader trends and the power of governments. I strongly agree with Stefan Schubert's take here on the latent hidden power of government: https://stefanschubert.substack.com/p/crises-reveal-centralisation
Consequently, the ea/ai safety community is often myopically focusing on boardroom politics that are relativily inconsequential in the grand scheme of things.
*where by mainline prediction I mean the scenario that is the mode of what I expect. This is the single likeliest scenario. However, since it contains a large number of details each of which could go differently, the probability on this specific scenario is still low.
Why no prediction markets for large infrastructure projects?
Been reading this excellent piece on why prediction markets aren't popular. They say that without subsidies prediction markets won't be large enough; the information value of prediction markets is often nog high enough.
Large infrastructure projects undertaken by governments, and other large actors often go overbudget, often hilariously so: 3x,5x,10x or more is not uncommon, indeed often even the standard.
One of the reasons is that government officials deciding on billion dollar infrastructure projects don't have enough skin in the game. Politicians are often not long enough in office to care on the time horizons of large infrastructure projects. Contractors don't gain by being efficient or delivering on time. To the contrary, infrastructure projects are huge cashcows. Another problem is that there are often far too many veto-stakeholders. All too often the initial bid is wildly overoptimistic.
Similar considerations apply to other government projects like defense procurement or IT projects.
Okay - how to remedy this situation? Internal prediction markets theoretically could prove beneficial. All stakeholders & decisionmakers are endowed with vested equity with which they are forced to bet on building timelines and other key performance indicators. External traders may also enter the market, selling and buying the contracts. The effective subsidy could be quite large. Key decisions could save billions.
In this world, government officials could gain a large windfall which may be difficult to explain to voters. This is a legitimate objection.
A very simple mechanism would simply ask people to make an estimate on the cost C and the timeline T for completion. Your eventual payout would be proportional to how close you ended up to the real C,T compared to the other bettors. [something something log scoring rule is proper].
I don't know what you mean by 'general intelligence' exactly but I suspect you mean something like human+ capability in a broad range of domains. I agree LLMs will become generally intelligent in this sense when scaled, arguably even are, for domains with sufficient data. But that's kind of the sticker right? Cave men didn't have the whole internet to learn from yet somehow did something that not even you seem to claim LLMs will be able to do: create the (date of the) Internet.
(Your last claim seems surprising. Pre-2014 games don't have close to the ELO of alphaZero. So a next-token would be trained to simulate a human player up tot 2800, not 3200+. )
I would be genuinely surprised if training a transformer on the pre2014 human Go data over and over would lead it to spontaneously develop alphaZero capacity. I would expect it to do what it is trained to: emulate / predict as best as possible the distribution of human play. To some degree I would anticipate the transformer might develop some emergent ability that might make it slightly better than Go-Magnus - as we've seen in other cases - but I'd be surprised if this would be unbounded. This is simply not what the training signal is.
Could you train an LLM on pre 2014 Go games that could beat AlphaZero?
I rest my case.
In my mainline model there are only a few innovations needed, perhaps only a single big one to product an AGI which just like the Turing Machine sits at the top of the Chomsky Hierarchy will be basically the optimal architecture given resource constraints. There are probably some minor improvements todo with bridging the gap between theoretically optimal architecture and the actual architecture, or parts of the algorithm that can be indefinitely improved but with diminishing returns (these probably exist due to Levin and possibly.matrix.multiplication is one of these). On the whole I expect AI research to be very chunky.
Indeed, we've seen that there was really just one big idea to all current AI progress: scaling, specifically scaling GPUs on maximally large undifferentiated datasets. There were some minor technical innovations needed to pull this off but on the whole that was the clinger.
Of course, I don't know. Nobody knows. But I find this the most plausible guess based on what we know about intelligence, learning, theoretical computer science and science in general.
My timelines were not 2026. In fact, I made bets against doomers 2-3 years ago, one will resolve by next year.
I agree iterative improvements are significant. This falls under "naive extrapolation of scaling laws".
By nanotech I mean something akin to drexlerian nanotech or something similarly transformative in the vicinity. I think it is plausible that a true ASI will be able to make rapid progress (perhaps on the order of a few years or a decade) on nanotech. I suspect that people that don't take this as a serious possibility haven't really thought through what AGI/ASI means + what the limits and drivers of science and tech really are; I suspect they are simply falling prey to status-quo bias.
Can somebody explain to me what's happening in this paper ?
Beautifully illustrated and amusingly put, sir!
A variant of what you are saying is that AI may once and for all allow us to calculate the true counterfactual Shapley value of scientific contributions.
( re: ancestor simulations
I think you are onto something here. Compare the Q hypothesis:
https://twitter.com/dalcy_me/status/1780571900957339771
see also speculations about Zhuangzi hypothesis here )
Why do you think there are these low-hanging algorithmic improvements?
I didn't intend the causes to equate to direct computation of \phi(x) on the x_i. They are rather other pieces of evidence that the powerful agent has that make it believe \phi(x_i). I don't know if that's what you meant.
I agree seeing x_i such that \phi(x_i) should increase credence in \forall x \phi(x) even in the presence of knowledge of C_j. And the Shapely value proposal will do so.
(Bad tex. On my phone)
Problem of Old Evidence, the Paradox of Ignorance and Shapley Values
Paradox of Ignorance
Paul Christiano presents the "paradox of ignorance" where a weaker, less informed agent appears to outperform a more powerful, more informed agent in certain situations. This seems to contradict the intuitive desideratum that more information should always lead to better performance.
The example given is of two agents, one powerful and one limited, trying to determine the truth of a universal statement ∀x:ϕ(x) for some Δ0 formula ϕ. The limited agent treats each new value of ϕ(x) as a surprise and evidence about the generalization ∀x:ϕ(x). So it can query the environment about some simple inputs x and get a reasonable view of the universal generalization.
In contrast, the more powerful agent may be able to deduce ϕ(x) directly for simple x. Because it assigns these statements prior probability 1, they don't act as evidence at all about the universal generalization ∀x:ϕ(x). So the powerful agent must consult the environment about more complex examples and pay a higher cost to form reasonable beliefs about the generalization.
Is it really a problem?
However, I argue that the more powerful agent is actually justified in assigning less credence to the universal statement ∀x:ϕ(x). The reason is that the probability mass provided by examples x₁, ..., xₙ such that ϕ(xᵢ) holds is now distributed among the universal statement ∀x:ϕ(x) and additional causes Cⱼ known to the more powerful agent that also imply ϕ(xᵢ). Consequently, ∀x:ϕ(x) becomes less "necessary" and has less relative explanatory power for the more informed agent.
An implication of this perspective is that if the weaker agent learns about the additional causes Cⱼ, it should also lower its credence in ∀x:ϕ(x).
More generally, we would like the credence assigned to propositions P (such as ∀x:ϕ(x)) to be independent of the order in which we acquire new facts (like xᵢ, ϕ(xᵢ), and causes Cⱼ).
Shapley Value
The Shapley value addresses this limitation by providing a way to average over all possible orders of learning new facts. It measures the marginal contribution of an item (like a piece of evidence) to the value of sets containing that item, considering all possible permutations of the items. By using the Shapley value, we can obtain an order-independent measure of the contribution of each new fact to our beliefs about propositions like ∀x:ϕ(x).
Further thoughts
I believe this is closely related, perhaps identical, to the 'Problem of Old Evidence' as considered by Abram Demski.
Suppose a new scientific hypothesis, such as general relativity, explains a well-know observation such as the perihelion precession of mercury better than any existing theory. Intuitively, this is a point in favor of the new theory. However, the probability for the well-known observation was already at 100%. How can a previously-known statement provide new support for the hypothesis, as if we are re-updating on evidence we've already updated on long ago? This is known as the problem of old evidence, and is usually levelled as a charge against Bayesian epistemology.
[Thanks to @Jeremy Gillen for pointing me towards this interesting Christiano paper]
Those numbers don't really accord with my experience actually using gpt-4. Generic prompting techniques just don't help all that much.
I've never done explicit timelines estimates before so nothing to compare to. But since it's a gut feeling anyway, I'm saying my gut is lengthening.
Yes agreed.
What I don't get about this position: If it was indeed just scaling - what's AI research for ? There is nothing to discover, just scale more compute. Sure you can maybe improve the speed of deploying compute a little but at the core of it it seems like a story that's in conflict with itself?
You may be right. I don't know of course.
At this moment in time, it seems scaffolding tricks haven't really improved the baseline performance of models that much. Overwhelmingly, the capability comes down to whether the rlfhed base model can do the task.
To some degree yes, they were not guaranteed to hold. But by that point they held for over 10 OOMs iirc and there was no known reason they couldn't continue.
This might be the particular twitter bubble I was in but people definitely predicted capabilities beyond simple extrapolation of scaling laws.
My timelines are lengthening.
I've long been a skeptic of scaling LLMs to AGI *. To me I fundamentally don't understand how this is even possible. It must be said that very smart people give this view credence. davidad, dmurfet. on the other side are vanessa kosoy and steven byrnes. When pushed proponents don't actually defend the position that a large enough transformer will create nanotech or even obsolete their job. They usually mumble something about scaffolding.
I won't get into this debate here but I do want to note that my timelines have lengthened, primarily because some of the never-clearly-stated but heavily implied AI developments by proponents of very short timelines have not materialized. To be clear, it has only been a year since gpt-4 is released, and gpt-5 is around the corner, so perhaps my hope is premature. Still my timelines are lengthening.
A year ago, when gpt-3 came out progress was blindingly fast. Part of short timelines came from a sense of 'if we got surprised so hard by gpt2-3, we are completely uncalibrated, who knows what comes next'.
People seemed surprised by gpt-4 in a way that seemed uncalibrated to me. gpt-4 performance was basically in line with what one would expect if the scaling laws continued to hold. At the time it was already clear that the only really important driver was compute data and that we would run out of both shortly after gpt-4. Scaling proponents suggested this was only the beginning, that there was a whole host of innovation that would be coming. Whispers of mesa-optimizers and simulators.
One year in: Chain-of-thought doesn't actually improve things that much. External memory and super context lengths ditto. A whole list of proposed architectures seem to serve solely as a paper mill. Every month there is new hype about the latest LLM or image model. Yet they never deviate from expectations based on simple extrapolation of the scaling laws. There is only one thing that really seems to matter and that is compute and data. We have about 3 more OOMs of compute to go. Data may be milked another OOM.
A big question will be whether gpt-5 will suddenly make agentGPT work ( and to what degree). It would seem that gpt-4 is in many ways far more capable than (most or all) humans yet agentGPT is curiously bad.
All-in-all AI progress** is developing according to the naive extrapolations of Scaling Laws but nothing beyond that. The breathless twitter hype about new models is still there but it seems to be believed more at a simulacra level higher than I can parse.
Does this mean we'll hit an AI winter? No. In my model there may be only one remaining roadblock to ASI (and I suspect I know what it is). That innovation could come at anytime. I don't know how hard it is, but I suspect it is not too hard.
* the term AGI seems to denote vastly different things to different people in a way I find deeply confusing. I notice that the thing that I thought everybody meant by AGI is now being called ASI. So when I write AGI, feel free to substitute ASI.
** or better, AI congress
addendum: since I've been quoted in dmurfet's AXRP interview as believing that there are certain kinds of reasoning that cannot be represented by transformers/LLMs I want to be clear that this is not really an accurate portrayal of my beliefs. e.g. I don't think transformers don't truly understand, are just a stochastic parrot, or in other ways can't engage in the abstract reasoning that humans do. I think this is clearly false, as seen by interacting with any frontier model.