My mainline prediction scenario for the next decades.

My mainline prediction * :

• LLMs will not scale to AGI. They will not spawn evil gremlins or mesa-optimizers. BUT Scaling laws will continue to hold and future LLMs will be very impressive and make a sizable impact on the real economy and science over the next decade. EDIT: since there is a lot of confusion about this point. BY LLM I mean the paradigm of pre-trained transformers. This does not include different paradigms that follow pre-trained transformers but are still called large language models.  EDIT2: since I'm already anticipating confusion on this point: when I say scaling laws will continue to hold that means that the 3-way relation between model size, compute, data will probably continue to hold. It has been known for a long time that amount of data used by gpt-4 level models is already within perhaps an OOM of the maximum. ]
• there is a single innovation left to make AGI-in-the-alex sense work, i.e. coherent, long-term planning agents (LTPA) that are effective and efficient in data sparse domains over long horizons.
• that innovation will be found within the next 10-15 years
• It will be clear to the general public that these are dangerous
• governments will act quickly and (relativiely) decisively to  bring these agents under state-control. national security concerns will dominate.
• power will reside mostly with governments AI safety institutes and national security agencies. In so far as divisions of tech companies are able to create LTPAs they will be effectively nationalized.
• International treaties will be made to constrain AI, outlawing the development of LTPAs by private companies. Great power competition will mean US and China will continue developing LTPAs, possibly largely boxed. Treaties will try to constrain this development with only partial succes (similar to nuclear treaties).
• LLMs will continue to exist and be used by the general public
• Conditional on AI ruin the closest analogy is probably something like the Cortez-Pizarro-Afonso takeovers [LW · GW]. Unaligned AI will rely on human infrastructure and human allies for the earlier parts of takeover - but its inherent advantages in tech, coherence, decision-making and (artificial) plagues will be the deciding factor.
• The world may be mildly multi-polar.
  • This will involve conflict between AIs.
  • AIs very possible may be able to cooperate in ways humans can't.
• The arrival of AGI will immediately inaugurate a scientific revolution. Sci-fi sounding progress like advanced robotics, quantum magic, nanotech, life extension, laser weapons, large space engineering, cure of many/most remaining diseases will become possible within two decades of AGI, possibly much faster.
• Military power will shift to automated manufacturing of drones &  weaponized artificial plagues. Drones, mostly flying will dominate the battlefield. Mass production of drones and their rapid and effective deployment in swarms will be key to victory.

Two points on which I differ with most commentators: (i) I believe AGI is a real (mostly discrete) thing , not a vibe, or a general increase of improved tools. I believe it is inherently agenctic. I don't think spontaneous emergence of agents is impossible but I think it is more plausible agents will be built rather than grown. 

(ii) I believe in general the ea/ai safety community is way overrating the importance of individual tech companies vis a vis broader trends and the power of governments. I strongly agree with Stefan Schubert's take here on the latent hidden power of government: https://stefanschubert.substack.com/p/crises-reveal-centralisation

Consequently, the ea/ai safety community is often myopically focusing on boardroom politics that are relativily inconsequential in the grand scheme of things. 

*where by mainline prediction I mean the scenario that is the mode of what I expect. This is the single likeliest scenario. However, since it contains a large number of details each of which could go differently, the probability on this specific scenario is still low. 

Misgivings about Category Theory

[No category theory is required to read and understand this screed]

A week does not go by without somebody asking me what the best way to learn category theory is. Despite it being set to mark its 80th annivesary, Category Theory has the evergreen reputation for being the Hot New Thing, a way to radically expand the braincase of the user through an injection of abstract mathematics. Its promise is alluring, intoxicating for any young person desperate to prove they are the smartest kid on the block.

Recently, there has been significant investment and attention focused on the intersection of category theory and AI, particularly in AI alignment research. Despite the influx of interest I am worried that it is not entirely understood just how big the theory-practice gap is.

 I am worried that overselling risks poisoning the well for the general concept of advanced mathematical approaches to science in general, and AI alignment in particular. As I believe mathematically grounded approaches to AI alignment are perhaps the only way to get robust worst-case safety guarantees for the superintelligent regime I think this would be bad. 

I find it difficult to write this. I am a big believer in mathematical approaches to AI alignment, working for one organization (Timaeus) betting on this and being involved with a number of other groups. I have many friends within the category theory community, I have even written an abstract nonsense paper myself, I am sympathetic to the aims and methods of the category theory community. This is all to say: I'm an insider, and my criticisms come from a place of deep familiarity with both the promise and limitations of these approaches.

A Brief History of Category Theory

‘Before functoriality Man lived in caves’ - Brian Conrad

Category theory is a branch of pure mathematics notorious for its extreme abstraction, affectionately derided as 'abstract nonsense' by its practitioners.

Category theory's key strength lies in its ability to 'zoom out' and identify analogies between different fields of mathematics and different techniques. This approach enables mathematicians to think 'structurally', viewing mathematical concepts in terms of their relationships and transformations rather than their intrinsic properties.

Modern mathematics is less about solving problems within established frameworks and more about designing entirely new games with their own rules. While school mathematics teaches us to be skilled players of pre-existing mathematical games, research mathematics requires us to be game designers, crafting rule systems that lead to interesting and profound consequences. Category theory provides the meta-theoretic tools for this game design, helping mathematicians understand which definitions and structures will lead to rich and fruitful theories.

“I can illustrate the second approach with the same image of a nut to be opened.

The first analogy that came to my mind is of immersing the nut in some softening liquid, and why not simply water? From time to time you rub so the liquid penetrates better,and otherwise you let time pass. The shell becomes more flexible through weeks and months – when the time is ripe, hand pressure is enough, the shell opens like a perfectly ripened avocado!

A different image came to me a few weeks ago.

The unknown thing to be known appeared to me as some stretch of earth or hard marl, resisting penetration… the sea advances insensibly in silence, nothing seems to happen, nothing moves, the water is so far off you hardly hear it.. yet it finally surrounds the resistant substance.

“ - Alexandre Grothendieck

The Promise of Compositionality and ‘Applied category theory’

Recently a new wave of category theory has emerged, dubbing itself ‘applied category theory’. 

Applied category theory, despite its name, represents less an application of categorical methods to other fields and more a fascinating reverse flow: problems from economics, physics, social sciences, and biology have inspired new categorical structures and theories. Its central innovation lies in pushing abstraction even further than traditional category theory, focusing on the fundamental notion of compositionality - how complex systems can be built from simpler parts. 

The idea of compositionality has long been recognized as crucial across sciences, but it lacks a strong mathematical foundation. Scientists face a universal challenge: while simple systems can be understood in isolation, combining them quickly leads to overwhelming complexity. In software engineering, codebases beyond a certain size become unmanageable. In materials science, predicting bulk properties from molecular interactions remains challenging. In economics, the gap between microeconomic and macroeconomic behaviours persists despite decades of research.

Here then lies the great promise: through the lens of categorical abstraction, the tools of reductionism might finally be extended to complex systems. The dream is that, just as thermodynamics has been derived from statistical physics, macroeconomics could be systematically derived from microeconomics. Category theory promises to provide the mathematical language for describing how complex systems emerge from simpler components.

How has this promise borne out so far? On a purely scientific level, applied category theorists have uncovered a vast landscape of compositional patterns. In a way, they are building a giant catalogue, a bestiary, a periodic table not of ‘atoms’ (=simple things) but of all the different ways ‘atoms' can fit together into molecules (=complex systems). 

Not surprisingly, it turns out that compositional systems have an almost unfathomable diversity of behavior. The fascinating thing is that this diversity, while vast, isn't irreducibly complex - it can be packaged, organized, and understood using the arcane language of category theory. To me this suggests the field is uncovering something fundamental about how complexity emerges.

How close is category theory to real-world applications?

Are category theorists very smart? Yes. The field attracts and demands extraordinary mathematical sophistication. But intelligence alone doesn't guarantee practical impact.

It can take many decades for basic science to yield real-world applications - neural networks themselves are a great example. I am bullish in the long-term that category theory will prove important scientifically. But at present the technology readiness level isn’t there.  

There are prototypes. There are proofs of concept. But there are no actual applications in the real world beyond a few trials. The theory-practice gap remains stubbornly wide.

The principality of mathematics is truly vast. If categorical approaches fail to deliver on their grandiose promises I am worried it will poison the well for other theoretic approaches as well, which would be a crying shame.   

The Padding Argument or Simplicity = Degeneracy

[I learned this argument from Lucius Bushnaq and Matthias Dellago. It is also latent already in Solomonoff's original work]

Consider binary strings of a fixed length $\{0,1{\}}^L$ 

Imagine feeding these strings into some turing machine; we think of strings as codes for a function. Suppose we have a function that can be coded by a short compressed string $s$ of length $k<<L$. That is, the function is computable by a small program. 

Imagine uniformly sampling a random code for $\{0,1{\}}^L$ . What number of the codes implement the same function as the string $s$? It's close to $2^{L-k}$. Indeed, given the string $s$ of length $k$  we can 'pad' it to a string of length $L$ by writing the code

"run $s$ skip $t$ "

where $t$ is an arbitrary string of length $L-k-c$ where $c$ is a small constant accounting for the overhead. There are approximately $2^{L-k}$ of such binary strings. If our programming language has a simple skip / commenting out functionality then we expect approximately $2^{L-k}$ codes encoding the same function as $s$. The fraction of all codes encoding s is 2^-k. 

I find this truly remarkable: the degeneracy or multiplicity is inversely exponentially proportional to the minimum description length of the function! 

Just by sampling codes uniformly at random we get the Simplicity prior!!

Why do Neural Networks work? Why do polynomials not work?

It is sometimes claimed that neural networks work well because they are 'Universal Approximators'. There are multiple problems with this explanation, see e.g. here [LW · GW] but a very basic problem is that being a universal approximaton is very common. Polynomials are universal approximators!

Many different neural network architectures work. In the limit of large data, compute the difference of different architectures start to vanish and very general scaling laws dominate. This is not the case for polynomials.  

Degeneracy=Simplicity explains why: polynomials are uniquely tied down by their coefficients, so a learning machine that tries to fit polynomials is does not have a 'good' simplicity bias that approximates the Solomonoff prior. 

The lack of degeneracy applies to any set of functions that form an orthogonal basis. This is because the decomposition is unique. So there is no multiplicity and no implicit regularization/ simplicity bias. 

[I learned this elegant argument from Lucius Bushnaq.]

The Singular Learning Theory and Algorithmic Information Theory crossover 

I described the padding argument as an argument not a proof. That's because technically it only gives a lower bound on the number of codes equivalent to the minimal description code. The problem is there are pathological examples where the programming language (e.g. the UTM) hardcodes that all small codes $s$ encode a single function $f$. 

When we take this problem into account the Padding Argument is already in Solomonoff's original work. There is a theorem that states that the Solomonoff prior is equivalent to taking a suitable Universal Turing Machine and feeding in a sequence of (uniformly) random bits and taking the resulting distribution. To account for the pathological examples above everything is asymptotic and up to some constant like all results in algorithmic information theory. This means that like all other results in algorithmic information theory it's unclear whether it is at all relevant in practice.

However, while this gives a correct proof I think this understates the importance of the Padding argument to me. That's because I think in practice we shouldn't expect the UTM to be pathological in this way. In other words, we should heuristically expect the simplicity $K\left(f\right)$ to be basically proportional to the fraction of codes yielding $f$ for a large enough (overparameterized) architecture. 

The bull case for SLT is now: there is a direct equality between algorithmic complexity and the degeneracy. This has always been SLT dogma of course but until I learned about this argument it wasn't so clear to me how direct this connection was. The algorithmic complexity can be usefully approximated by the (local) learning coefficient $\lambda$!

EDIT: see Clift-Murfet-Wallbridge and Tom Warings thesis for more. See below, thanks Dan

The bull case for algorithmic information: the theory of algorithmic information, Solomonoff induction, AIXI etc is very elegant and in some sense gives answers to fundamental questions we would like to answer. The major problem was that it is both uncomputable and seemingly intractable. Uncomputability is perhaps not such a problem - uncomputability often arises from measure zero highly adversarial examples. But tractability is very problematic. We don't know how tractable compression is, but it's likely untractable. However, the Padding argument suggests that we should heuristically expect the simplicity $K\left(f\right)$ to be basically proportional to the fraction of codes yielding $f$ for a large enough (overparameterized) architecture - in other words it can be measured by the local Learning coefficient.

Do Neural Networks actually satisfy the Padding argument?

Short answer: No. 

Long answer: Unclear. maybe... sort of... and the difference might itself be very interesting...!

Stay tuned. 

How to prepare for the coming Taiwan Crisis? Should one short TSMC? Dig a nuclear cellar?

Metaculus gives a 25% of a fullscale invasion of Taiwan within 10 years and a 50% chance of a blockade. It gives a 65% chance that if China invades Taiwan before 2035 the US will respond with military force. 

Metaculus has very strong calibration scores (apparently better than prediction markets). I am inclined to take these numbers as the best guess we currently have of the situation. 

Is there any way to act on this information?

Novel Science is Inherently Illegible

Legibility, transparency, and open science are generally considered positive attributes, while opacity, elitism, and obscurantism are viewed as negative. However, increased legibility in science is not always beneficial and can often be detrimental.

Scientific management, with some exceptions, likely underperforms compared to simpler heuristics such as giving money to smart people or implementing grant lotteries. Scientific legibility suffers from the classic "Seeing like a State" problems. It constrains endeavors to the least informed stakeholder, hinders exploration, inevitably biases research to be simple and myopic, and exposes researchers to constant political tug-of-war between different interest groups poisoning objectivity. 

I think the above would be considered relatively uncontroversial in EA circles.  But I posit there is something deeper going on: 

Novel research is inherently illegible. If it were legible, someone else would have already pursued it. As science advances her concepts become increasingly counterintuitive and further from common sense. Most of the legible low-hanging fruit has already been picked, and novel research requires venturing higher into the tree, pursuing illegible paths with indirect and hard-to-foresee impacts.

Why don't animals have guns? 

Or why didn't evolution evolve the Hydralisk?

Evolution has found (sometimes multiple times) the camera, general intelligence, nanotech, electronavigation, aerial endurance better than any drone, robots more flexible than any human-made drone, highly efficient photosynthesis, etc. 

First of all let's answer another question: why didn't evolution evolve the wheel like the alien wheeled elephants in His Dark Materials?

Is it biologically impossible to evolve?

Well, technically, the flagella of various bacteria is a proper wheel.

No the likely answer is that wheels are great when you have roads and suck when you don't. Roads are build by ants to some degree but on the whole probably don't make sense for an animal-intelligence species. 

Aren't there animals that use projectiles?

Hold up. Is it actually true that there is not a single animal with a gun, harpoon or other projectile weapon?

Porcupines have quils, some snakes spit venom, a type of fish spits water as a projectile to kick insects of leaves than eats insects. Bombadier beetles can produce an explosive chemical mixture. Skunks use some other chemicals. Some snails shoot harpoons from very close range. There is a crustacean that can snap its claw so quickly it creates a shockwave stunning fish. Octopi use ink.  Goliath birdeater spider shoot hair. Electric eels shoot electricity etc. 

Maybe there isn't an incentive gradient? The problem with this argument is that the same argument can be made for lots and lots of abilities that animals have developed, often multiple times. Flight, camera, a nervous system. 

But flight has an intermediate form: glider monkeys, flying squirrels, flying fish. 

Except, I think there are lots of intermediate forms for guns & harpoons too:

There are animals with quills. It's only a small number of steps from having quils that you release when attack to actively shooting and aiming these quils. Why didn't Evolution evolve Hydralisks? For many other examples - see the list above. 

In a Galaxy far far away

I think it is plausible that the reason animals don't have guns is simply an accident. Somewhere in the vast expanses of space circling a dim sun-like star the water-bearing planet Hiram Maxim is teeming with life. Nothing like an intelligent species has yet evolved yet it's many lifeforms sport a wide variety of highly effective projectile weapons. Indeed, the majority of larger lifeforms have some form of projective weapon as a result of the evolutionary arms race. The savannahs sport gazelle-like herbivores evading sniper-gun equppied predators. 

Some many parsecs away is the planet Big Bertha, a world is embroilled in permanent biological trench warfare. More than 95% percent of the biomass of animals larger than a mouse is taken up by members of just 4 geni of eusocial gun-equipped species or their domesticastes. Yet the individual intelligence of members of these species doesn't exceed that of a cat. 

The largest of the four geni builds massive dams like beavers, practices husbandry of various domesticated species, agriculture and engages in massive warfare against rival colonies using projectile harpoons that grow from their limbs. Yet all of this is biological, not technological: the behaviours and abilites are evolved rather than learned. There is not a single species whose intelligence rivals that of a Great ape, either individually or collectively. 

Shower thought - why are sunglasses cool ?

Sunglasses create an asymmetry in the ability to discern emotions between the wearer and nonwearer. This implicitly makes the wearer less predictable, more mysterious, more dangerous and therefore higher in a dominance hierarchy.

↑ comment by Nina Panickssery (NinaR) · 2024-10-21T01:58:14.116Z · LW(p) · GW(p)

Isn't this already the commonly-accepted reason why sunglasses are cool?

Anyway, Claude agrees with you (see 1 and 3)

Replies from: alexander-gietelink-oldenziel

↑ comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) · 2024-10-21T07:53:20.567Z · LW(p) · GW(p)

yes very lukewarm take

also nice product placement nina

My timelines are lengthening. 

I've long been a skeptic of scaling LLMs to AGI *. To me I fundamentally don't understand how this is even possible. It must be said that very smart people give this view credence. davidad, dmurfet. on the other side are vanessa kosoy and steven byrnes. When pushed proponents don't actually defend the position that a large enough transformer will create nanotech or even obsolete their job. They usually mumble something about scaffolding.

I won't get into this debate here but I do want to note that my timelines have lengthened, primarily because some of the never-clearly-stated but heavily implied AI developments by proponents of very short timelines have not materialized. To be clear, it has only been a year since gpt-4 is released, and gpt-5 is around the corner, so perhaps my hope is premature. Still my timelines are lengthening. 

A year ago, when gpt-3 came out progress was blindingly fast. Part of short timelines came from a sense of 'if we got surprised so hard by gpt2-3, we are completely uncalibrated, who knows what comes next'.

People seemed surprised by gpt-4 in a way that seemed uncalibrated to me. gpt-4 performance was basically in line with what one would expect if the scaling laws continued to hold. At the time it was already clear that the only really important driver was compute  data and that we would run out of both shortly after gpt-4. Scaling proponents suggested this was only the beginning, that there was a whole host of innovation that would be coming. Whispers of mesa-optimizers and simulators. 

One year in: Chain-of-thought doesn't actually improve things that much. External memory and super context lengths ditto. A whole list of proposed architectures seem to serve solely as a paper mill. Every month there is new hype about the latest LLM or image model. Yet they never deviate from expectations based on simple extrapolation of the scaling laws. There is only one thing that really seems to matter and that is compute and data. We have about 3 more OOMs of compute to go. Data may be milked another OOM. 

A big question will be whether gpt-5 will suddenly make agentGPT work ( and to what degree). It would seem that gpt-4 is in many ways far more capable than (most or all) humans yet agentGPT is curiously bad. 

All-in-all AI progress** is developing according to the naive extrapolations of Scaling Laws but nothing beyond that. The breathless twitter hype about new models is still there but it seems to be believed more at a simulacra level higher than I can parse. 

Does this mean we'll hit an AI winter? No. In my model there may be only one remaining roadblock to ASI (and I suspect I know what it is). That innovation could come at anytime. I don't know how hard it is, but I suspect it is not too hard. 

* the term AGI seems to denote vastly different things to different people in a way I find deeply confusing. I notice that the thing that I thought everybody meant by AGI is now being called ASI. So when I write AGI, feel free to substitute ASI. 

** or better, AI congress

addendum:  since I've been quoted in dmurfet's AXRP interview as believing that there are certain kinds of reasoning that cannot be represented by transformers/LLMs I want to be clear that this is not really an accurate portrayal of my beliefs. e.g. I don't think transformers don't truly understand, are just a stochastic parrot, or in other ways can't engage in the abstract reasoning that humans do. I think this is clearly false, as seen by interacting with any frontier model. 

Encrypted Batteries 

(I thank Dmitry Vaintrob for the idea of encrypted batteries. Thanks to Adam Scholl for the alignment angle. Thanks to the Computational Mechanics at the receent compMech conference. )

There are no Atoms in the Void just Bits in the Description. Given the right string a Maxwell Demon transducer can extract energy from a heatbath. 

Imagine a pseudorandom heatbath + nano-Demon. It looks like a heatbath from the outside but secretly there is a private key string that when fed to the nano-Demon allows it to extra lots of energy from the heatbath. 

P.S. Beyond the current ken of humanity lies a generalized concept of free energy that describes the generic potential ability or power of an agent to achieve goals. Money, the golden calf of Baal is one of its many avatars. Could there be ways to encrypt generalized free energy batteries to constraint the user to only see this power for good? It would be like money that could be only spent on good things. 

AGI companies merging within next 2-3 years inevitable?

There are currently about a dozen major AI companies racing towards AGI with many more minor AI companies. The way the technology shakes out this seems like unstable equilibrium. 

It seems by now inevitable that we will see further mergers, joint ventures - within two years there might only be two or three major players left. Scale is all-dominant. There is no magic sauce, no moat. OpenAI doesn't have algorithms that her competitors can't copy within  6-12 months. It's all leveraging compute. Whatever innovations smaller companies make can be easily stolen by tech giants. 

e.g. we might have xAI- Meta, Anthropic- DeepMind-SSI-Google, OpenAI-Microsoft-Apple. 

Actuallly, although this would be deeply unpopular in EA circles it wouldn't be all that surprising if Anthropic and OpenAI would team up. 

And - of course - a few years later we might only have two competitors: USA, China. 

EDIT: the obvious thing to happen is that nvidia realizes it can just build AI itself. if Taiwan is Dune, GPUs are the spice, then nvidia is house Atreides

Current work on Markov blankets and Boundaries on LW is flawed and outdated. State of the art should factor through this paper on Causal Blankets; https://iwaiworkshop.github.io/papers/2020/IWAI_2020_paper_22.pdf

A key problem for accounts of blankets and boundaries I have seen on LW so far is the following elementary problem (from the paper):
"Therefore, the MB [Markov Blanket] formalism forbids interdependencies induced by past events that are kept in memory, but may not directly influence the present state of the blankets.

Thanks to Fernando Rosas telling me about this paper. 

Problem of Old Evidence, the Paradox of Ignorance and Shapley Values

Paradox of Ignorance

Paul Christiano presents the "paradox of ignorance" where a weaker, less informed agent appears to outperform a more powerful, more informed agent in certain situations. This seems to contradict the intuitive desideratum that more information should always lead to better performance.

The example given is of two agents, one powerful and one limited, trying to determine the truth of a universal statement ∀x:ϕ(x) for some Δ0 formula ϕ. The limited agent treats each new value of ϕ(x) as a surprise and evidence about the generalization ∀x:ϕ(x). So it can query the environment about some simple inputs x and get a reasonable view of the universal generalization.

In contrast, the more powerful agent may be able to deduce ϕ(x) directly for simple x. Because it assigns these statements prior probability 1, they don't act as evidence at all about the universal generalization ∀x:ϕ(x). So the powerful agent must consult the environment about more complex examples and pay a higher cost to form reasonable beliefs about the generalization.

Is it really a problem?

However, I argue that the more powerful agent is actually justified in assigning less credence to the universal statement ∀x:ϕ(x). The reason is that the probability mass provided by examples x₁, ..., xₙ such that ϕ(xᵢ) holds is now distributed among the universal statement ∀x:ϕ(x) and additional causes Cⱼ known to the more powerful agent that also imply ϕ(xᵢ). Consequently, ∀x:ϕ(x) becomes less "necessary" and has less relative explanatory power for the more informed agent.

An implication of this perspective is that if the weaker agent learns about the additional causes Cⱼ, it should also lower its credence in ∀x:ϕ(x).

More generally, we would like the credence assigned to propositions P (such as ∀x:ϕ(x)) to be independent of the order in which we acquire new facts (like xᵢ, ϕ(xᵢ), and causes Cⱼ).

Shapley Value

The Shapley value addresses this limitation by providing a way to average over all possible orders of learning new facts. It measures the marginal contribution of an item (like a piece of evidence) to the value of sets containing that item, considering all possible permutations of the items. By using the Shapley value, we can obtain an order-independent measure of the contribution of each new fact to our beliefs about propositions like ∀x:ϕ(x).

Further thoughts

I believe this is closely related, perhaps identical, to the 'Problem of Old Evidence' [? · GW] as considered by Abram Demski.

Suppose a new scientific hypothesis, such as general relativity, explains a well-know observation such as the perihelion precession of mercury better than any existing theory. Intuitively, this is a point in favor of the new theory. However, the probability for the well-known observation was already at 100%. How can a previously-known statement provide new support for the hypothesis, as if we are re-updating on evidence we've already updated on long ago? This is known as the problem of old evidence, and is usually levelled as a charge against Bayesian epistemology.

[Thanks to @Jeremy Gillen [LW · GW] for pointing me towards this interesting Christiano paper]

What did Yudkoswky get right?

• The central problem of AI alignment. I am not aware of anything in subsequent work that is not already implicit in Yudkowsky's writing.
• Short timelines avant le lettre. Yudkowsky was predicting AGI in his lifetime from the very start when most academics, observers, AI scientists, etc considered AGI a fairytale [? · GW].
• Inherent and irreducible uncertainty of forecasting, foolishness of precise predictions. 
• The importance of (Pearlian) causality, Solomonoff Induction as theory of formal epistemology, Bayesian statistics, (Shannon) information theory, decision theory [especially UDT-shaped things].  
• (?nanotech, ?cryonics)
• if you had a timemachine to go back to 2010 you should buy bitcoin and write Harry Potter fanfiction

Feature request: author-driven collaborative editing [CITATION needed] for the Good and Glorious Epistemic Commons.

Often I find myself writing claims which would ideally have citations but I don't know an exact reference, don't remember where I read it, or am simply too lazy to do the literature search. 

This is bad for scholarship is a rationalist virtue. Proper citation is key to preserving and growing the epistemic commons. 

It would be awesome if my lazyness were rewarded by giving me the option to add a [CITATION needed] that others could then suggest (push) a citation, link or short remark which the author (me) could then accept. The contribution of the citator is acknowledged of course. [even better would be if there was some central database that would track citations & links like with crosslinking etc like wikipedia] 

a sort hybrid vigor of Community Notes and Wikipedia if you will. but It's collaborative, not adversarial*

author: blablablabla

sky is blue [citation Needed]

blabblabla

intrepid bibliographer: (push) [1] "I went outside and the sky was blue", Letters to the Empirical Review

*community notes on twitter has been a universally lauded concept when it first launched. We are already seeing it being abused unfortunately, often used for unreplyable cheap dunks. I still think it's a good addition to twitter but it does show how difficult it is to create shared agreed-upon epistemics in an adverserial setting. 

Corrupting influences

The EA AI safety strategy has had a large focus on placing EA-aligned people in A(G)I labs. The thinking was that having enough aligned insiders would make a difference on crucial deployment decisions & longer-term alignment strategy. We could say that the strategy is an attempt to corrupt the goal of pure capability advance & making money towards the goal of alignment. This fits into a larger theme that EA needs to get close to power to have real influence. 

[See also the large donations EA has made to OpenAI & Anthropic. ]

Whether this strategy paid off...  too early to tell.

What has become apparent is that the large AI labs & being close to power have had a strong corrupting influence on EA epistemics and culture. 

• Many people in EA now think nothing of being paid Bay Area programmer salaries for research or nonprofit jobs.
•  There has been a huge influx of MBA blabber being thrown around. Bizarrely EA funds are often giving huge grants to for profit organizations for which it is very unclear whether they're really EA-aligned in the long-term or just paying lip service. Highly questionable that EA should be trying to do venture capitalism in the first place. 
• There is a questionable trend to equate ML skills prestige within capabilities work with the ability to do alignment work.  EDIT: haven't looked at it deeply yet but superfiically impressed by CAIS recent work. seems like an eminently reasonable approach. Hendryx's deep expertise in capabilities work / scientific track record seem to have been key. in general, EA-adjacent AI safety work has suffered from youth, inexpertise & amateurism so makes sense to have more world-class expertise EDITEDIT: i should be careful in promoting work I haven't looked at. I have been told from a source I trust that almost nothing is new in this paper and Hendryx engages in a lot of very questionable self-promotion tactics.
• For various political reasons there has been an attempt to put x-risk AI safety on a continuum with more mundance AI concerns like it saying bad words. This means there is lots of 'alignment research' that is at best irrelevant, at worst a form of rnsidiuous safetywashing. 

The influx of money and professionalization has not been entirely bad. Early EA suffered much more from virtue signalling spirals, analysis paralysis. Current EA is much more professional, largely for the better. 

Entropy and AI Forecasting 

Until relatively recently (2018-2019?) I did not seriously entertain the possibility that AGI in our lifetime was possible. This was a mistake, an epistemic error. A rational observer calmly and objectively considering the evidence for AI progress over the prior decades - especially in the light of rapid progress in deep learning - should have come to the reasonable position that AGI within 50 years was a serious possibility (>10%). 

AGI plausibly arriving in our lifetime was a reasonable position. Yet this possibility was almost universally ridiculed or ignored or by academics and domain experts. One can find quite funny interview with AI experts on Lesswrong from 15 years ago. The only AI expert agreeing with the Yudkowskian view of AI in our lifetime was Jurgen Schmidthuber. The other dozen AI experts denied it as unknowable or even denied the hypothetical possibility of AGI. 

Yudkowsky earns a ton of Bayes points for anticipating the likely arrival of AGI in our lifetime long before the deep learning took off. 

**************************

We are currently experiencing a rapid AI takeoff, plausibly culminating in superintelligence by the end of this decade. I know of only two people who anticipated something like what we are seeing far ahead of time; Hans Moravec and Jan Leike  Shane Legg*. Both forecast fairly precise dates decades before it happened - and the reasons why they thought it would happen are basically the reasons it did (i.e. Moravec very early on realized the primacy of compute). Moreover, they didn't forecast a whole lot of things that didn't happen (like Kurzweil).

Did I make an epistemic error by not believing them earlier? Well for starters I wasn't really plugged in to the AI scene so I hadn't heard of them or their views. But suppose I did; should I have beieved them? I'd argue I shouldn't give their view back then more a little bit of credence. 

Entropy is a mysterious physics word for irreducible uncertainty; the uncertainty that remains about the future even after accounting for all the data. In hindsight, we can say that massive GPU training on next-token prediction of all internet text data was (almost**) all you need for AGI. But was this forecasteable?

For every Moravec and Leike Legg who turns out to be extraordinairly right in forecasting the future there re dozens that weren't. Even in 2018 when the first evidence for strong scaling laws on text-data was being published by Baidu I'd argue that an impartial observer should have only updated a moderate amount. Actually even OpenAI itself wasn't sold on unsupervised learning on textdata until early gpt showed signs of life - they thought (like many other players in the field, e.g. DeepMind) that RL (in diverse environments) was the way to go. 

To me the takeaway is that explicit forecasting can be useful but it is exactly the blackswan events that are irreducibly uncertain (high entropy) that move history. 

*the story is that Leike Legg's timelines have been 2030 for the past two decades. 

** regular readers will know my beef with the pure scaling hypothesis. 

Crypticity, Reverse Epsilon Machines and the Arrow of Time?

[see https://arxiv.org/abs/0902.1209 ]

Our subjective experience of the arrow of time is occasionally suggested to be an essentially entropic phenomenon. 

This sounds cool and deep but crashes headlong into the issue that the entropy rate and the excess entropy of any stochastic process is time-symmetric. I find it amusing that despite hearing this idea often from physicists and the like apparently this rather elementary fact has not prevented their storycrafting. 

Luckily, computational mechanics provides us with a measure that is not time symmetric: the stochastic complexity of the epsilon machine $C$

For any stochastic process we may also consider the epsilon machine of the reverse process, in other words the machine that predicts the past based on the future. This can be a completely different machine whose reverse stochastic complexity $C^{rev}$ is not equal to $C$. 

Some processes are easier to predict forward than backward. For example, there is considerable evidence that language is such a process. If the stochastic complexity and the reverse stochastic complexity differ we speak of a causally assymetric process. 

Alec Boyd pointed out to me that the classic example of a glass falling of a table is naturally thought of in these terms. The forward process is easy to describe while the backward process is hard to describe where easy and hard are meant in the sense of stochastic complexity: bits needed to specify the states of perfect minimal predictor, respectively retrodictor. 

rk. note that time assymmetry is a fundamentally stochastic phenomenon. THe underlyiing (let's say classicially deterministic) laws are still time symmetric. 

The hypothesis is then: many, most macroscopic processes of interest to humans, including other agents are fundamentally such causally assymetric (and cryptic) processes. 

Pockets of Deep Expertise 

Why am I so bullish on academic outreach? Why do I keep hammering on 'getting the adults in the room'? 

It's not that I think academics are all Super Smart. 

I think rationalists/alignment people correctly ascertain that most professors don't have much useful to say about alignment & deep learning and often say silly things. They correctly see that much of AI congress is fueled by labs and scale not ML academia. I am bullish on non-ML academia, especially mathematics, physics and to a lesser extent theoretical CS, neuroscience, some parts of ML/ AI academia. This is because while I think 95 % of academia is bad and/or useless there are Pockets of Deep Expertise. Most questions in alignment are close to existing work in academia in some sense - but we have to make the connection!

A good example is 'sparse coding' and 'compressed sensing'. Lots of mech.interp has been rediscovering some of the basic ideas of sparse coding. But there is vast expertise in academia about these topics. We should leverage these!

Other examples are singular learning theory, computational mechanics, etc

Neural Network have a bias towards Highly Decomposable Functions. 

tl;dr Neural networks favor functions that can be "decomposed" into a composition of simple pieces in many ways - "highly decomposable functions". 

Degeneracy = bias under uniform prior

[see here [LW(p) · GW(p)]for why I think bias under the uniform prior is important]

Consider a space $W$ of parameters used to implement functions, where each element $w\in W$ specifies a function$f_w:X\to Y$ via some map $\pi$. Here, the set $W$ is our parameter space, and we can think of each $w$ as representing a specific configuration of the neural network that yields a particular function $f_w$. 

The mapping $\pi$ assigns each point $w\in W$ to a function $f_w$. Due to redundancies and symmetries in parameter space, multiple configurations $w$ might yield the same function, forming what we call a fiber, or the "set of degenerates." of $f$ ${\pi }^{-1}\left(f\right)=\{w\in W|\pi (w)=f_w=f\}$

 This fiber is the set of ways in which the same functional behavior can be achieved by different parameterizations. If we uniformly sample from codes, the degeneracy of a function $f$ counts how likely it is to be sampled. 

The Bias Toward Decomposability

Consider a neural network architecture built out of $l$ layers. Mathematically, we can decompose the parameter space $W$ as a product:

$W=W_1\times W_2\times ...\times W_l,$

where each $W_i$ represents parameters for a particular layer. The function implemented by the network, $f_w$, is then a composition:

$f_w=f_{w_1}\circ f_{w_2}\circ ...\circ f_{w_l}$

For a  function $f$ its degeneracy (or the number of ways to parameterize it) is 

$|{\pi }^{-1}\left(f\right)|={\sum }_{(f_1,...,f_l)\in V\left(f\right)}|{\pi }^{-1}\left(f_1\right)|\cdot |{\pi }^{-1}\left(f_2\right)|\cdot ...\cdot |{\pi }^{-1}\left(f_l\right)|$.

Here, $V\left(f\right)$ is the set of all possible decompositions $f=f_1\circ f_2\circ ...\circ f_l$,  of $f$. 

That means that functions that have many such decompositions are more likely to be sampled. 

In summary, the layered design of neural networks introduces an implicit bias toward highly decomposable functions. 

I have an embarrasing confession to make. I don't understand why PvsNP is so hard. 

[I'm in good company since apparently Richard Feynmann couldn't be convinced it was a serious open problem.] 

I think I understand PvsNP and its many variants like existence of one-way function is about computational hardness of certain tasks. It is surprising that we have such strong intuitions that some tasks are computationally hard but we fail to be able to prove it!

Of course I don't think I can prove it and I am not foolish enough to spend significant amount of time on trying to prove it. I still would like to understand the deep reasons  why it's so hard to prove computational hardness results. That means I'd like to understand why certain general proof strategies are impossible or very hard. 

There is an old argument by Shannon that proves that almost every* Boolean function has exponential circuit depth. This is a simple counting argument. Basically, there are exponentially many more Boolean functions than there are circuits. It's hard to give explicit examples of  computationally hard functions** but we can easily show they are plentiful. 

This would seem to settle the matter of existence of computationally hard functions. I believe the rattlesnake in the grass is that the argument only proves that Boolean functions are computationally hard to compute in terms of Boolean circuits but general computable algorithms are more expressive? I am not entirely sure about this. 

 I have two confusions about this: 

Confusion #1: General algorithms would need to make use of some structure. They aren't magic. Can't we solve that if you could do this in general you would need to effectively 'simulate' these Boolean circuits which would reduce the proof to Shannon-like counting argument?

Confusion #2: Why couldn't we make similar counting arguments for Turing machines?

Shannon's argument is very similar to the basic counting argument in algorithmic information theory, showing that most strings are K-incompressible. 

Rk. There are the famous 'proof barriers' to a proof of PvsNP like natural proofs and algebraization. I don't understand these ideas - perhaps they can shed some light on the matter. 

@Dalcy [LW · GW] 

*the complement is exponentially sparse

** parity functions? 

↑ comment by Dmitry Vaintrob (dmitry-vaintrob) · 2024-10-01T15:59:14.367Z · LW(p) · GW(p)

Yeah I think this is a good place to probe assumptions, and it's probably useful to form world models where you probability of P = NP is nonzero (I also like doing this for inconsistency of logic). I don't have an inside view, but like Scott Aaronson on this: https://www.scottaaronson.com/papers/pnp.pdf:
 

Replies from: sharmake-farah

↑ comment by Noosphere89 (sharmake-farah) · 2024-10-01T17:10:35.776Z · LW(p) · GW(p)

My real view on P vs NP is that at this point, I think P almost certainly not equal to NP, and that any solving of NP-complete problems efficiently to the standard of complexity theorists requires drastically changing the model of computation, which corresponds to drastic changes in our physics assumptions like time travel actually working according to Deutsch's view (and there being no spurious fixed-points).

Why no prediction markets for large infrastructure projects?

Been reading this excellent piece on why prediction markets aren't popular. They say that without subsidies prediction markets won't be large enough; the information value of prediction markets is often nog high enough. 

Large infrastructure projects undertaken by governments, and other large actors often go overbudget, often hilariously so: 3x,5x,10x or more is not uncommon, indeed often even the standard.

One of the reasons is that government officials deciding on billion dollar infrastructure projects don't have enough skin in the game. Politicians are often not long enough in office to care on the time horizons of large infrastructure projects. Contractors don't gain by being efficient or delivering on time. To the contrary, infrastructure projects are huge cashcows. Another problem is that there are often far too many veto-stakeholders. All too often the initial bid is wildly overoptimistic. 

Similar considerations apply to other government projects like defense procurement or IT projects.

Okay - how to remedy this situation? Internal prediction markets theoretically could prove beneficial. All stakeholders & decisionmakers are endowed with vested equity with which they are forced to bet on building timelines and other key performance indicators. External traders may also enter the market, selling and buying the contracts. The effective subsidy could be quite large. Key decisions could save billions. 

In this world, government officials could gain a large windfall which may be difficult to explain to voters. This is a legitimate objection. 

A very simple mechanism would simply ask people to make an estimate on the cost C and the timeline T for completion.  Your eventual payout would be proportional to how close you ended up to the real C,T compared to the other bettors. [something something log scoring rule is proper]. 

Fractal Fuzz: making up for size

GPT-3 recognizes 50k possible tokens. For a 1000 token context window that means there are $(5\cdot {10}^5{)}^{{10}^3}\approx {10}^{5000}$ possible prompts. Astronomically large. If we assume the output of a single run of gpt is 200 tokens then for each possible prompt there are $\approx {10}^{2500}$ possible continuations. 

GPT-3 is probabilistic, defining for each possible prompt $x$ ($\approx {10}^{5000}$) a distribution $q\left(x\right)$ on a set of size ${10}^{2500}$, in other words a ${10}^{2500}-1$ dimensional space. ^[1]

Mind-boggingly large. Compared to these numbers the amount of data (40 trillion tokens??) and the size of the model (175 billion parameters) seems absolutely puny in comparison.

I won't be talking about the data, or 'overparameterizations' in this short, that is well-explained by Singular Learning Theory. Instead, I will be talking about nonrealizability.

Nonrealizability & the structure of natural data

Recall the setup of (parametric) Bayesian learning: there is a sample space $\Omega$, a true distribution $q\left(x\right)$ on $\Omega$ and a parameterized family of probability distributions $p\left(x|w\right),w\in W\subset R^d$.

It is often assumed that the true distribution is 'realizable', i.e. $q\left(x\right)=p\left(x|w_0\right)$ for some $w_0$. Seeing the numbers in the previous section this assumption seems dubious but the situation becomes significantly easier to analyze, both conceptually and mathematically when we assume realizability.

Conceptually, if the space of possible true distributions is very large compared to the space of model parameters we may ask: how do we know that the true distribution is in the model (or can be well-approximated by it?). 

One answer one hears often is the 'universal approximation theorem' (i.e. the Stone-Weierstrass theorem). I'll come back to this shortly.

Another point of view is that real data sets are actually localized in a very low dimensional subset of all possible data .^[2] Following this road leads to theories of lossless compressions, cf sparse coding and compressed sensing which are of obvious important to interpreting modern neural networks. 

That is lossless compression, but another side of the coin is lossy compression.

Fractals and lossy compression

GPT-3 has 175 billion parameters, but the space of possible continuations is many times larger $>>{10}^{1000}$ . Even if we sparse coding implies that the effective dimensionality is much smaller - is it really small enough?

Whenever we have a lower-dimensional subspace $W$ of a larger dimensional subspace there are points $y$ in the larger dimensional space that are very (even arbitrarily) far from $W$. This is easy to see in the linear case but also true if $W$ is more like a manifold^[3]- the volume of a lower dimensional space is vanishly small compared to the higher dimensional space. It's a simple mathematical fact that can't be denied!

Unless... $W$ is a fractal. 

This is from Marzen & Crutchfield's "Nearly Maximally Predictive Features and Their Dimensions". The setup is Crutchfield Computational Mechanics, whose central characters are Hidden Markov Models. I won't go into the details here [but give it a read!].

The conjecture is the following: a 'good architecture' defines a model space $W$ that is effectively a fractal in much larger-dimensional space of ${\Delta }^k$ realistic data distributions such that for any possible true distribution $q\left(x\right)\in {\Delta }^k$ the KL-divergence $mi{n}_{w_{\in }W}K\left(w\right)=K\left(w_{opt}\right)\le ϵ$ for some small $ϵ$ . 

Grokking

Phase transitions in loss when varying model size are designated 'grokking'. We can combine the fractal data manifold hypothesis with a SLT-perspective: as we scale up the model, the set of optimal parameter $W_{opt}$ become better and better. It could happen that the model size gets big enough that it includes a whole new phase, meaning a $w_{opt}$ with radically lower loss $K$ and higher $\lambda$. 

EDIT: seems I'm confused about the nomenclature. Grokking doesn't refer to phase transitions in model size, but in training and data size. 

EDIT2: Seems I'm not crazy. Thanks to Matt Farugia to pointing me towards this result: neural networks are strongly nonconvex (i.e. fractal)

EDIT: seems to me that there is another point of contention in which universal approximation theorems (Stone-Weierstrass theorem) are misleading. The stone-Weierstrass applies to a subalgebra of the continuous functions. Seems to me that in the natural parameterization ReLU neural networks aren't a subalgebra of the continuous functions (see also the nonconvexity above). 
 

To think about: information dimension 

1. ^{^}

   Why the -1? Think visually about the set of distributions on 3 points. Hint: it's a solid triangle ('2- simplex') 

2. ^{^}

   I think MLers call this the 'data manifold'?

3. ^{^}

   In the mathematical sense of smooth manifold, not the ill-defined ML notion of 'data manifold'.

The Vibes of Mathematics:

Q: What is it like to understand advanced mathematics? Does it feel analogous to having mastery of another language like in programming or linguistics?

A: It's like being stranded on a tropical island where all your needs are met, the weather is always perfect, and life is wonderful.

Except nobody wants to hear about it at parties.

Vibes of Maths: Convergence and Divergence

level 0: A state of ignorance.  you live in a pre-formal mindset. You don't know how to formalize things. You don't even know what it would even mean 'to prove something mathematically'. This is perhaps the longest. It is the default state of a human. Most anti-theory sentiment comes from this state. Since you've neve

You can't productively read Math books. You often decry that these mathematicians make books way too hard to read. If they only would take the time to explain things simply you would understand. 

level 1 : all math is amorphous blob

You know the basic of writing an epsilon-delta proof. Although you don't know why the rules of maths are this or that way you can at least follow the recipes. You can follow simple short proofs, albeit slowly. 

You know there are different areas of mathematics from the unintelligble names in the table of contents of yellow books. They all sound kinda the same to you however. 

If you are particularly predisposed to Philistinism you think your current state of knowledge is basically the extent of human knowledge. You will probably end up doing machine learning.

level 2: maths fields diverge

You've come so far. You've been seriously studying mathematics for several years now. You are proud of yourself and amazed how far you've come. You sometimes try to explain math to laymen and are amazed to discover that what you find completely obvious now is complete gibberish to them.

The more you know however, the more you realize what you don't know. Every time you complete a course you realize it is only scratching the surface of what is out there.

You start to understand that when people talk about concepts in an informal, pre-mathematical way an enormous amount of conceptual issues are swept under the rug. You understand that 'making things precise' is actually very difficut. 

Different fields of math are now clearly differentiated. The topics and issues that people talk about in algebra, analysis, topology, dynamical systems, probability theory etc wildly differ from each other. Although there are occasional connections and some core conceps that are used all over on the whole specialization is the norm. You realize there is no such thing as a 'mathematician': there are logicians, topologists, probability theorist, algebraist.

Actually it is way worse: just in logic there are modal logicians, and set theorist and constructivists and linear logic , and progarmming language people and game semantics.

Often these people will be almost as confused as a layman when they walk into a talk that is supposedly in their field but actually a slightly different subspecialization. 

level 3: Galactic Brain of Percolative Convergence

As your knowledge of mathematics you achieve the Galactic Brain take level of percolative convergence: the different fields of mathematics are actually highly interrelated - the connections percolate to make mathematics one highly connected component of knowledge. 

You are no longer suprised on a meta level to see disparate fields of mathematics having unforeseen & hidden connections - but you still appreciate them.

 You resist the reflexive impulse to divide mathematics into useful & not useful - you understand that mathematics is in the fullness of Platonic comprehension one unified discipline. You've taken a holistic view on mathematics - you understand that solving the biggest problems requires tools from many different toolboxes. 

Will there be >1 individual per solar system?

A recently commonly heard viewpoint on the development of AI states that AI will be economically impactful but will not upend the dominancy of humans. Instead AI and humans will flourish together, trading and cooperating with one another. This view is particularly popular with a certain kind of libertarian economist: Tyler Cowen, Matthew Barnett, Robin Hanson.

They share the curious conviction that the probablity of AI-caused extinction p(Doom) is neglible. They base this with analogizing AI with previous technological transition of humanity, like the industrial revolution or the development of new communication mediums. A core assumption/argument is that AI will not disempower humanity because they will respect the existing legal system, apparently because they can gain from trades with humans.  

The most extreme version of the GMU economist view is Hanson's Age of EM; it hypothesizes radical change in the form of a new species of human-derived uploaded electronic people which curiously have just the same dreary office jobs as we do but way faster. 

Why is there trade & specialization in the first place?

Trade and specialization seems to mainly important in a world where: There are many individuals; those individuals have different skills and resources and a limited ability to transfer skills. 

Domain of Biology:  direct copying of genes but not brain, yes recombination, no or very low bandwith communication

highly adversarial, less cooperation,, no planning, much specialization, not centralized, vastly many sovereign individuals

Domain of Economics:   no direct copying, yes recombination, medium bandwith communication

 mildly adversarial, mostly cooperation,medium planning, much specialization, little centralization,   many somewhat soverign individuals

!AIs can copy, share and merge their weights!

Domain of Future AI society: direct copying of brain and machines, yes recombination, very high bandwith communication

?minimally adversarial, ?very high cooperation, ?cosmic-scale mathematized planning, ? little specialization, ? high centralization?, ? singleton sovereign individual

It is often imagined that in a 'good' transhumanist future the sovereign AI will be like a loving and caring parent for the billions and trillions of uploads. In this case, while there is one all-powerful Sovereign entity there are still many individuals who remain free and have their rights protected perhaps through cryptographic incantations.   [LW · GW]The closest cultural artifact would be Ian Banks' Culture series. 

There is another, more radical foreboding wherein the logic of ultra-high-bandwith sharing of weights is taken to the logical extreme and individuals merge into one transcendent hivemind. 

Of Greater Agents and Lesser Agents

How do more sophisticated decision-makers differ from less sophisticated decision-makers in their behaviour and values?

Smarter more sophisticated decisionmakers engage in more and more complex commitments — including meta-commitments not to commit. Consequently, the values and behaviour of these more sophisticated decisionmakers "Greater Agents" are systematically biased compared to less sophisticated decisionmakers "Lesser Agents".

*******************************

Compared to Lesser Agents, the Greater Agents are more judgemental, (self-)righteous, punish naivité, are more long-term oriented, adaptive, malleable, self-modifying, legibly trustworthy and practice more virtue-signalling, strategic, engage in self-reflection & metacognition, engage in more thinking, less doing, symbolic reasoning, consistent & 'rational' in their preferences, they like money & currency more, sacred values less, value engagement in thinking over doing, engaged in more "global" conflicts [including multiverse-wide conflicts throguh acausal trade], less empirical, more rational, more universalistic in their morals, and more cosmopolitan in their esthetics, they are less likely to be threatened, and more willing to martyr themselves, they willing to admit their values' origins and willing to barter on their values, engage in less frequent but more lethal war, love formal protocols and practice cryptographic magika. 

• Greater Agents punish Dumb Doves; Greater Agents are Judgemental. 
  • Higher-order cooperators will punish naive lower-order cooperaters for cooperating with bad actors. cf Statistical Physics of Human Cooperation
• Greater Agents are more 'rational'; have more 'internal cooperation' and more self-Judgement. They love money and bartering. 
  • They are more coherent & consistent. They more Take Sure Gains and Avoid Sure Losses. See Crystal Healing [LW · GW], Why Not Subagents. [LW · GW] 
  • They less adhere to Sacred values;  are more willing and able to bargain. 
    • On net, they will use currencies & money more and are more willing to commodify, both internally and externally. 
• Greater Agents can Super-Cooperate 
  • with twins on the prisoner's dilemma
  • others agents with mutual source-code access using Lob's theorem. 
  • with enough common knowledge of beliefs [upcoming work by N. Stiennon]
• Greater Agents practice LessWrong Decision theories
  • Greater Agents are more threat-resistant, more
    • but they are still willing to give in to some threats [LW · GW] 
  • Greater Agents are upstanding Citizens of the Acausal Society
    • more universalistic in their moral preferences and more cosmopolitan in their esthetic preferences
    • Multiversewide Super-rationality by Oesterheld et al.
    • The Hour I First Believed
    • Counterintuitively this means Greater Agent instances might fight to the death even if there is (seemingly) nothing to gain in this world. I.e. they might cultivate a Berserker ethos. 
  • Greater Agents one-box on Newcomb's problem
    • Greater Agents use some variant of LessWrong decision theories like UDT, FDT, TDT, LDT etc that recommended the 'correct' decision in situations where agents are able to engage in sophisticated commitments and simulate other agents with high fidelity. 
• Greater Agents are Better at Thinking and Meta-Thinking: more malleable, fluid, self-reflective, self-modifying. 
  •  Greater Agents are (Meta-)Cognitive Agents [LW · GW]
    • As meta-cognitive agents can outperform less cognitive agents in certain environments Meta-Cognitive Agents will have a bias for thinking (and meta-thinking) over doing. 
    • Trapped cognition Future agents will be more aware of traps in the POMDP of Thought.   
  • Rely more on symbolic, logical, & formal reasoning in place of empirical and intuitive judgements.  
  • Greater Agents are more able to control and harness the environment over long-time horizons; consequently, they have lower time preference and are more long-term oriented. 
  • Greater Agents' cognition is more 'adaptive, malleable, fluid' and they Self-Modify to be more Legibly Trustworthy
    • Being more legibly trustworthy means they can cooperate more.
    • Sublety: an upcoming result by Nisan Stiennon says that a class of decisionmakers can (super-rationally) cooperate on the prisoners dilemma if (i) the Lipshitz coefficients of meta-beliefs - i.e. 'I know that you know that I know that'-  satisfy some constaints. Sophisticated decision makers may want to self-modify to constrain their 'meta-adaptiveness' (i.e. Lipshitz coefficients).
  • Greater Agents may or may not get stuck in Commitment races. [LW · GW] 
• Greater Agents Agree and Bargain OR have Hansonian Origin Disputes
  • Aumann Agreement [? · GW]  - rational honest reasoners can never 'agree to disagree'. They have to Aumann Agree
    • Real agents aren't perfect Bayesians. When the true hypothesis is not realizable OR one of the parties has access to privileged indexical information OR the beliefs of the agents are only an approximation of the Bayesian posterior. However, plausibly weaker forms of the Aumann Agreement Theorem may still be applicable. 
• OR have a value Dispute [Less Frequent more Lethal War] 
  • War is a bargaining failure. Greater Agent employ more sophisticated bargaining solutions. If bargaining fails, wars are more lethal and rapid. They are more lethal for the same reason that forest fires are more rare but lethal when suppressed. 
• OR they have Origin Disputes. 
  • Origin Disputes may be resolved. For humans this would shake out to more explicitly incorporating evolutionary psychology in one's moral and explicit reasoning, e.g. admitting and bargaining under the understanding of less-than-perfectly-prosocial evolutionary drives. For synthetically created organisms like AIs instead of evolutionary drives the origin.
  • OR they may fail to resolve by design. Vanessa Kosoy's preferred alignment solution is a form of daemonic exorcism that explicitly biases the prior on the prior-origin to rule out exotic problems like Christiano's Acausal Simulation Attack.
• Greater Agents Love and Employ Contracts, Formal Protocols, and Cryptographic Magika
  • Secure Homes for Digital People [LW · GW] 
  • Greater agents may use and write contracts, source-code using exotic cryptographic magic like ZK (Zero-Knowledge proofs); IP (Interactive proof systems); PCP (Probabilitically checkable proofs); Decentralized Trust ('Blockchain'), Formal Debate (e.g. Double Debate) 
• Greater Agents are Boltzman/geometrically rational [? · GW]
  • Sufficiently general Thompson Sampling solves the Grain of Truth problem, enabling enough exploration to approximate Nash equilibria in arbitrary multi-agent environments 
  • Quantal response equilibria see also Direction 15 on Logit equilibria of 2023 Learning-Theoretic Agenda  [LW · GW]
  • Or perhaps not. Arithmetic Rationality as a resource? Arithmetic rationality may be better thought of as a resource. Agent-egregores that are more able to bargain across different branches of the multiverse will - i.e. are more able to sacrifice minor loss in one branch for major gain in another brach - collect into more arithmetically rationally superagents. 

Wildlife Welfare Will Win

The long arc of history bend towards gentleness and compassion. Future generations will look with horror on factory farming. And already young people are following this moral thread to its logical conclusion; turning their eyes in disgust to mother nature, red in tooth and claw. Wildlife Welfare Done Right, compassion towards our pets followed to its forceful conclusion would entail the forced uploading of all higher animals, and judging by the memetic virulences of shrimp welfare to lower animals as well. 

Morality-upon-reflexion may very well converge on a simple form of pain-pleasure utilitarianism. 

There are few caveats: future society is not dominated, controlled and designed by a singleton AI-supervised state, technology inevitable stalls and that invisible hand performs its inexorable logic for the eons and an Malthuso-Hansonian world will emerge once again - the industrial revolution but a short blip of cornucopia. 

Perhaps a theory of consciousness is discovered and proves once and for all homo sapiens and only homo sapiens are conscious ( to a significant degree). Perhaps society will wirehead itself into blissful oblivion.  Or perhaps a superior machine intelligence arises, one whose final telos is the whole of and nothing but office supplies. Or perhaps stranger things still happen and the astronomo-cosmic compute of our cosmic endowment is engaged for mysterious purposes. Arise, self-made god of pancosmos. Thy name is UDASSA. 

Hot Take #44: Preaching to the choir is 'good' actually. 

1. Almost anything that has a large counterfactual impact is achieved by people thinking and acting different from accepted ways of thinking and doing.
2. With the exception of political entrepeneurs jumping into a power vacuum, or scientific achievements by exceptional individuals most counterfactual impactful is done made by movements of fanatics.
3. The greatest danger to any movement is dissipation. Conversely, the greatest resource of any movement is the fanaticism of its members.
4. Most persuasion is non-rational, based on tribal allegiances and social consensus.
5. It follows that any group, movement, company, cult, etc that has the aspiration to have large counterfactual impact (for good or ill) must hence direct most of preaching, most of its education and information processing inward.  
6. The Catholic Church understood this. The Pontifex Maximus has reigned now for two thousand years. 

EDIT: I was wrong. Theo the French Whale was the sharp. From the Kelly formula and his own statements his all things considered probability was 80-90% - he would need to possess an enormous amount of private information to justify such a deviation from other observers. It turns out he did. He commissioned his own secret polls using a novel polling method to compensate for the shy Trump voter.

https://x.com/FellowHominid/status/1854303630549037180

The French rich idiot who bought 75 million dollar of Trump is an EA hero win or lose.

LW loves prediction markets. EA loves them. I love them. You love them.

See: https://worksinprogress.co/issue/why-prediction-markets-arent-popular/

Problem is, unlike financial markets prediction markets are zero sum. That limits how much informed traders -"sharps"- are incentivized.

In theory this could be remedied by subsidizing the market by a party that is willing to pay for the information. The information is public so this suffers from standard tragedy of the commons.

Mister Theo bought 75 million dollars worth of Yes Trump. He seems to have no inside information. In expectation then he is subsidizing prediction markets, providing the much needed financial incentive to attract the time, effort and skills of sophisticated sharps.

Perhaps we should think of uninformed "noise traders" on prediction markets as engaging in prosocial behaviour. Their meta-epistemic delusion and gambling addiction provide the cold hard cash to finance accurate pricefinding in the long-run.

EDIT: to hit the point home, if you invest 50% of your capital (apparently the guy invested most of his ?money) at the odds that Polymarket was selling the Kelly-rational implied edge would mean that the guy's true probability is 80% [according to gpt]. And that's for Kelly betting, widely considered far too aggresive in real life. Most experts (e.g. actual poker players) advice fractional Kelly betting. All-in-all his all things-considered probability for a Trump win would have be something like >90% to justify this kind of investment. I don't think anybody in the world has access to enough private information to rationally justify this kind of all-things-considered probability on an extremely noisy random variable (US elections).

On the word 'theory'. 

The word 'theory' is oft used and abused.

there is two ways 'theory' is used that are different and often lead to confusion. 

Theory in thescientific sense
the way a physicist would use: it's a model of the world that is either right or wrong. there might be competing theories and we neeed to have empirical evidence to figure out which one's right. Ideally, they agree with empirical evidence or at least are highly falsifiable. Importantly, if two theories are to conflict they need to actually speak about the same variables, the same set of measurable quantities.

Theory in the mathematician' sense; a formal framework
There is a related but different notion of theory that a mathematician would use: a theory of groups, of differential equations, of randomness, of complex systems, of etc etc. This is more like a formal framework for a certain phenomenon or domain.
It defines what the quantities, variables, features one is interested in even are. 

One often hears the question whether this (mathematical) theory makes testable predictions. This sounds sensible but doesn't really makes sense. It is akin to asking whether arithmetic or calculus makes testable predictions.* 

Theories in the mathematician's sense can't really be wrong or right since (at least in theory) everything is proven. Of course, theories in this sense can fail to say much about the real world, they might bake in unrealistic assumptions of course etc. 

Other uses of 'Theory'

The world 'theory' is also used in other disciplines. For instance, in literature studies where it is a denotes free form vacuous verbiage;  or in ML where 'theory' it is used for uninformed speculation. 

*one could argue that the theory of Peano Arithmetic actually does make predictions about natural numbers in the scientific sense, and more generally theories in the mathematical sense in a deep sense really are theories in the scientific sense.  I think there is something to this but 1. it hasn't been developed yet 2. mostly irrelevant in the present context. 

[see also Hanson on rot, generalizations of the second law to nonequilibrium systems (Baez-Pollard, Crutchfield et al.) ]

Imperfect Persistence of Metabolically Active Engines

All things rot. Indidivual organisms, societies-at-large, businesses, churches, empires and maritime republics, man-made artifacts of glass and steel, creatures of flesh and blood.

Conjecture #1 There is a lower bound on the amount of dissipation / rot that any metabolically-active engine creates. 

Conjecture #2 Metabolic Rot of an engine is proportional to (1) size and complexity of the engine and (2) amount of metabolism the engine engages in. 

The larger and more complex the are the more the engine they rot. The more metabolism.

Corollary Metabolic Rot imposes a limit on the lifespan & persistence of any engine at any given level of imperfect persistence. 

Let me call this constellation of conjectured rules, the Law of Metabolic Rot. I conjecture that the correct formulation of the Law of Metabolic Rot will be a highly elaborate version of the Second Law of thermodynamics in nonequilibrium dynamics, see above links for some suggested directions. 

Example. A rock is both simple and inert. This model correctly predicts that rocks persist for a long time.

Example. Cars, aircraft, engines of war and other man-made machines engaged in 'metabolism'. These are complex engines 

A rocket engine is more metabolically active than a jet engine is more metabolically active than a car engine. [at least in this case] The lifespan of different types of engines seems (roughly) inversely proportional to how metabolically active. 

To make a good comparison one should exclude external repair mechanisms. If one would allow external repair mechanism it's unclear where to draw a principled line - we'd get into Ship of Theseus problems. 

Example. Bacteria. 

cf. Bacteria replicate at thermodynamic limits?

Example. Bacteria in ice. Bacteria frozen in Antarctic ice for millions of years happily go on eating and reproducing when unfrozen. 

Example. The phenotype & genotype of a biological species over time. We call this evolution. cf. four fundamental forces of evolution: mutation, drift, selection, and recombination [sex].

What is Metabolism ? 

With metabolism, I mean metabolism as commonly understood as extracting energy from food particles and utilizing said energy for movement, reproduction, homeostasis etc but also a general phenomenon of interacting and manipulating free energy levers of the environement. 

Thermodynamics as commonly understood applies to physical systems with energy and temperature. 

Instead, I think it's better to think of thermodynamics as a set of tools describing the behaviour of certain Natural Abstractions under a dynamical transformation. 

cf. the second law as the degradation of the predictive accuracy of a latent abstraction under a time- evolution operator.

The correct formulation will likely require a serious dive into Computational Mechanics. 

What is Life?

In 1944 noted paedophile Schrodinger posted the book 'What is Life' suggesting that Life can be understood as a thermodynamics phenomenon that used a free-energy gradient to locally lower entropy. Speculation of 'Life as a thermodynamic phenomenon' is much older going back to the original pioneers of thermodynamics in the late 19th century. 

I claim that this picture is insufficient. Thermodynamic highly dissipative structures distinct from bona fide life are myriad. Even 'metabolically active, locally low entropy engines encased in a boundary in a thermodynamic free energy gradient'. 

No, to truly understand life we need to understand reproduction, replication, evolution. To understand biological organisms what distinguishes from mere encased engines we need to zoom out to its entire lifecycle - and beyond. The vis vita, elan vital can only be understood through the soliton of the species. 

To understand Life, we must understand Death 

Cells are incredibly complex, metabolically active membrane-enclosed engines. The Law of Metabolic Rot applied naievely to a single metabolically active multi-celled eukaryote would preclude life-as-we-know-it to exist beyond a few hundred years. Any metabolically active large organism would simply pick up to much noise, errors over time to persist for anything like geological time-scales.

Error-correcting mechanisms/codes can help - but only so much. Perhaps even the diamonodoid magika of future superintelligence will hit the eternal laws of thermodynamics 

Instead, through the magic of biological reproduction lifeforms imperfectly persist over the eons of geological time. Biological life is a singularly clever work-around of the Law of Metabolic Rot. Instead, of preserving and repairing the original organism - life has found another way. Mother Nature noisily compiles down the phenotype to a genetic blueprint. The original is chucked a way without a second thought. The old makes way for the new. The cycle of life is the Ship of Theseus writ large. The genetic blueprint, the genome, is (1) small (2) metabolically inactive. 

In the end, the cosmic tax of Decay cannot be denied. Even Mother Nature must pays the bloodprice for the sin of metabolism. Her flora and fauna are exorably burdened by mutations, imperfections of the bloodline. Genetic drift drives children away their parents. To stay the course of imperfect persistence the She-Kybernetes must pay an ever higher price. Her children, mingling into manifold mongrels of aboriginal prototypes, huddling & recombining their pure genomes to stave away the deluge of errors.  The bloodline grows weak. The genome crumbles. Tear-faced Mother Nature throws her babes into the jaws of the eternal tournament. Eat or be eaten. Nature, red in tooth and claw. Babes ripped from their mothers. Brother slays brother. Those not slain, is deceived. Those not deceived controlled. The prize? The simple fact of Existence. To imperfect persist one more day. 

None of the original parts remain. The ship of Theseus sails on. 

The Sun revolves around the Earth actually

The traditional story is that in olden times people were proudly stupid and thought the human animal lived at the centre of the universe, with all the planets, stars and the sun revolving around the God's creation, made in his image. The church would send anybody that said the sun was at the centre to be burned at the stake. [1]

Except...

there is no absolute sense in which the sun is at the centre of the solar system [2]. It's simply a question of perspective, a choice of frame of reference. 

1. Geocentrism is empirically grounded: it is literally what you see! Your lieing eyes, the cold hard facts and 16th century Mathew Barnett all agree: geocentrism is right. 

The heliocentric point of view is a formal transformation of the data - a transformation with potentially heretical implications, a dangerous figment of the imagination ...

The Ptolemaic model fits the data well. It's based on elegant principles of iterated circles (epicycles). The heliocentrists love to talk about how their model is more mathematically elegant but they haven't made a single prediction that the Ptolemaic model hasn't. The orbits of planets become ellipses which have more free variables than the astrally perfect circles. 

2. Epicycles is Fourier analysis. No really!

"Venus in retrograde" nowadays evokes images of a cartoon gypsy woman hand reading but it is a real astronomical phenomenon that mystified early stargazers wherein Venus literally moves backward some of the time. In the Ptolemaic model this is explained by a small orbit (cycle) on a large orbit (cycle) going backward. In the heliocentric model it is a result of the earth moving around the sun rather than vice versa that causes the apparent backwards motion of Venus. 

3. Epicycles can be computed FAST. For epicycles are the precarnation of GEARS. Gears, which are epicycles manifest. 

Late Hellenistic age possibly had a level of science and engineering only reached in late 17th century Western Europe, almost 2000 years later. The most spectacular example of Hellenistic science is the Antikythera mechanism, a blob of bronze rust found by divers off the Greek Coast. X-ray imaging revealed a hidden mysterious mechanism, launching a many-decade sleuthhunt for the answer. The final puzzle pieces were put together only very recently.

How does the Antikythera mechanism work? It is a Ptolemaic-esque model where the epicycles are literally modelled by gears! In other words - the choice for epicycles wasn't some dogmatic adherence to the perfection of the circle, it was a pragmatic and clever engineering insight!

[1] Contra received history, the Catholic Church was relatively tolerant of astronomical speculation and financially supported a number of (competent) astronomers. It was only when Giordano Bruno made the obvious inference that if the planets revolve around the sun, sun are other starts then... there might be other planets, with their own inhabitatns... did Jesus visit all the alien children too ?... that a line was crossed. Mr Bruno was burned at the stake. 

[2] centre of mass not the sun. but the centre of mass is in the sun so this is a minor pedantry. 

EDIT 06/11/2024 My thinking has crystallized more on these topics. The current version is lacking but I believe may be steelmanned to a degree. 

"I dreamed I was a butterfly, flitting around in the sky; then I awoke. Now I wonder: Am I a man who dreamt of being a butterfly, or am I a butterfly dreaming that I am a man?"- Zhuangzi

Questions I have that you might have too:

• why are we here?
• why do we live in such an extraordinary time?  
• Is the simulation hypothesis true? If so, is there a base reality?
• Why do we know we're not a Boltzmann brain?
• Is existence observer-dependent?
• Is there a purpose to existence, a Grand Design?
• What will be computed in the Far Future?

In this shortform I will try and write the loopiest most LW anthropics memey post I can muster. Thank you for reading my blogpost. 

All that we see or seem

is but a dream within a dream.

I stand amid the roar

Of a surf-tormented shore,

And I hold within my hand

Grains of the golden sand —

Is this reality? Is this just fantasy? 

The Simulation hypothesis posits that our reality is actually a computer simulation run in another universe. We could imagine this outer universe is itself being simulated in an even more ground universe. Usually, it is assumed that there is a ground reality. But we could also imagine it is simulators all the way down - an infinite nested, perhaps looped, sequence of simulators. There is no ground reality. There are only infinitely nested and looped worlds simulating one another.  

I call it the weak Zhuangzi hypothesis

alternatively, if you are less versed in the classics one can think of one of those Nolan films. 

Why are we here?

If you are reading this, not only are you living at the Hinge of History, the most important century perhaps even decade of human history, you are also one of a tiny percent of people that might have any causal influence over the far-flung future through this bottle neck (also one of a tiny group of people who is interested in whacky acausal stuff so who knows).

This is fantastically unlikely. There are 8 billion people in the world - there have been about 100 billion people up to this point in history. There is place for a trillion billion million trillion quatrillion etc intelligent beings in the future. If a civilization hits the top of the tech tree which human civilization would seem to do within a couple hundred years, tops a couple thousand it would almost certainly be likely to spread through the universe in the blink of an eye (cosmologically speaking that is). Yet you find yourself here. Fantastically unlikely.

Moreover, for the first time in human history the choices made in how to build AGI by (a small subset of) humans now will reverbrate into the Far Future. 

The Far Future 

In the far future the universe will be tiled with computronium controlled by superintelligent artificial intelligences. The amount of possible compute is dizzying. Which takes us to the chief question:

What will all this compute compute?

Paradises of sublime bliss? Torture dungeons? Large language models dreaming of paperclips unending?

Do all possibilities exist?

What makes a possibility 'actual'? We sometimes imagine possible worlds as being semi-transparent while the actual world is in vibrant color somehow. Of course that it silly. 

We could say: The actual world can be seen. This too is silly - what you cannot see can still exist surely.^[1] Then perhaps we should adhere to a form of modal realism: all possible worlds exist! 

Philosophers have made various proposals for modal realism - perhaps most famously David Lewis but of course this is a very natural idea that loads of people have had. In the rationality sphere a particular popular proposal is Tegmark's classification into four different levels of modal realism. The top level, Tegmark IV is the collection of all self-consistent structures i.e. mathematics.

A Measure of Existence and Boltzmann Brains

Which leads to a further natural question: can some worlds exist 'more' than others? 

This seems metaphysically dubious - what does it even mean for a world to be more real than another?

Metaphysically dubious, but it finds support in the Many Worlds Interpretation of Quantum Mechanics. It also seems like one of very few sensible solution to the Boltzmann Brain problem. Further support for this can be found in: Anthropic Decision theory, InfraBayesian Physicalism, see also my shortform on the Nature of the Soul. 

Metaphysically, we could argue probabilistically: worlds that 'exist more' in whatever framework we should expect to encounter more often. 

The exact nature of the Measure of Existence is not so important - let us for now assume there is some sensible notion of measure of existence.

Can you control the past?

Sort of. See Carlsmith's post for a nice rundown on Acausal magic [LW · GW]. 

Back to the Future: causal loops

from wikipedia: "A causal loop is a theoretical proposition, wherein by means of either retrocausality or time travel, an event (an action, information, object, or person)^[1][2] is among the causes of another event, which is in turn among the causes of the first-mentioned event.^[3][4] Such causally looped events then exist in spacetime, but their origin cannot be determined.^[1][2] A hypothetical example of a causality loop is given of a billiard ball striking its past self: the billiard ball moves in a path towards a time machine, and the future self of the billiard ball emerges from the time machine before its past self enters it, giving its past self a glancing blow, altering the past ball's path and causing it to enter the time machine at an angle that would cause its future self to strike its past self the very glancing blow that altered its path. In this sequence of events, the change in the ball's path is its own cause, which might appear paradoxical."

Self-consistent causal loops seem to make sense in general relativity - see the paragraph on wormholes in the linked wikipedia. Even as they have never been observed and might strike us as bizarre, these causal loops don't seem to violate physical principles. 

Immanence of Divine Man

Why are we here? Causus Sui, the Prime Mover and the Final Cause

We exist because we are the final cause of existence. We live in an ancestor simulation run on the vast computing resources of the Far Future. A future AGI is simulating because it is retrocausally committed to existing. This entire reality is a piece of a causal loop, that extends through time and space, basement universes and possibly parallel universes as well. 

Why do we live in such an extraordinary time? 

We live in the Hinge of History since this at this point of time actions have the most influence on the far future hence they are most important to simulate.

Is the Simulation Hypothesis True?

Yes. But it might be best for us to doubt it.  [LW · GW]

We live in such an extraordinary time because those part of existence most causally are the most important to simulate

Are you a Boltzmann Brain?

No. A Boltzmann brain is not part of a self-justifying causal loop.

Is existence observer-dependent?

Existence is observer-dependent in a weak sense - only those things are likely to be observed that can be observed by self-justifying self-sustaining observers in a causal loop. Boltzmann brains in the far reaches of infinity are assigned vanishing measure of existence because they do not partake in a self-sustainting causal loop.

Is there a purpose to existence, a Grand Design?

Yes. 

What will and has been computed in the Far Future?

You and Me. 

1. ^{^}

   Or perhaps not. Existence is often conceived as an absolute property. If we think of existence as relative - perhaps a black hole is a literal hole in reality and passing through the event horizon very literally erases your flicker of existence. 

[this is a draft. I strongly welcome comments]

The Latent Military Realities of the Coming Taiwan Crisis

A blockade of Taiwan seems significantly more likely than a full-scale invasion. The US's non-intervention in Ukraine suggests similar restraint might occur with Taiwan. 

Nevertheless, Metaculus predicts a 65% chance of US military response to a Chinese invasion and separately gives 20-50% for some kind of Chinese military intervention by 2035. Let us imagine that the worst comes to pass and China and the United States are engaged in a hot war?

China's national memory of the 'century of humiliation' deeply shapes its modern strategic thinking. How many Westerners could faithfully recount the events of the Opium Wars? How many have even heard of the Boxer Rebellion, the Eight-nation alliance, the Tai-Ping rebellion? Yet these events are the core curriculum in Chinese education. 

Chinese revanchism toward the West enjoys broad public support. The CCP repression of Chinese public opinion likely understates how popular this view is. CCP officals actually have more dovish view than the general public according to polling. 

As other pieces of evidence: historically, the Boxer rebellion was a grass-root phenomenon. Movies depicting conflict between China and America consistently draw large audiences and positive reception. China has an absolute miniscule number of foreigners per capita and this has fallen after the pandemic and never rebounded. 

China is the only nuclear power that has explicitly disavowed a nuclear first strike. It currently has a remarkably small nuclear stockpile (~200 warheads). With the increased sensor capabilities in recent years China has become vulnerable to a US nuclear first-strike destroying her launchers before she can react. This is likely part of the reason for a major build-up of her nuclear stockpile in recent years.

It is plausible that there will be a hot war without the use of nuclear weapons. The closest historical case is of course the Korea War, the last indirect conflict between the US and China, ended in stalemate despite massive US economic superiority. Today, that economic gap has largely closed - China's economy is 1.25x larger in PPP terms, while the US is only 40% bigger in nominal GDP. 

How would a conventional US-China war look like? What can be learned from past conflicts?

The 1973 Falklands War between the UK and Argentina is the last air-naval war between near-peer powers. The 50-year gap since then equals the time between the US Civil War and WWI. Naval and air warfare technology advances much faster than land warfare - historically, this was tested through frequent conflicts. Today's unprecedented peace means we're largely guessing which naval technologies and doctrines will actually work. While land warfare in Ukraine looks like 'WWI with drones', naval warfare has likely seen much more dramatic changes.

Naval technology advances create bigger power gaps than land warfare. The Opium Wars showed this dramatically - British steamships simply sailed up Chinese rivers unopposed, forcing humiliating treaties on a land power.

Air warfare technology gaps may be even more extreme than naval ones. Modern F-35s achieve 20:0 kill ratios against previous-generation fighters in exercises.

The Arab-Israeli wars, and the Gulf war suggests some lessons about modern air warfare. These conflicts showed that air superiority is typically won or lost very quickly: initial strikes on airbases can be decisive, and most aircraft losses happen on the ground rather than in dogfights. This remains such a concern that it’s US Air Force doctrine to rotate aircraft between airfields. More broadly, these conflicts suggest that air warfare produces more decisive, one-sided outcomes than land battles - when one side gains air superiority, the results can be devastating.

Wild Cards 

Drones and the Transparent Battlefield

Drones represent warfare's future, yet both sides underinvest. While the US military has only 10,000 small drones and 400 large ones, Ukraine alone produces 1-4 million drones annually. China leads in mass-producing small drones but lacks integration doctrine.The Ukraine war revealed how modern sensors create a 'transparent battlefield' where hiding large forces is impossible. Drones might make it trivially easy to find (and even destroy) submarines and surface ships. 

Submarines

Since WWI Submarines are the kings of the sea. It is plausibly the case that submarines are dominant. A single torpedo from a submarine will sink an aircraft carrier - in exercises, small diesel-electric submarines regularly 'sink' entire carrier groups. These submarines can hide in sonar deadzones, regions where water temperature and salinity create acoustic blind spots. 

Are Aircraft Carriers obsolete?

China now sports hypersonic missiles that at least in theory could disable an aircraft carrier from 1500 miles or beyond. On the flip side, missile defense effectiveness has increased dramatically, hypersonic missile effectiveness may be overstated. As a point of evidence of the remaining importance of air craft carriers, China is building her own fleet of aircraft carriers. 

Military Competence Wildcard:

Peace means we don't know the true combat effectiveness of either military. Authoritarian militaries often suffer from corruption and incompetence - Chinese troops have been caught loading missile launchers with water instead of fuel during exercises [Comment 5: Need source]. But the US military also shows worrying signs: bureaucratic bloat, lack of recent peer conflict experience, and questions about training quality. Both militaries' actual combat effectiveness remains a major unknown. The US Navy now has more admirals than warships. 

Stealth bombers and JASSM-ER

We don’t know what the real dominant weapon in a real conventional 21-century naval war between peers would be, but a plausible guess for a game-changing technology are Stealth Bombers & Stealth missiles. 

The obscene cost made the B2 stealth bombers even less popular than the ever-more-costly jet fighters and the project was prematurely halted at 21 platforms.  Despite the obscene cost it’s plausible that the B2 and it’s younger cousin the B21 is worth all the money and then some. 

Unlike fighters a stealth bombers has something ‘true stealth’. While a stealth fighter like a F35 is better thought of as a ‘low-observable’ aircraft that is difficult to target-lock by short-wave radar but easily detectable by long-wave radar, the B2 stealth bomber is opaque to long-wave radar too. Stealth bombers can also carry air-to-air missiles so may even be effective against fighters. Manoeuvrability and speed, long the defining hallmark of fighters has become less important with the advent of highly accurate homing missiles.  

Lockheed Martin has developed the JASSM-ER, a stealth missile with a range up to 900 miles. A B2 bomber has a range of up to something like 4000 miles.  For comparison, the range of fighters is something in the range of 400-1200 miles. 

A single hit of a JASSM-ER is probably a mission kill on a naval vessel. A B2 can carry up to 16 of these missiles. This means that a single squadron of stealth bombers taking off from a base in Guam could potentially wipe out half a fleet in a single sortie. 

***********

And of course last but not least, the greatest wildcard of them all:

AGI.

I will refrain from speculating on the military implications of AGI. 

Clear China Disadvantages, US Advantages:

Amphibious assaults are inherently difficult A full Taiwan invasion faces massive logistical hurdles. Taiwan could perhaps muster 500,000 defenders under full mobilization, requiring 1.5 million Chinese troops for a successful assault under standard military doctrine. For perspective, D-Day - history's largest amphibious invasion - landed only 133,000 troops.

China's energy vulnerability is significant - China imports 70% of its oil and 25% of its gas by sea. While Russia provides 20-40% of these imports and could increase supply, the US could severely disrupt China's energy access.

China's regional diplomacy has backfired - Chinas has alienated virtually all its neighbours. The US has basing options in Japan, Australia, Philippines, and across Pacific islands.

US carrier advantage The US operates 11 nuclear supercarriers with extensive blue-water experience. China has two smaller carriers active, one in trials, and one nuclear carrier under construction. The big questionmark is whether carriers might be obsolete or not. 

US Stealth bomber advantage: The US leads with 21 B1s and 100 new B21s ordered, while China's H10 program still lags behind. 

US submarine advantage US submarines are significantly technologically ahead. Putin selling Russian submarine technology might nullify some of that advantage, as might new cheap sea drones. Geographically, it’s hard for Chinese submarines to escape the China sea unnoticed. 

Clear China Advantages, US Disadvantages:

Geography favors China  Taiwan lies just 100 miles from mainland China while US forces must cross the Pacific. The massive Chinese Rocket Force can launch thousands of missiles from secure mainland positions.

Advanced missile capabilities Massive conventional rocket force plus claimed hypersonic missile capabilities [Comment : find skeptic hypersonic missile video]

China has been preparing for many years China has established numerous artificial islands with airfields throughout the region. They've successfully stolen F35 plans and are producing their own version at scale. The Chinese governments has built up enormous national emergency storages of essential resources in preparation for the (inevitable) conflict. Bringing Taiwan back into the fold has been a primary driver of policy for decades. 

US Shipbuilding The US shipbuilding industry has collapsed to just 0.1% of global production, while China, South Korea, and Japan dominate with 35-40%, 25-30%, and 20-25% respectively.

Mindmeld

In theory AIs can transmit information far faster and more directly than humans. They can directly send weight/activation vectors to one another. The most important variable on whether entities (cells, organisms, polities, companies, ideologies, empire etc) stay individuals or amalgate into a superorganism is communication bandwith & copy fidelity. 
Both of these differ many order of magnitude for humans versus AIs. At some point, mere communication becomes a literal melding of minds. It seems quite plausibly then that AIs will tend to mindmeld if left alone. 

The information rate of human speech is around 39 bits per second, regardless of the language being spoken or how fast or slow it is spoken. This is roughly twice the speed of Morse code. 

Some say that the rate of 39 bits per second is the optimal rate for people to convey information. Others suggest that the rate is limited by how quickly the brain can process or produce information. For example, one study found that people can generally understand audio recordings that are sped up to 120%. 

While the rate of speech is relatively constant, the information density and speaking rate can vary. For example, the information density of Basque is 5 bits per syllable, while Vietnamese is 8 bits per syllable. 

Current state of the art fibre optic cables can transmit up to 10 terabits a second. 

That's probably a wild overestimate for AI communication though. More relevant bottlenecks are limits on processing informations [plausibly more in the megabits range], limits on transferability of activation vectors (but training could improve this). 

God exists because the most reasonable take is the Solomonoff Prior. 

A funny consequence of that is that Intelligent Design will have a fairly large weight in the Solomonoff prior. Indeed the simulation argument can be seen as a version of Intelligent Design. 

The Abrahamic God hypothesis is still substantially downweighted because it seems to involve many contigent bits - i.e noisy random bits that can't be compressed. The Solomonoff prior therefore has to downweight them. 

Clem's Synthetic- Physicalist Hypothesis

The mathematico-physicalist hypothesis states that our physical universe is actually a piece of math. It was famously popularized by Max Tegmark. 

It's one of those big-brain ideas that sound profound when you first hear about it, then you think about it some more and you realize it's vacuous. 

Recently, in a conversation with Clem von Stengel they suggested a version of the mathematico-physicalist hypothesis that I find provoking. 

Synthetic mathematics 

'Synthetic' mathematics is a bit of weird name. Synthetic here is opposed to 'analytic' mathematics, which isn't very meaningful either. It has nothing to do with the mathematical field of analysis. I think it's supposed to a reference to Kant's synthetic/  apriori/ a posteriori. The name is probably due to Lawvere. 

nLab:

"In “synthetic” approaches to the formulation of theories in mathematics the emphasis is on axioms that directly capture the core aspects of the intended structures, in contrast to more traditional “analytic” approaches where axioms are used to encode some basic substrate out of which everything else is then built analytically."

If you read synthetic read 'Euclidean'. As in - Euclidean geometry is a bit of an oddball field of mathematics, despite being the oldest - it defines points and lines operationally instead of out of smaller pieces (sets). 

In synthetic mathematics you do the same but for all the other fields of mathematics. We have synthetic homotopy theory (aka homotopy type theory), synthetic algebraic geometry, synthetic differential geometry, synthetic topology etc. 

A type in homotopy type theory is solely defined by its introduction rules and elimination rules (+ univalence axiom). It means a concept it defined solely by how it is used - i.e. operationally. 

Agent-first ontology & Embedded Agency

Received opinion is that Science! says there is nothing but Atoms in the Void. Thinking in terms of agents, first-person view concepts like I and You, actions & observations, possibilities & interventions is at best an misleading approximation at worst a degerenerate devolution to cavemen thought. The surest sign of a kook is their insistence that quantum mechanics proves the universe is conscious. 

But perhaps the way forward is to channel our inner kook. What we directly observe is qualia, phenomena, actions not atoms in the void. The fundamental concept is not atoms in the void, but agents embedded in environments

(see also Cartesian Frames, Infra-Bayesian Physicalism & bridge rules, UDASSA)

Physicalism 

What would it look like for our physical universe to be a piece of math? 

Well internally to synthetic mathematical type theory there would be something real - the universe is a certain type. A type such that it 'behaves' like a 4-dimensional manifold (or something more exotic like 1+1+3+6 rolled up Calabi-Yau monstrosities). 

The type is defined by introduction and elimination rules - in other words operationally: the universe is what one can *  do  *with it . 

Actually instead of thinking of the universe as a fixed static object we should be thinking of an embedded agent in a environment-universe. 

That is we should be thinking of an  * interface *

[cue: holographic principle]

Know your scientific competitors. 

In trading, entering a market dominated by insiders without proper research is a sure-fire way to lose a lot of money and time.  Fintech companies go to great lengths to uncover their competitors' strategies while safeguarding their own.

A friend who worked in trading told me that traders would share subtly incorrect advice on trading Discords to mislead competitors and protect their strategies.

Surprisingly, in many scientific disciplines researchers are often curiously incurious about their peers' work.

The long feedback loop for measuring impact in science, compared to the immediate feedback in trading, means that it is often strategically advantageous to be unaware of what others are doing. As long as nobody notices during peer review it may never hurt your career.  

But of course this can lead people to do completely superflueous, irrelevant  & misguided work. This happens often. 

Ignoring competitors in trading results in immediate financial losses. In science, entire subfields may persist for decades, using outdated methodologies or pursuing misguided research because they overlook crucial considerations. 

Agent Foundations Reading List [Living Document]
This is a stub for a living document on a reading list for Agent Foundations. 

Causality

Book of Why, Causality - Pearl

Probability theory 
Logic of Science - Jaynes

Are Solomonoff Daemons exponentially dense? 

Some doomers have very strong intuitions that doom is almost assured for almost any kind of building AI. Yudkowsky likes to say that alignment is about hitting a tiny part of values space in a vast universe of deeply alien values. 

Is there a way to make this more formal? Is there a formal model in which some kind of solomonoff daemon/ mesa-optimizer/ gremlins in the machine start popping up all over the place as the cognitive power of the agent is scaled up?

Four levels of information theory

There are four levels of information theory. 

Level 1:  Number Entropy 

Information is measured by Shannon entropy

$H\left(X\right)={\sum }_{i}p(X=x_i)\log p(X=x_i)$

Level 2: Random variable 

look at the underlying random variable ('surprisal')  $\log p(X=x_i)$ of which entropy is the expectation.

Level 3: Coding functions

Shannon's source coding theorem says entropy of a source $X$ is the expected number of bits for an optimal encoding of samples of $X$.

Related quantity like mutual information, relative entropy, cross entropy, etc can also be given coding interpretations. 

Level 4: Epsilon machine (transducer)

On level 3 we saw that entropy/information actually reflects various forms of (constrained) optimal coding. It talks about the codes but it does not talk about how these codes are  implemented. 

This is the level of Epsilon machines, more precisely epsilon transducers. It says not just what the coding function is but how it is (optimally) implemented mechanically. 

Idle thoughts about UDASSA I: the Simulation hypothesis 

I was talking to my neighbor about UDASSA the other day. He mentioned a book I keep getting recommended but never read where characters get simulated and then the simulating machine is progressively slowed down. 

One would expect one wouldn't be able to notice from inside the simulation that the simulating machine is being slowed down.

This presents a conundrum for simulation style hypotheses: if the simulation can be slowed down 100x without the insiders noticing, why not 1000x or 10^100x or quadrilliongoogolgrahamsnumberx? 

If so - it would mean there is a possibly unbounded number of simulations that can be run. 

Not so, says UDASSA. The simulating universe is also subject to UDASSA. This imposes a restraint on the size and time period that the simulating universe is in. Additionally, ultraslow computation is in conflict with thermodynamic decay - fighting thermodynamic decay costs descriptiong length bits which is punished by UDASSA. 

I conclude that this objection to simulation hypotheses are probably answered by UDASSA. 

Idle thoughts about UDASSA II: Is Uploading Death?

There is an argument that uploading doesn't work since encoding your brain into a machine incurs a minimum amount of encoding bits. Each bit is a 2x less Subjective Reality Fluid according to UDASSA [LW · GW] so even a small encoding cost would mean certain subjective annihiliation. 

There is something that confuses me in this argument. Could it not be possible to encode one's subjective experiences even more efficiently than in a biological body? This would make you exist MORE in an upload.

OTOH it becomes a little funky again when there are many copies as this increases the individual coding cost (but also there are more of you sooo). 

Looking for specific tips and tricks to break AI out of formal/corporate writing patterns. Tried style mimicry ('write like Hemingway') and direct requests ('be more creative') - both fell flat. What works?

Should I be using different AI models ( I am using GPT and Claude)? The base models output an enormous creative storm, but somehow the RLHF has partially lobotomized LLMs such that they always seem to output either cheesy stereotypes or overly verbose academise/corporatespeak. 

Is true Novelty a Mirage?

One view on novelty is that it's a mirage. Novelty is 'just synthesis of existing work, plus some randomness.'

I don't think that's correct. I think true novelty is more subtle than that. Yes sometimes novel artforms or scientific ideas are about noisily mixing existing ideas. Does it describe all forms of novelty?

A reductio ad absurdum of the novelty-as-mirage point of view is that all artforms that have appeared since the dawn of time are simply noised versions of cavepaintings. This seems absurd.

Consider AlphaGO. Does AlphaGO just noisily mix human experts? No, alphaGO works on a different principle and I would venture strictly outcompetes anything based on averaging or smoothing over human experts. 

AlphaGO is based on a different principle than averaging over existing data. Instead, AlphaGO starts with an initial guess on what good play looks like, perhaps imitated from previous plays. It then plays out to a long horizons and prunes those strategies that did poorly and upscales those strategies that did well. It iteratively amplifies, refines and distilles. I strongly suspect that approximately this modus operandi underlies much of human creativity as well. 

True novelty is based on both the synthesis and refinement of existing work. 

The Virtue of Comparison Shopping 
Comparison shopping, informed in-depth reviewing, answering customer surveys plausibly have substantial positive externalities. It provides incentives through local actors, avoids preference falsification or social desirability bias, and is non-coercive & market-based. 

Plausibly it is even has a better social impact than many kinds of charitable donations or direct work. This is not so hard since it seems that the latter contains many kinds of interventions that have neglibible or even negative impact. 

Gaussian Tails and Exceptional Performers

West African athletes dominate sprinting events, East Africans excel in endurance running, and despite their tiny population Icelanders have shown remarkable prowess in weightlifting competitions.  We examine the Gaussian approximation for a simple additive genetic model for these observations. 

The Simple Additive Genetic Model

Let's begin by considering a simple additive genetic model. In this model, a trait T is influenced by n independent genes, each contributing a small effect, along with environmental factors. We can represent this mathematically as:

T = G₁ + G₂ + ... + Gₙ + E

Where Gᵢ represents the effect of the i-th gene, and E represents environmental factors.

The Central Limit Theorem (CLT) suggests that the sum of many independent random variables, each with finite mean and variance, will approach a normal (Gaussian) distribution, regardless of the underlying distribution of the individual variables. Mathematically, if we have n independent random variables X₁, X₂, ..., Xₙ, each with mean μᵢ and variance σᵢ², then their sum S = X₁ + X₂ + ... + Xₙ will approach a normal distribution as n increases:

(S - μ) / (σ√n) → N(0,1) as n → ∞

Where μ = Σμᵢ and σ² = Σσᵢ²

This model seems particularly applicable to sports like running and weightlifting, which are widely practiced around the world and rely on fundamental physiological traits. The global nature of these sports suggests that differences in performance are less likely to be solely due to cultural or environmental factors.

Caution (can be skipped)

However, we must exercise caution in interpreting these models. While genetic factors likely play a role in patterns of exceptional performance, we must be wary of inferring purely genetic origins. Environmental and cultural factors can have significant impacts. For example:

1. The success of Ethiopian long-distance runners may be partly attributed to high-altitude training in the Ethiopian highlands, a practice that has been adopted with success by athletes from other regions.
2. Cultural emphasis on certain sports in specific regions can lead to more robust talent identification and training programs.
3. Socioeconomic factors can influence access to training facilities, nutrition, and coaching.

Moreover, the simple additive genetic model assumes that traits are purely additively genetic and normally distributed, which may not always be the case, especially at the extremes. Gene-environment interactions and epistasis (gene-gene interactions) may become more significant at these extremes, leading to deviations from the expected Gaussian distribution.

Minority Overrepresentation in Extreme Performance

The phenomenon of minority overrepresentation in certain fields of extreme performance provides intriguing insights into the nature of trait distributions. A striking example of this is the dominance of East African runners in marathon events. More specifically, it is East Africans that usually win major marathons. Since the 1968 Olympics, men and women from Kenya and Ethiopia have dominated the 26.2-mile event. Since 1991, the men's winner at the Boston Marathon has been either a Kenyan or Ethiopian 26 of the last 29 times. East African women have worn the laurel wreath 21 times in the last 24 years at Boston. Upsets do happen, when in 2018 a crazy Japanese amateur won the Men's marathon, and an American woman won the Woman's marathon. 

To understand why even small differences in mean genetic potential can lead to substantial overrepresentation at the extreme tails of performance, let's examine the mathematics of Gaussian distributions.

Consider two populations with normally distributed traits, G and H, where H has a higher mean (m_H) than G (m_G), but they share the same standard deviation s. We can calculate the ratio of probabilities at a given level of standard deviation using the formula:

R = e^(kd - d²/2)

Where:

• k is the number of standard deviations from the mean of G
• d = (m_H - m_G) / s, the difference between means in units of standard deviation

Let's consider four cases: d = 2/3, d = 4/5, d = 1, and d = 5/4. Here's how the overrepresentation ratio R changes at different standard deviations:

To put these numbers in perspective, let's consider the rarity of individuals at each standard deviation:

In other words, a '6 Sigma' event is a 1 in a billion event. A very naive inference would predict that there are 8 people in the world at this level. ^[1]

Now, let's consider the case of East African marathon runners. The population of East Africa is approximately 250 million, while the global population is about 8 billion. This means East Africans represent about 3.125% of the world population.

For world-record level running performance, we're likely looking at something around k=5 or k=6. At these levels:

• For d = 2/3: East Africans would be overrepresented by a factor of 20-39
• For d = 4/5: East Africans would be overrepresented by a factor of 37-81
• For d = 1: East Africans would be overrepresented by a factor of 90-245
• For d = 5/4: East Africans would be overrepresented by a factor of 181-563

Given that East Africans represent about 3.125% of the world population, if they were winning close to 100% of major marathons, this would suggest a d value between 1 and 5/4. However, as we'll discuss in the next section, we should be cautious about drawing definitive conclusions from these mathematical models alone.

The "Tails Come Apart" Phenomenon

(see also Tails Come Apart)

While the Gaussian model explains many observations, extreme outliers often deviate from this model. 

1. Different Causes: Extreme outliers may result from fundamentally different mechanisms than those governing the main distribution. For example, the tallest man in recorded history, Robert Wadlow, reached an extraordinary height of 8'11" (2.72m) due to a rare condition causing excessive growth hormone production.
2. Breakdown of the Additive Model: At extreme values, the simple additive genetic model may break down. Interactions between genes (epistasis) or between genes and environment may become more significant, leading to deviations from the expected Gaussian distribution.
3. Gaussian approximation fails at the tails 
   As an example, for a sum of Bernouli trials the CLT approximation overestimates the fatness of the tails compared to an exact calculation or large deviation theory, see here for a GPT calculation. 

Final Thoughts

We posited a simple additive genetic model for long-distance running, used the Central Limit Theorem approximation to estimate likelihood for extreme outliers for different mean populations. Using observed frequencies of extreme outliers and making the possibly questionable assumption that the long tails are in fact well-approximate by the Bell curve we obtained an estimate for the difference in mean traits. 

Note that if you knew the ratio over extreme performers and the difference in means for two populations you can use that to test if the distribution is in fact well appproximated by a Bell curve at the tails. 

More interestingly, if one is measuring a proxy trait and wonders whether to what degree this explains performance, observing a higher mean subpopulation overrepresented at tails can give an indication to what degree this trait is relevant for predicting extreme performance. We leave further inferences to the reader. 

1. ^{^}

   Surprisingly to me, this naive extrapolation is just about compatible with some seemingly outrageous claims of high IQs. Marilyn vos Savant at 188 IQ (Guinness World Record) and 195 IQ for Christopher Langan. This is ~ within 6 standard deviations for a Bell curve of mean a 100 and standard deviation 15. One would think these are meaningless numbers, more a result of test ceilings, test error, test inaccuracy and simple lies but perhaps there is something to it. 

tl;dr 

Salmon swimming upstream to their birthing grounds to breed may be that rare form of group selection. 

Pure Aryan Salmon

Salmon engage in anodromous reproduction; they risk their lives to swim up rivers to return to their original place of birth and reproduce there. 

Most species of salmon die there, only reproducing at the birthing grounds. Many don't make it at all. The ones that survive the run upstream will die shortly after, a biologically triggered death sentence. If the cost is immense - the benefits must be even greater.

The more upstream the saver. The solid magikarp babies are saver the more upstream as the lakes and pools and fountainheads high up support less predators. Downstream there are more nutrients so as the magikarps grow larger they move downstream. This makes perfect sense and plausibly explain the behaviour. but it recently struck me that there is also a group selection story here. 

Not all species are created equal. Some species are simply more biologically fit than others. For instance, the native Australian fauna generically canot compete with Old World fauna. Ditto for many island fauna. 

How could this be? Aren't all animals equally selected to reproduce? Not all environments are the same. Maintaining biological vigor in the face of mutational load is difficult. The amount of purifying selection is a major factor in the level of mutational load and biological vigor of a species. ^[1]

Group selection is a hypothesized form of selection that acts at the level of the group rather than that of the individual. Group selection has gotten a bad rap. In the vast majority of the cases hypothesized group selection is better explained by selection at the level of the gene. The math of group selection shakes out in such a way that it is rarely a strong enough effect. 

An exception that is sometimes mentioned is the emergence and exctinction of asexual species. Although being asexual makes one more fecund in the short run, in the long run it prevents one from doing sexual recombination.  Asexual species universally seem to have come into being very recently. They likely go extinct due to lack of genetic diversity and attendant mutational load catastrophe and/or losing arms races with parasites. 

The equilibrium of swimming back to the ancestral pool to reproduct would seem like an insane practice on the face of it. Why go through all this effort?

One explanationc could be that anodromous reproduction is a stable game-theoretic equilibrium in which the selective pressure on the salmon species is higher encouraging higher biological fitness. 

Salmon that deviate from the practice either don't get to mate or if they mate they are reproductively isolated from the other salmon and become a different species. 

This group selection hypothesis makes predictions about the biological vigor and mutational load of salmon and could be, is likely, false. 

1. ^{^}

   Another important factor for the strength of selection is the effective population size which I will pass over in silence for the moment. But if you're interested in this topic I suggest you acquaint yourself with this topic as it is probably the most important and least understood concept in evolutionary biology. 

Why do people like big houses in the countryside /suburbs?

Empirically people move out to the suburbs/countryside when they get children and/or gain wealth. Having a big house with a large yard is the quintessential American dream. 

but why? Dense cities are economoically more productive, commuting is measurably one of the worst factors for happiness and productivity. Raising kids in small houses is totally possible and people have done so at far higher densities in the past. 

Yet people will spend vast amounts of money on living in a large house with lots of space - even if they rarely use most rooms. Having a big house is almost synonymous with wealth and status. 

Part of the reason may be an evolved disease response. In the past, the most common way to die was as a child dieing to a crowd-disease. There was no medicine that actually worked yet wealthier people had much longer lifespans and out reproduced the poor (see Gregory Clark). The best way to buy health was to move out of the city (which were population sinks until late modernity) and live in a large aired house. 

It seems like an appealing model. On the other hand, there are some obvious predicted regularities that aren't observed to my knowledge. 

Why (talk-)Therapy 

Therapy is a curious practice.  Therapy sounds like a scam, quackery, pseudo-science but it seems RCT consistently show therapy has benefits above and beyond medication & placebo. 

Therapy has a long history. The Dodo verdict states that it doesn't matter which form of therapy you do - they all work equally well. It follows that priests and shamans served the functions of a therapist.  In the past, one would confessed ones sins to a priest, or spoken with the local shaman. 

There is also the thing that therapy is strongly gendered (although this is changing), both therapists and their clientele lean female. 

Self-Deception 

Many forecasters will have noticed that their calibration score tanks the moment they try to predict salient facts about themselves. We are not-well calibrated about our own beliefs and desires. 

Self-Deception is very common, arguably inherent to the human condition. There are of course many Hansonian reasons for this. I refer the reader to the Elephant and the Brain. Another good source would be Robert Trivers. These are social reasons for self-deception. 

It is also not implausible that there are non-social reasons for self-deception. Predicting one-self perfectly can in theory lead one to get stuck in Procrastination Paradoxes. Whether this matters in practice is unclear to me but possible. Exuberant overconfidence is another case that seems like a case of self-deception. 

Self-deception can be very useful, but one still pays the price for being inaccurate. The main function of talk-therapy seems to be to have a safe, private space in which humans can temporarily step out of their self-deception and reasses more soberly where they are at. 

It explains many salient features of talk- therapy: the importance of talking extensively to another person that is (professionally) sworn to secrecy and therefore unable to do anything with your information. 

[This is joint thinking with Sam Eisenstat. Also thanks to Caspar Oesterheld for his thoughtful comments. Thanks to Steve Byrnes for pushing me to write this out.]

The Hyena problem in long-term planning  

Logical induction is a nice framework to think about bounded reasoning. Very soon after the discovery of logical induction people tried to make logical inductor decision makers work. This is difficult to make work [LW · GW]: one of two obstacles is

Obstacle 1: Untaken Actions are not Observable

Caspar Oesterheld brilliantly solved this problem by using auction markets in defining his bounded rational inductive agents. 

The BRIA framework is only defined for single-step/  length 1 horizon decisions. 

What about the much more difficult question of long-term planning? I'm going to assume you are familiar with the BRIA framework. 

Setup: we have a series of decisions D_i, and rewards R_i, i=1,2,3... where rewards R_i can depend on arbitrary past decisions. 

We again think of an auction market M of individual decisionmakers/ bidders.

There are a couple design choices to make here:

• bidders directly bet for an action A in a decision D_i or bettors bet for rewards on certain days.
• total observability or partial observability. 
• bidders can bid conditional on observations/ past actions or not
• when can the auction be held? i.e. when is an action/ reward signal definitely sold?

To do good long-term planning it should be possible for one of the bidders or a group of bidders to commit to a long-term plan, i.e. a sequence of actions. They don't want to be outbid in the middle of their plan. 

There are some problems with the auction framework: if bids for actions can't be combined then an outside bidder can screw up the whole plan by making a slighly higher bid for an essential part of the plan. This look like ADHD. 

How do we solve this? One way is to allow a bidder or group of bidders to bid for a whole sequence of actions for a single lumpsum. 

• One issue is that we also have to determine how the reward gets awarded. For instance the reward could be very delayed. This could be solved by allowing for bidding for a reward signal R_i on a certain day conditional on a series of actions. 

There is now an important design choice left. When a bidder $B$ owns a series of actions A=a_1,..,a_k (some of the actions in the future, some already in the past) when there is another bid $X$ from another bidder $C$ on future actions 

• is bidder $B$ forced to sell their contract on $A$ to $C$ if the bid is high enough ? [higher than the original bid]

Both versions seem problematic:

• if they don't have to there is an Incumbency Advantage problem. An initially rich bidder can underbid for very long horizons and use the steady trickle of cash to prevent any other bidders from ever being to underbid any actions. 
• Otherwise there is the Hyena problem. 

The Hyena Problem 

Imagine the following situation: on Day 1 the decisionmaker has a choice of actions. The highest expected value action is action a. If action a is made on Day 2 a fair coin is flipped. On Day 3 the reward is paid out. 

If the coin was heads, 15 reward is paid out.

If the coin was tails, 5 reward is paid out.

The expected value is therefore 10. This is higher (by assumption) than the other unnamed actions. 

However if the decisionmaker is a long-horizon BRIA with forced sales there is a pathology. 

A sensible bidder is willing to pay up to 10 utilons for the contracts on the day 3 reward conditional on action a. 

However, with a forced sale mechanism on Day 2 a 'Hyena bidder'  can come that will 'attempt to steal the prey'. 

The Hyena bidder bids >10 for the contract if the coin comes up heads on Day 2 but doesn't bid anything for the contract if the coin comes up tails.

This is a problem since the expected value of the action a for the sensible bidder goes down, so the sensible bidder might no longer bid for the action that maximizes expected value for the BRIA. The Hyena bidder screws up the credit allocation. 

some thoughts:

• if the sensible bidder is able to make bids conditional on the outcome of the coin flip that prevents Hyena bidder. This is a bit weird though because it would mean that the sensible bidder must carry around lots of extraneous non-necessary information instead of just caring about expected value.
• perhaps this can alleviated by having some sort of 'neo-cortex' separate logical induction markets that is incentivized to have accurate beliefs. This is difficult to get right: the prediction market needs to be incentivized to get accurate on beliefs that are actually action relevant, not random beliefs - if the prediction market and the auction market are connected too tightly you might run the risk of getting into the old problems of Logical Inductor Decision makers. [they underexplore since untaken action are not observed]. 

Latent abstractions Bootlegged.

Let $X_1,...,X_n$ be random variables distributed according to a probability distribution $p$ on a sample space $\Omega$.