Posts

Are extreme probabilities for P(doom) epistemically justifed? 2024-03-19T20:32:04.622Z
Timaeus's First Four Months 2024-02-28T17:01:53.437Z
What's next for the field of Agent Foundations? 2023-11-30T17:55:13.982Z
Announcing Timaeus 2023-10-22T11:59:03.938Z
Open Call for Research Assistants in Developmental Interpretability 2023-08-30T09:02:59.781Z
Apply for the 2023 Developmental Interpretability Conference! 2023-08-25T07:12:36.097Z
Optimisation Measures: Desiderata, Impossibility, Proposals 2023-08-07T15:52:17.624Z
Brain Efficiency Cannell Prize Contest Award Ceremony 2023-07-24T11:30:10.602Z
Towards Developmental Interpretability 2023-07-12T19:33:44.788Z
Crystal Healing — or the Origins of Expected Utility Maximizers 2023-06-25T03:18:25.033Z
Helio-Selenic Laser Telescope (in SPACE!?) 2023-05-26T11:24:26.504Z
Towards Measures of Optimisation 2023-05-12T15:29:33.325Z
$250 prize for checking Jake Cannell's Brain Efficiency 2023-04-26T16:21:06.035Z
Singularities against the Singularity: Announcing Workshop on Singular Learning Theory and Alignment 2023-04-01T09:58:22.764Z
Hoarding Gmail-accounts in a post-CAPTCHA world? 2023-03-11T16:08:34.659Z
Interview Daniel Murfet on Universal Phenomena in Learning Machines 2023-02-06T00:00:29.407Z
New Years Social 2022-12-26T01:22:31.930Z
Alexander Gietelink Oldenziel's Shortform 2022-11-16T15:59:54.709Z
Entropy Scaling And Intrinsic Memory 2022-11-15T18:11:42.219Z
Beyond Kolmogorov and Shannon 2022-10-25T15:13:56.484Z
Refine: what helped me write more? 2022-10-25T14:44:14.813Z
Refine Blogpost Day #3: The shortforms I did write 2022-09-16T21:03:34.448Z
All the posts I will never write 2022-08-14T18:29:06.800Z
[Linkpost] Hormone-disrupting plastics and reproductive health 2021-10-19T11:01:37.292Z
Self-Embedded Agent's Shortform 2021-09-02T10:49:45.449Z
Are we prepared for Solar Storms? 2021-02-17T15:38:03.338Z
What's the evidence on falling testosteron and sperm counts in men? 2020-08-10T08:58:47.851Z
[Reference request] Can Love be Explained? 2020-07-07T10:09:17.508Z
What is the scientific status of 'Muscle Memory'? 2020-07-07T09:57:12.311Z
How credible is the theory that COVID19 escaped from a Wuhan Lab? 2020-04-03T06:47:08.646Z
The Intentional Agency Experiment 2018-07-10T20:32:20.512Z

Comments

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Alexander Gietelink Oldenziel's Shortform · 2024-03-28T01:04:20.078Z · LW · GW

Thank you, Thomas. I believe we find ourselves in broad agreement. The distinction you make between lay-legibility and expert-legibility is especially well-drawn.

One point: the confidence of researchers in their own approach may not be the right thing to look at. Perhaps a better measure is seeing who can predict not only their own approach will succed but explain in detail why other approaches won't work. Anecdotally, very succesful researchers have a keen sense of what will work out and what won't - in private conversation many are willing to share detailed models why other approaches will not work or are not as promising. I'd have to think about this more carefully but anecdotally the most succesful researchers have many bits of information over their competitors not just one or two. (Note that one bit of information means that their entire advantage could be wiped out by answering a single Y/N question. Not impossible, but not typical for most cases)

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Modern Transformers are AGI, and Human-Level · 2024-03-28T00:56:03.562Z · LW · GW

Perhaps.

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Daniel Kahneman has died · 2024-03-27T17:40:15.552Z · LW · GW

What a heart-warming story!

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Alexander Gietelink Oldenziel's Shortform · 2024-03-27T17:25:14.517Z · LW · GW

Thanks for your skepticism, Thomas. Before we get into this, I'd like to make sure actually disagree. My position is not that scientific progress is mostly due to plucky outsiders who are ignored for decades. (I feel something like this is a popular view on LW). Indeed, I think most scientific progress is made through pretty conventional (academic) routes.

I think one can predict that future scientific progress will likely be made by young smart people at prestigious universities and research labs specializing in fields that have good feedback loops and/or have historically made a lot of progress: physics, chemistry, medicine, etc

My contention is that beyond very broad predictive factors like this, judging whether a research direction is fruitful is hard & requires inside knowledge. Much of this knowledge is illegible, difficult to attain because it takes a lot of specialized knowledge etc.

Do you disagree with this ?

I do think that novel research is inherently illegible. Here are some thoughts on your comment :

1.Before getting into your Nobel prize proposal I'd like to caution for Hindsight bias (obvious reasons).

  1. And perhaps to some degree I'd like to argue the burden of proof should be on the converse: show me evidence that scientific progress is very legible. In some sense, predicting what directions will be fruitful is a bet against the (efficiënt ?) scientific market.

  2. I also agree the amount of prediction one can do will vary a lot. Indeed, it was itself an innovation (eg Thomas Edison and his lightbulbs !) that some kind of scientific and engineering progress could by systematized: the discovery of R&D.

I think this works much better for certain domains than for others and a to large degree the 'harder' & more 'novel' the problem is the more labs defer 'illegibly' to the inside knowledge of researchers.

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Modern Transformers are AGI, and Human-Level · 2024-03-26T20:09:53.488Z · LW · GW

 I don't know how I feel about pushing this conversation further. A lot of people read this forum now. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Modern Transformers are AGI, and Human-Level · 2024-03-26T20:08:20.393Z · LW · GW

Yes. This seems so obviously true to me in way that it is profoundly mysterious to me that almost everybody else seems to disagree. Then again, probably it's for the best.  Maybe this is the one weird timeline where we gmi because everybody thinks we already have AGI. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Alexander Gietelink Oldenziel's Shortform · 2024-03-26T12:04:42.305Z · LW · GW

Novel Science is Inherently Illegible

Legibility, transparency, and open science are generally considered positive attributes, while opacity, elitism, and obscurantism are viewed as negative. However, increased legibility in science is not always beneficial and can often be detrimental.

Scientific management, with some exceptions, likely underperforms compared to simpler heuristics such as giving money to smart people or implementing grant lotteries. Scientific legibility suffers from the classic "Seeing like a State" problems. It constrains endeavors to the least informed stakeholder, hinders exploration, inevitably biases research to be simple and myopic, and exposes researchers to constant political tug-of-war between different interest groups poisoning objectivity. 

I think the above would be considered relatively uncontroversial in EA circles.  But I posit there is something deeper going on: 

Novel research is inherently illegible. If it were legible, someone else would have already pursued it. As science advances her concepts become increasingly counterintuitive and further from common sense. Most of the legible low-hanging fruit has already been picked, and novel research requires venturing higher into the tree, pursuing illegible paths with indirect and hard-to-foresee impacts.

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Alexander Gietelink Oldenziel's Shortform · 2024-03-26T11:59:06.995Z · LW · GW

Know your scientific competitors. 

In trading, entering a market dominated by insiders without proper research is a sure-fire way to lose a lot of money and time.  Fintech companies go to great lengths to uncover their competitors' strategies while safeguarding their own.

A friend who worked in trading told me that traders would share subtly incorrect advice on trading Discords to mislead competitors and protect their strategies.

Surprisingly, in many scientific disciplines researchers are often curiously incurious about their peers' work.

The long feedback loop for measuring impact in science, compared to the immediate feedback in trading, means that it is often strategically advantageous to be unaware of what others are doing. As long as nobody notices during peer review it may never hurt your career.  

But of course this can lead people to do completely superflueous, irrelevant  & misguided work. This happens often. 

Ignoring competitors in trading results in immediate financial losses. In science, entire subfields may persist for decades, using outdated methodologies or pursuing misguided research because they overlook crucial considerations. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Leading The Parade · 2024-03-25T14:00:25.801Z · LW · GW

Averaging over all coalitions seems quite natural to me; it averages out the "incidental, contigent, unfair" factor of who got in what coalition first. But tastes may differ. 

Shapley value has many other good properties nailing it down as a canonical way to allocate credit. 

Quoting from nunoSempere's article

The Shapley value is uniquely determined by simple properties.

These properties:

  • Property 1: Sum of the values adds up to the total value (Efficiency)
  • Property 2: Equal agents have equal value (Symmetry)
  • Property 3: Order indifference: it doesn't matter which order you go in (Linearity). Or, in other words, if there are two steps, Value(Step1 + Step2) = Value(Step1) + Value(Step2).

And an extra property:

  • Property 4: Null-player (if in every world, adding a person to the world has no impact, the person has no impact). You can either take this as an axiom, or derive it from the first three properties.

 

In the context of scientific contributions, one might argue that property 1 & 2 are very natural, axiomatic while property 3 is merely very reasonable. 

I agree Shapley value per se isn't the answer to all questions of credit. For instance, the Shapley value is not compositional: merging players into a single player doesn't preserve Shapley values. 

Nevertheless, I feel it is a very good idea that has many or all properties people want when they talk about a right notion of credit. 

 

  • I don't know what you mean by UDT/EDT in this context - I would be super curious if you could elucidate! :)
  • What do you mean by maximizing Shapley value gives crazy results? (as I point out above, Shapley value isn't the be all and end all of all questions of credit and in e.g. hierarchichal composition of agency isn't well-behaved). 
Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Leading The Parade · 2024-03-22T12:41:34.574Z · LW · GW

Great comment Ajsja, you hit the mark. Two small comments:

(i) The 'correct', 'mathematically righteous' way to calculate credit is through a elaboration of counterfactual impact: the Shapley value. I believe it captures the things you want from credit that you write here. 

(ii) On Turing being on the scent of information theory -  I find this quote not that compelling. The idea of information as a logarithmic quantity was important but only a fraction of what Shannon did. In general, I agree with Schmidthuber's assesment that Turing' scientific stature is a little overrated.

A better comparison would probably be Ralph Hartley pioneered information-theoretic ideas  (see e.g. the Shannon-Hartley theorem). I'm sure you know more about the history here than I do. 

I'm certain one could write an entire book about the depth, significance and subtlety of Claude Shannon's work. Perennially underrated. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Are extreme probabilities for P(doom) epistemically justifed? · 2024-03-22T12:11:25.641Z · LW · GW

I think I agree. 

For my information, what's your favorite reference for superforecasters outperforming domain experts?

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Are extreme probabilities for P(doom) epistemically justifed? · 2024-03-22T11:03:18.548Z · LW · GW

Thank you ! glad you liked it. ☺️

LessWrong & EA is inundated with repeating the same.old arguments for ai x-risk in a hundred different formats. Could this really be the difference ?

Besides, arent superforecasters supposed to be the Kung Fu masters of doing their own research ;-)

I agree with you that a crux is base rate relevancy. Since there is no base rate for x-risk I'm unsure how to translate this to superforecaster language tho

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Alexander Gietelink Oldenziel's Shortform · 2024-03-16T20:08:56.948Z · LW · GW

Feature request: author-driven collaborative editing [CITATION needed] for the Good and Glorious Epistemic Commons.

Often I find myself writing claims which would ideally have citations but I don't know an exact reference, don't remember where I read it, or am simply too lazy to do the literature search. 

This is bad for scholarship is a rationalist virtue. Proper citation is key to preserving and growing the epistemic commons. 

It would be awesome if my lazyness were rewarded by giving me the option to add a [CITATION needed] that others could then suggest (push) a citation, link or short remark which the author (me) could then accept. The contribution of the citator is acknowledged of course. [even better would be if there was some central database that would track citations & links like with crosslinking etc like wikipedia] 

a sort hybrid vigor of Community Notes and Wikipedia if you will. but It's collaborative, not adversarial*

author: blablablabla

sky is blue [citation Needed]

blabblabla

intrepid bibliographer: (push) [1] "I went outside and the sky was blue", Letters to the Empirical Review

 

*community notes on twitter has been a universally lauded concept when it first launched. We are already seeing it being abused unfortunately, often used for unreplyable cheap dunks. I still think it's a good addition to twitter but it does show how difficult it is to create shared agreed-upon epistemics in an adverserial setting. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Natural Abstractions: Key claims, Theorems, and Critiques · 2024-03-16T16:15:35.132Z · LW · GW

John has several lenses on natural abtractions:

  • natural abstraction as information-at-a-distance 
  • natural abstraction = redundant & latent representation of information 
  • natural abstraction = convergent abstraction for 'broad' class of minds

the thing that felt closest to me to the Quantum Darwinism story that Jess was talking about as the 'redudant/ latent story, e.g. https://www.lesswrong.com/posts/N2JcFZ3LCCsnK2Fep/the-minimal-latents-approach-to-natural-abstractions and https://www.lesswrong.com/posts/dWQWzGCSFj6GTZHz7/natural-latents-the-math

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Natural Abstractions: Key claims, Theorems, and Critiques · 2024-03-16T16:06:20.259Z · LW · GW

Curious if @johnswentworth has any takes on this.

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Epistemic Motif of Abstract-Concrete Cycles & Domain Expansion · 2024-03-15T15:53:22.794Z · LW · GW

Update: have been reading Grothenieck, a mathematical portrait. A remarkable book. Recommend. One does need a serious acquitance with scheme theory & related fields to get most out of it. 

One takeaway for me is that Grothenieck's work was more evolutionary than revolutionary. 
Many ideas often associated with scheme theory were already pioneered by others, e.g. : 

The idea of generic points  and specializations(Andre Weil),  lifting from characteristic p to zero (Weil & Zariski), definition of etale fundamental group & description in terms of complex fundamental group (Abhyankar, Zariski), the concept & need for the etale cohomology (Weil*), notion of etale maps & importance in cohomological computations (Serre), prime spectrum of a ring (Krull, many others), notion of scheme (Cartier, Serre, Chevalley, others. The name of schema is due to Chevalley iirc?), infinitesimals as key to deformations (Italian school - famously imprecise, but had many of the geometric ideas), jet & arc schemes (pioneered by John Nash Jr. - yes the Beautiful Mind John Nash), category theory & Yoneda lemma (invented and developed in great detail by Maclane-Eilenberg), locally ringed spaces (Cartan's school), spectral sequences (invented by Leray), sheaf theory and sheaf cohomology (Leray, then introduced by Serre into algebraic geometry), injective/projective resolutions and abstract approach to cohomology (common technique in group cohomology).

More generally the philosophy of increased abstrraction and rising-sea style mathematics was common in the 'French school', famously as espounded by Nicolas Bourbaki. 

One wouldn't be wrong to say that, despite the scheme-theoretic language, 90% of the ideas in the standard algebraic geometry textbook of Hartshorne precede Grothendieck. 

As pointed out clearly by Jean-Pierre Serre, the Grothendieckian style of mathematics wasn't universally succesful as many mathematical problems & phenomena resist a general abstract framework.

[Jean-Pierre Serre favored the big tent esthetic of 'belle choses' (all things beautiful), appreciating the whole spectrum, the true diversity of mathematical phenomena, from the ultra-abstract and general to the super-specific and concrete.]

What then were the great contributions to Alexandre Grothendieck?

Although the abstraction has become increasingly dominant in modern mathematics, most famously pioneered by the French school of Bourbarki, Grothendieck was surely the more DAKKA Master of this movement in mathematics pushing the creation and utilization of towering abstractions to new heights. 

Yet, in a sense, much of the impact of Grothendieck's work was only felt many decades later, indeed much of its impact is perhaps yet to come. 

To name just a few: the central role of scheme theory in the later breakthrus of arithmetic geometry (Mazur, Faltings, Langlands, most famously Wiles), (higher) stacks, anabelian geometry, galois-teichmuller theory, the elephant of topos theory. There are many other fields of which I must remain silent. 

On the other hand, although Grothendieck envisioned topos theory he did not appreciate the (imho) very important logical & type-theoretic aspects of topos theory, which were pioneered by Lawvere, Joyal and (many) others. And although Grothendieck envisioned the centrality and importance of a very abstract homotopy theory very similar to the great influence and character of homotopy theory today, he was weirdly allergic for the simplicial techniques that are the bread-and-butter of modern homotopy theory. Indeed, simplicial techniques lie at the heart of Jacob Lurie's work, surely the mathematician who most can lay the claim to be Grothendieck's heir. 

* indeed Weil's conjectures were much more than a set of mere guesses but a precise set of conjectures, of which he proved important special cases, provided numerical evidence. Central to the conjectures was the realization & proof strategy that a conjectural cohomology theory -'Weil cohomology'- would lead to the proof of these conjectures.  Importantly, this involved a very precise description of conjectured properties the conjectured cohomology theory. Clearly circumscribing the need and use for a hypothetical mathematical object is an important contribution. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Fixing The Good Regulator Theorem · 2024-03-13T19:00:56.718Z · LW · GW

Would you be able to share Abram's answer in written form? I'd be keen to hear what he has to say. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Why does generalization work? · 2024-03-12T12:40:45.590Z · LW · GW

A footnote to information degrades over distance that you might be interested in: 

Usually long-range correlations are small ('information degrades over distance'), both over distance and scale. But not always. In very special situations long-range correlations can be large both over distance and over scale. I.e. the proverbial butterfly wingclap that causes a hurricane at the other side of the world. 

in solid-state physics, condensed matter and a number of other fields people are interested in phase transitions. During phase transitions long-range correlations can become very large. 

There is some fancy math going under monickers like 'conformal field theory, virasoro algebra' iirc. I know nothing about this but @Daniel Murfet  might be able to say more. 

see also: https://en.wikipedia.org/wiki/Self-organized_criticality, sandpiles

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Semi-Simplicial Types, Part I: Motivation and History · 2024-03-11T08:18:19.793Z · LW · GW

If I remember correctly Benabou points out that indexed categories correspond to split fibrations. The splitting internally corresponds to a notion of equality. The existence of a splitting for every fibration uses (is equivalent to?) the axiom of choice. In my reading, Benabou seems to say something to the effect that in naive category theory (read 'synthetically') not all categories come with an equality, each such equality structure corresponds to a splitting of the fibration obtained by externalization.

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Semi-Simplicial Types, Part I: Motivation and History · 2024-03-10T15:42:24.171Z · LW · GW

Synthetics comes to LessWrong! Very cool.  

For people just tuning in wondering what the significance of this math is:  I see this kind of research potentially may lead to new and powerful methods to deal with self-reflection & self-modification. At the moment, current tools are quite limited.  

I'd be curious what your take on Benabou's perspective is which sees fibrations as more fundamental than indexed categories and would probably bristle at trying to prove they're 'equal'/ equivalent. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on 0th Person and 1st Person Logic · 2024-03-10T15:23:38.332Z · LW · GW

see also Infra-Bayesian physicalism which aims to formulate 'bridge rules' that bridge between the 0P and 1P perspective. 

 

In the context of 0P vs 1P perspectives, I've always found Diffractor's Many faces of Beliefs to be enticing. One day anthropics will be formulated in adjoint functors no doubt. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Why does generalization work? · 2024-02-20T23:12:40.103Z · LW · GW

Yes. The ultimate theory of physics will be agent-centric. It never bottoms out in low-level physics, simulation of simulations all the way down. Dude.

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Examples of governments doing good in house (or contracted) technical research · 2024-02-15T01:22:09.024Z · LW · GW

I upgraded strongly on the competence of the U.S. military-industrial complex. The development and deployment of the F-35 Lightning by Lockheed Martin, under the Joint Strike Fighter (JSF) program, stand as a testament to the success of government contracting in complex defense projects.

Despite the barrage of criticisms and doubts surrounding the JSF program—ranging from its perceived financial imprudence to its operational viability—the F-35 has emerged as a remarkable technical and military achievement. Critics often highlighted the program's excessive costs, questioned the aircraft's maneuverability, and debated its cost-effectiveness compared to fourth-generation fighters. The Air Force's emphasis on stealth technology was mockingly dismissed as an overindulgence in high-tech gadgetry.

Indeed, the JSF program, with its budget exceeding one trillion dollars and notable delays, faced significant challenges. These were, in part, due to the ambitious engineering advancements it pursued. However, the narrative has shifted, recognizing the F-35 as a superior aircraft. Its per-unit cost, now aligning with that of fourth-generation fighters at 75-90 million dollars, and its operational capabilities, particularly in stealth, highlight its value. The F-35's stealth technology, reducing its detection range to 20-30 miles compared to the 100 miles or more range of standard air-to-air missiles, gives it an enormous advantage over non-stealth fighters. And infact, in military exercises, where the F-35 demonstrated a 20:1 kill ratio against fourth-generation fighters.

The global military community's growing preference for the F-35 over its European and Russian counterparts, along with China's efforts to replicate its technology, further underscores its strategic importance.* While acknowledging the JSF program's cost overruns and delays, I think the overwhelming tactical edge provided by stealth technology would justify its expense in the context of hopefully hypothetical conventional future conflicts. **

*The JSF plans were stolen by Chinese spies. Though a significant security breach, it does not detract from the program's engineering successes. 

** Yes, nukes are a thing. As is military propaganda. Let us hope none of these gadgets will ever be tested. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Alexander Gietelink Oldenziel's Shortform · 2024-02-09T22:47:43.010Z · LW · GW

The point you make in the your first paragraph is contained in the original shortform post. The point of the post is exactly that an UDASSA-style argument can nevertheless recover something like a 'distribution of likely slowdown factors'. This seems quite curious.

I suggest reading Falkovich's post on UDASSA to get a sense whats so intriguing abouy the UDASSA franework.

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Alexander Gietelink Oldenziel's Shortform · 2024-02-09T01:57:46.104Z · LW · GW

[This is joint thinking with Sam Eisenstat. Also thanks to Caspar Oesterheld for his thoughtful comments. Thanks to Steve Byrnes for pushing me to write this out.]


The Hyena problem in long-term planning  

Logical induction is a nice framework to think about bounded reasoning. Very soon after the discovery of logical induction people tried to make logical inductor decision makers work. This is difficult to make work: one of two obstacles is

Obstacle 1: Untaken Actions are not Observable

Caspar Oesterheld brilliantly solved this problem by using auction markets in defining his bounded rational inductive agents. 

The BRIA framework is only defined for single-step/  length 1 horizon decisions. 

What about the much more difficult question of long-term planning? I'm going to assume you are familiar with the BRIA framework. 

 

Setup: we have a series of decisions D_i, and rewards R_i, i=1,2,3... where rewards R_i can depend on arbitrary past decisions. 

We again think of an auction market M of individual decisionmakers/ bidders.

There are a couple design choices to make here:

  • bidders directly bet for an action A in a decision D_i or bettors bet for rewards on certain days.
  • total observability or partial observability. 
  • bidders can bid conditional on observations/ past actions or not
  • when can the auction be held? i.e. when is an action/ reward signal definitely sold?

To do good long-term planning it should be possible for one of the bidders or a group of bidders to commit to a long-term plan, i.e. a sequence of actions. They don't want to be outbid in the middle of their plan. 

There are some problems with the auction framework: if bids for actions can't be combined then an outside bidder can screw up the whole plan by making a slighly higher bid for an essential part of the plan. This look like ADHD. 

How do we solve this? One way is to allow a bidder or group of bidders to bid for a whole sequence of actions for a single lumpsum. 

  • One issue is that we also have to determine how the reward gets awarded. For instance the reward could be very delayed. This could be solved by allowing for bidding for a reward signal R_i on a certain day conditional on a series of actions. 

There is now an important design choice left. When a bidder  owns a series of actions A=a_1,..,a_k (some of the actions in the future, some already in the past) when there is another bid  from another bidder  on future actions 

  • is bidder  forced to sell their contract on  to  if the bid is high enough ? [higher than the original bid]

Both versions seem problematic:

  • if they don't have to there is an Incumbency Advantage problem. An initially rich bidder can underbid for very long horizons and use the steady trickle of cash to prevent any other bidders from ever being to underbid any actions. 
  • Otherwise there is the Hyena problem. 

The Hyena Problem 

Imagine the following situation: on Day 1 the decisionmaker has a choice of actions. The highest expected value action is action a. If action a is made on Day 2 a fair coin is flipped. On Day 3 the reward is paid out. 

If the coin was heads, 15 reward is paid out.

If the coin was tails, 5 reward is paid out.

The expected value is therefore 10. This is higher (by assumption) than the other unnamed actions. 

However if the decisionmaker is a long-horizon BRIA with forced sales there is a pathology. 

A sensible bidder is willing to pay up to 10 utilons for the contracts on the day 3 reward conditional on action a. 

However, with a forced sale mechanism on Day 2 a 'Hyena bidder'  can come that will 'attempt to steal the prey'. 

The Hyena bidder bids >10 for the contract if the coin comes up heads on Day 2 but doesn't bid anything for the contract if the coin comes up tails.

This is a problem since the expected value of the action a for the sensible bidder goes down, so the sensible bidder might no longer bid for the action that maximizes expected value for the BRIA. The Hyena bidder screws up the credit allocation. 

some thoughts:

  • if the sensible bidder is able to make bids conditional on the outcome of the coin flip that prevents Hyena bidder. This is a bit weird though because it would mean that the sensible bidder must carry around lots of extraneous non-necessary information instead of just caring about expected value.
  • perhaps this can alleviated by having some sort of 'neo-cortex' separate logical induction markets that is incentivized to have accurate beliefs. This is difficult to get right: the prediction market needs to be incentivized to get accurate on beliefs that are actually action relevant, not random beliefs - if the prediction market and the auction market are connected too tightly you might run the risk of getting into the old problems of Logical Inductor Decision makers. [they underexplore since untaken action are not observed]. 
Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Entropy Scaling And Intrinsic Memory · 2024-02-07T21:32:11.364Z · LW · GW

Here you go https://arxiv.org/abs/1412.2690

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Alexander Gietelink Oldenziel's Shortform · 2024-02-05T18:04:48.615Z · LW · GW

Idle thoughts about UDASSA I: the Simulation hypothesis 

I was talking to my neighbor about UDASSA the other day. He mentioned a book I keep getting recommended but never read where characters get simulated and then the simulating machine is progressively slowed down. 

One would expect one wouldn't be able to notice from inside the simulation that the simulating machine is being slowed down.

This presents a conundrum for simulation style hypotheses: if the simulation can be slowed down 100x without the insiders noticing, why not 1000x or 10^100x or quadrilliongoogolgrahamsnumberx? 

If so - it would mean there is a possibly unbounded number of simulations that can be run. 

Not so, says UDASSA. The simulating universe is also subject to UDASSA. This imposes a restraint on the size and time period that the simulating universe is in. Additionally, ultraslow computation is in conflict with thermodynamic decay - fighting thermodynamic decay costs descriptiong length bits which is punished by UDASSA. 

I conclude that this objection to simulation hypotheses are probably answered by UDASSA. 

Idle thoughts about UDASSA II: Is Uploading Death?

There is an argument that uploading doesn't work since encoding your brain into a machine incurs a minimum amount of encoding bits. Each bit is a 2x less Subjective Reality Fluid according to UDASSA so even a small encoding cost would mean certain subjective annihiliation. 

There is something that confuses me in this argument. Could it not be possible to encode one's subjective experiences even more efficiently than in a biological body? This would make you exist MORE in an upload.

OTOH it becomes a little funky again when there are many copies as this increases the individual coding cost (but also there are more of you sooo). 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Leading The Parade · 2024-02-02T16:39:23.788Z · LW · GW

This is obviously the correct way to interpret what's happening. At some point the per person Shapley value becomes small but I'd guess that the shapely impact of Newton & Leibniz is substantial for quite a long time.

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Will quantum randomness affect the 2028 election? · 2024-01-25T19:37:55.832Z · LW · GW

I strongly feel Habryka is right here. Things are not that contingent. In particular, the invocation of chaos theory feels misleading here. The weather is chaotic on relevant timescales but most of our world and society is very much not.

Interested to hear different intuitions

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on 1a3orn's Shortform · 2024-01-18T20:05:30.375Z · LW · GW

The No Agentic Foundation Models Club ? 😁

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on 1a3orn's Shortform · 2024-01-15T19:52:34.314Z · LW · GW

I agree. AI safety advocates seem to be myopically focused on current-day systems. There is a lot of magical talk about LLMs. They do exactly what they're trained to: next-token prediction. Good predictions requires you to implicitly learn natural abstractions. I think when you absorb this lesson the emergent abilities of gpt isn't mega surprising.

Agentic AI will come. It won't be just a scaled up LLM. It might grow as some sort of gremlin inside the llm but much more likely imho is that people build agentic AIs because agentic AIs are more powerful. The focus on spontaneous gremlin emergence seems like a distraction and motivated partially by political reasons rather than a dispassionate analysis of what's possible.

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Against most, but not all, AI risk analogies · 2024-01-14T09:03:12.295Z · LW · GW

I guess I'm missing something crucial here.

How could you reason without analogies about a thing that doesn't exist (yet)?

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Epistemic Motif of Abstract-Concrete Cycles & Domain Expansion · 2024-01-11T13:00:37.508Z · LW · GW

I know 'Theorems as Definitions' as 'French definitions'. 

In algebraic geometry the main objects of study are algebraic varieties and more generally schemes. Although the proper definition of schemes is famously involved one can best think of them as the space of solutions of polynomials equations ('zeroes of polynomials'). The natural topology on this space (the Zariski topology) is quite weird - it is generically non-Hausdorff and can even have non-closed points.  For instance, the 'right way' of saying a scheme is Hausdorff is not asking for the underlying topological space to be Hausdorff [which only gives very trivial examples] but to ask for the diagonal map to be a closed subscheme. 

This is an example of your abstract-concrete cycle: for ordinary topological spaces Hausdorff and closed diagonal are equivalent, but the latter can be defined more broadly for schemes. 

Grothendieck is famous for his many French definitions. Many of the constructions he proposed were inspired by constructions and theorems in classicial differential and complex analytic geometry but then ported to the algebraic and arithmetic realm. 

Grothendieck predecessors  noted that many features of algebraic varieties resemble those of more traditionally geometric spaces. I.e. Andre Weil, in his Weil conjectures observed that the number of solutions of a variety over finite fields (i.e. modulo p) is closely related to the number of higher-dimensional holes ('Betti numbers') of the complex analytic manifold associated to the set of equations of the variety. This geometric shadow was a guiding motif for Grothendieck's work. He realized that the Zariski topology did not properly capture the 'inherent' geometric structure of algebraic equations. 

The naive solution to this conundrum would be to look for a topology on the sets of solutions of polynomials that is Hausdorff to replace the Zariski topology. But this is the wrong move - there is no such topology. Instead Grothendieck noted that the properties that are required these geometric analogues are only a subset of the properies of a topological space. Roughly speaking, for (co)homological reasoning to go through one only needs to known when a collection of opens 'cover' another open. This is an instance of your 'Abstract-Concrete-Cycle'.

By pursuing this analogy He generalized the notion of topological space to Grothendieck topology (which leads to fancy things like sites and topoi). This leads to the famous etale topology and etale cohomology which eventually lead to the proof of the Weil conjectures.

An early triumph of the Grothendieckian way of thinking was his French definition of the etale fundamental group. Naively trying to define a fundamental group for an algebraic variety fails because there aren't enough loops, hence the usual fundamental group is almost always trivial. This is Zariski topology weirdness. 

Instead, Grothendieck noted that there is a classical theorem that the fundamental group can also be seen as an automorphism group of locally trivial topological covers. He then identified a property 'etaleness' (following a suggestion by Serre) that is the analogy of locally trivial topological covers for algebraic varieties and schemes. The étale fundamental group is then simply defined in terms of automorphisms of these etale covers. This may strike one as just a mummer's trick - but it's actually profound because it turns out Galois theory becomes a special case of the theory of etale fundamental groups ('Grothendieck-Galois theory'). 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Self-Embedded Agent's Shortform · 2024-01-10T15:49:37.560Z · LW · GW

Wow, I missed this comment! This is a fantastic example, thank you!

have been meaning to write the concept splintering megapost - your comment might push me to finish it before the Rapture :D

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Dalcy's Shortform · 2024-01-10T15:46:30.203Z · LW · GW

Great question. I don't have a satisfying answer. Perhaps a cynical answer is survival bias - we remember the asymptotic results that eventually become relevant (because people develop practical algorithms or a deeper theory is discovered) but don't remember the irrelevant ones. 

Existence results are categorically easier to prove than explicit algorithms. Indeed,  classical existence may hold (the former) while intuitioinistically (the latter) might not. We would expect non-explicit existence results to appear before explicit algorithms. 

One minor remark on 'quantifying over all boolean algorithms'. Unease with quantification over large domains may be a vestige of set-theoretic thinking that imagines types as (platonic) boxes. But a term of a for-all quantifier is better thought of as an algorithm/ method to check the property for any given term (in this case a Boolean circuit). This doesn't sound divorced from practice to my ears. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Defending against hypothetical moon life during Apollo 11 · 2024-01-08T08:46:15.682Z · LW · GW

Oh I'm sorry.. I was mistaken and garbled abiogenesis and panspermia.

Your post was fascinating thank you

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Defending against hypothetical moon life during Apollo 11 · 2024-01-07T11:53:47.639Z · LW · GW

"extremely-widely-accepted earth-abiogenesis hypothesis ". Link?

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Johannes C. Mayer's Shortform · 2024-01-06T10:41:45.024Z · LW · GW

This is an interesting concept. I wish it became a post.

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on What technical topics could help with boundaries/membranes? · 2024-01-05T22:39:29.634Z · LW · GW

Iirc it's in the original boundaries sequence. Sorry I'm too lazy to look it up.

It's the part about natural bargaining points and people being able to 'go home'

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on What technical topics could help with boundaries/membranes? · 2024-01-05T21:06:56.975Z · LW · GW

Great initiative !

I've always been intrigued by Andrew Critch's suggestion that the problem of commitment races in game theory and other forms of threats may be 'resolved' by thinking in terms of boundaries as natural BATNA points

https://www.lesswrong.com/tag/commitment-races

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Apologizing is a Core Rationalist Skill · 2024-01-03T11:51:14.771Z · LW · GW

Seems people are reading my message in passive aggresive tone. The original message should be read without any irony. I think it's good to (publicly) apologize and I think it's even better that John is writing a separate post to say it again (I missed the original). 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Cortés, AI Risk, and the Dynamics of Competing Conquerors · 2024-01-02T20:56:45.838Z · LW · GW

Thanks gpt !

 the model of AI x-risk based on an analogue with Cortes and similar colonial adventurous has been and still is to my simple mind the best model to think about AI x-risk.

I think this is currently probably one of the best resources on What Failure Looks Like. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Apologizing is a Core Rationalist Skill · 2024-01-02T20:50:02.083Z · LW · GW

Here. Have some more karma. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on The Plan - 2023 Version · 2023-12-30T13:19:18.965Z · LW · GW

largely-correct physics).

https://mathwithbaddrawings.com/2019/03/20/the-historical-evidence-that-galileo-was-a-hack/

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on The Plan - 2023 Version · 2023-12-30T13:12:02.191Z · LW · GW

My current best guess is that singular learning theory has the right conceptual picture for the "selection theorems" prong, and the folks working on it are headed in the right direction.

News to me 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on re: Yudkowsky on biological materials · 2023-12-29T16:56:53.189Z · LW · GW

Are you saying it is possible to construct a flying machine utilizing wings that would compete with a jet fighter in speed and fuel-efficiency?

[I would doubt it]

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on re: Yudkowsky on biological materials · 2023-12-29T16:39:51.442Z · LW · GW

A single generation difference in military technology is an overwhelming advantage. The JSF F35 Lockheed Martin Lightning II cannot be missile-locked by an adversary beyond 20-30 miles. Conversely, it can see and weapon lock an opposing 4th gen fighter from >70 miles fire a beyond-visual-range missile that is almost impossible to evade for a manned fighter. 

In realistic scenarios with adequate preparation and competent deployment a generation difference in aircraft can lead to 20/1 K/D ratios. 5th generations fighters are much better than 4th generation fighters are much better than 3rd generation fighters etc. Same for tanks, ships, artillery, etc. This difference is primarily technological. 

It is not at all unlikely to suppose that a machine superintelligence could not only rapidly design new materials, artificial organisms and military technologies vastly better than those constructed by humans today. These could indeed be said  to form superweapons. 

The idea that AI-designed nanomachines will outcompete bacteria and consume the world in a grey goo swarm perhaps may seem fanciful but that's not at all evidence that it isn't in the cards. Now, there are goodish technical arguments that bacteria are already at various thermodynamic limits. As bhauth notes it seems that Yudkowsky underrates the ability of evolution-by-natural-selection to find highly optimal structures. 

However, I don't see this enough evidence to prohibiting grey goo scenarios. Being somewhere at a Pareto optimum doesn't mean you can't be outcompeted. Evolution is much more efficient than it is sometimes given credit for but it still seems to miss obvious improvements. 

Of course, nanotech is likely a superweapon even without grey goo scenarios so this is only a possible extreme. And finally of course (a) mechanical superintelligence(s) posesses many advantages over biological humans any of which may prove more relevant for a take-over scenario in the short-term. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Natural Latents: The Math · 2023-12-29T16:11:59.336Z · LW · GW

It is a truth universally acknowledged that any article about the hidden nature of reality will not garner close to the number of comments as one about your sexy sexy homies. As the inaugural and possibly lone commenter, let me say I'm delighted to see finally this long-awaited post. I look forward to do a deep dive soon. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on Critical review of Christiano's disagreements with Yudkowsky · 2023-12-28T13:16:59.067Z · LW · GW

Scaling laws are an important phenomena and probably deeply tied with the nature of intelligence. 

I do take issue with the assertion that scaling laws imply slow takeoff. One key takeaway of the modern ML revolution is that specific details of architectures-in-the-narrow-sense* is mostly not that important and compute and data dominate.

The natural implication is that scaling laws are a function of the data distribution - and mostly not of the architecture. Just because we see a 'smooth, slow' scaling law on text data doesn't mean that this will generalize to other domains/situations/ horizons. In fact, I think we should mostly expect this not to be the case. 

*I think the jump from architectures-in-the-narrow-sense don't matter to architectures-in-the-broad-sense don't matter is often made. I think this obviously not suppored by the evidence we have sofar (despite many claims to the contrary) and likely wrong. 

Comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) on AI Girlfriends Won't Matter Much · 2023-12-24T11:57:01.133Z · LW · GW

Is your contention that without electricity US GDP/capita would be say within 70% of what it is now?

(Roll to doubt)