LessWrong 2.0 Reader

On the contrary, convex agents are wildly abundant -- we call them r-selected organisms.

Since AI interpretability is a big issue for AI safety, let's completely interpret the results of evolutionary computation. 

Disclaimer: This interpretation of the results of AI does not generalize to interpreting deep neural networks. This is a result for interpreting a solution to a very specific problem that is far less complicated than deep learning, and by interpreting, I mean that we iterate a mathematical operation hundreds of times to get an object that is simpler than our original object, so don't get your hopes up too much.

A basis matroid is a pair $(X,M)$ where $X$ is a finite set, and $M\subseteq P\left(X\right)$ where $M$ denotes the power set of $X$ that satisfies the following two properties:

1. If $A,B\in M$, then $|A|=|B|$.
2. if $A,B\in M,A\ne B,a\in A\setminus B$, then there is some $b\in B\setminus A$ with $(A\setminus \{a\left\}\right)\cup \left\{b\right\}\in M$ (the basis exchange property).

I ran a computer experiment where I obtained a matroid $(X,M)$ where $|X|=9$   $|M|=68$ and where each element in $M$ has size $4$ through evolutionary computation, but the population size was kept so low that this evolutionary computation mimicked hill climbing algorithms. Now we need to interpret the matroid $(X,M)$.

The notion of a matroid has many dualities. Our strategy is to apply one of these dualities to the matroid $(X,M)$ so that the dual object is much smaller than the original object $(X,M)$. One may formulate the notion of a matroid in terms of closure systems (flats),hyperplanes, closure operators, lattices, a rank function, independent sets, bases, and circuits. If these seems to complicated, many of these dualities are special cases of other dualities common with ordered sets. For example, the duality between closure systems, closure operators, and ordered sets applies to contexts unrelated to matroids such as in general and point-free topology. And the duality between the basis, circuit, and the hyperplanes may be characterized in terms of rowmotion.

If $(Z,\le )$ is a partially ordered set, then a subset $A\subseteq Z$ is said to be an antichain if whenever $a,b\in A,a\le b$, then $a=b$. In other words, an antichain is a subset $A$ of $Z$ where the restriction of $\le$ to $A$ is equality. We say that a aubset $L$ of $Z$ is downwards closed if whenever $x\le y$ and $y\in L$, then $x\in L$ as well. If $A\subseteq Z$, then let $L\left(A\right)$ denote the smallest downwards closed subset of $Z$ containing $A$. Suppose that $Z$ is a finite poset. If $A$ is an antichain in $Z$, then let $A^{\prime }$ denote the set of all minimal elements in $Z\setminus L\left(A\right)$. Then $A^{\prime }$ is an antichain as well, and the mapping $A\mapsto A^{\prime }$ is a bijection from the set of all antichains in $Z$ to itself. This means that if $A$ is an antichain, then we may define $A^{\left(n\right)}$ for all integers $n$ by setting $A^{\left(0\right)}=A,A^{(n+1)}=(A^{\left(n\right)}{)}^{\prime }$.

If $(X,M)$ is a basis matroid, then $M$ is an antichain in $P\left(X\right)$, so we may apply rowmotion, so we say that $(X,M^{\left(n\right)})$ is an $n$-matroid. In this case, the $1$

-matroids are the circuit matroids while the $-1$-matroids are the hyperplane matroids. Unfortunately, the $n$-matroids have not been characterized for $|n|>1$. We say that the rowmotion order of $(X,M)$ is the least positive integer $n$ where $M^{\left(n\right)}=M$. If $(X,M)$ is a matroid of order $n$, then my computer experiments indicate that $gcd(|X|+2,n)>1$ whichs lends support to the idea that the rowmotion of a matroid is a sensible mathematical notion that may be satisfied mathematically. The notion of rowmotion of a matroid also appears to be a sensible mathematical notion for other reasons; if we apply iteratively apply a bijective operation $g$ (such as a reversible cellular automaton) to a finite object $x$, then that bijective operation will often increase the entropy in the sense that if $x$ has low entropy, then $g^{\left(n\right)}\left(x\right)$ will typically have a high amount of entropy and look like noise. But this is not the case with matroids as $n$-matroids do not appear substantially more complicated than basis matroids. Until and if there is a mundane explanation for this behavior of the rowmotion of matroids, I must consider the notion of rowmotion of matroids to be a mathematically interesting notion even though it is currently not understood by anyone.

With the matroid $(X,M)$ obtained from evolutionary computation, I found that $(X,M)$ has order $1958$ which factorizes as $1958=2\cdot 79\cdot 11$. Set $X=\{1,\dots ,9\}$. By applying rowmotion to this matroid, I found that $(X,M^{\left(342\right)})$.

={{1, 8, 9},{2, 3, 6, 8},{2, 3, 7, 9},{4, 5},{4, 6, 9},{4, 7, 8},{5, 6, 9},{5, 7, 8}}. If $(X,M^{\left(m\right)})$ is a basis matroid, then $M^{\left(m\right)}=M$, so the set $M^{\left(342\right)}$ is associated with a unique basis matroid. This is the smallest way to represent $(X,M)$ in terms of rowmotion since if $|M^{\left(n\right)}|\le 8$, then $M^{\left(n\right)}=M^{\left(342\right)}$.

I consider this a somewhat satisfactory interpretation of the matroid $(X,M)$ that I have obtained through evolutionary computation, but there is still work to do because nobody has researched the rowmotion operation on matroids and because it would be better to simplify a matroid without needing to go through hundreds of layers of rowmotion. And even if we were to understand matroid rowmotion better, this would not help us too much with AI safety since here this interpretability of the result of evolutionary computation does not generalize to other AI's and it certainly does not apply to deep neural networks.

I made a video here where one may see the rowmotion of this matroid and that video is only slightly interpretable.

 Deep matroid duality visualization: Rowmotion of a matroid 

I partially agree, but I think this must only be a small part of the issue.

- I think there's a whole lot of key insights people could raise that aren't info-hazards. 
- If secrecy were the main factor, I'd hope that there would be some access-controlled message boards or similar. I'd want the discussion to be intentionally happening somewhere. Right now I don't really think that's happening. I think a lot of tiny groups have their own personal ideas, but there's surprisingly little systematic and private thinking between the power players. 
- I think that secrecy is often an excuse not to open ideas to feedback, and thus not be open to critique. Often, what what I see, this goes hand-in-hand with "our work just really isn't that great, but we don't want to admit it"

In the last 8 years or so, I've kept on hoping there would be some secret and brilliant "master plan" around EA, explaining the lack of public strategy. I have yet to find one. The closest I know of is some over-time discussion and slack threads with people at Constellation and similar - I think these are interesting in terms of understanding the perspectives of these (powerful) people, but I don't get the impression that there's all too much comprehensiveness of genius that's being hidden. 

That said,
- I think that policy orgs need to be very secretive, so agree with you regarding why those orgs don't write more big-picture things.

you seem to be the only user, although not the only account, who experienced this problem.

Are you accusing me of sockpuppetting?

I like Nick Land (see e.g. my comment [LW(p) · GW(p)] on jessicata's post). I've read plenty of Xenosystems. I was still confused reading your post (there are lots of headings and quotations and so on in it).

I told you my experience and opinion, mostly because you asked for feedback. Up to you how/whether you update based on it. 

does anyone think now that it's still possible to prevent recursively self-improving agents? esp now that r1 is open-source... materials for smart self-iterating agents seem accessible to millions of developers.

prompted in particular by the circulation of this essay in past three days https://huggingface.co/papers/2502.02649

As far as I can tell, OAI's new current safety practices page only names safety issues related to current LLMs, not agents powered by LLMs. https://openai.com/index/openai-safety-update/

Am I missing another section/place where they address x-risk?

I'd say that the ability to produce more energy overall than what is being spend on the whole cycle would count as a "GPT-3 moment". No price constraints, so it does not need to reach the level of "economically feasible", but it should stop being "net negative" energy-wise (when one honestly counts all energy inputs needed to make it work).

I, of course, don't know how to translate Q into this. GPT-4o tells me that it thinks that Q=10 is what is approximately needed for that (for "Engineering Break-even (reactor-level energy balance)"), at least for some of the designs, and Q in the neighborhood of 20-30 is what's needed for economic viability, but I don't really know if these are good estimates.

But assuming that these estimates are good, Q passing 10 would count as the GPT-3 moment.

What happens then might depend on the economic forecast (what's the demand for energy, what are expected profits, and so on). If they only expect to make profits typical for public utilities, and the whole thing is still heavily oriented towards publicly regulated setups, I would expect continuing collaboration.

If they expect some kind of super-profits, with market share being really important and with expectations of chunks of it being really lucrative, then I would not bet on continuing collaboration too much...

Yup, if you actually have enough knowledge to narrow it down to e.g. a 65% chance of one particular major route, then you're good. The challenging case is when you have no idea what the options even are for the major route, and the possibility space is huge.

The fourth friend, Becky the Backward Chainer, started from their hotel in LA and...

Well, no. She started at home with a telephone directory. A directory seems intelligent but is actually a giant look-up table. It gave her the hotel phone number. Ring ring.

Heidi the Hotel Receptionist: Hello?

Becky: Hi, we have a reservation for tomorrow evening. I'm back-chaining here, what's the last thing we'll do before arriving?

Heidi: It's traditional to walk in through the doors to reception. You could park on the street, or we have a parking lot that's a dollar a night. That sounds cheap but it's not because we're in the past. Would you like to reserve a spot?

Becky: Yes please, we're in the past so our car's easy to break into. What's the best way to drive to the parking lot, and what's the best way to get from the parking lot to reception?

Heidi: We have signs from the parking lot to reception. Which way are you driving in from?

Becky: Ah, I don't know, Alice is taking care of that, and she's stepped out to get more string.

Heidi: Oh, sure, can't plan a car trip without string. In the future we'll have pet nanotech spiders that can make string for us, road trips will never be the same. Anyway, you'll probably be coming in via Highway 101, or maybe via the I-5, so give us a buzz when you know.

Becky: Sorry, I'm actually calling from an analogy, so we're planning everything in parallel.

Heidi: No worries, I get stuck in thought experiments all the time. Yesterday my friend opened a box and got a million dollars, no joke. Look, get something to take notes and I'll give you directions from the three main ways you could be coming in.

Becky: Ack! Hang on while I...

Gerald the General Helper: Here's a pen, Becky.

Trevor the Clever: Get off the phone! I need to call a gas station!

Susan the Subproblem Solver: Alice, I found some string and.... Hey, where's Alice?

This doesn't seem right. Suppose there are two main candidates for how to get there, I-5 and J-6 (but who knows, maybe we'll be surprised by a K-7) and I don't know which Alice will choose. Suppose I know there's already a Very General Helper and Kinda Decent Generalizer, then I might say "I assign 65% chance that Alice is going to choose the I-5 and will try to contribute having conditioned on that". This seems like a reasonable thing to do. It might be for naught, but I'd guess in many case the EV of something definitely helpful if we go down Route A is better than the EV of finding something that's helpful no matter the choice.

One should definitely track the major route they're betting on and make updates and maybe switch, but seems okay to say your plan is conditioning on some bigger plan.