LessWrong 2.0 Reader

Yeah, my involvement was providing draft feedback on the article and providing some of the images. Looks like my post got taken down for being a duplicate, though

what are Smith lotteries?

Lotteries over the Smith set. That definitely wasn't clear - I'll fix that.

which result do you mean by "above result"?

• Proposition: (Lottery-lotteries are strongly characterized by their selectivity of partitions of unity) 

 This one. "You can tell whether a lottery-lottery is maximal or not by how good the partitions of unity it admits are." Sorry, didn't really know a good way to link to myself internally and I forgot to number the various statements.

What does it mean for a lottery to be part of maximal lottery-lotteries?

Just that some maximal lottery-lottery gives it nonzero probability.

does "also subject to the partition-of-unity" refer to the smith lotteries or to the lotteries that are part of maximal lottery-lotteries? (it also feels like there is a word missing somewhere)

Oh no! I thought I caught all the typos! That should be "also subject to the partition-of-unity condition", that is, you look at all the lotteries (which we know are over the Smith set, btw) that some arbitrary maximal lottery-lottery gives any nonzero probability to, and you should expect to be able to sort them into groups by what final probability over candidates they induce; those final probabilities over candidates should themselves result in identical geometric-expected utility for the voterbase.

Why would this suffice?

Consider: at this point we know that a maximal lottery-lottery would not just have to be comprised of lottery-Smith lotteries, i.e., lotteries that are in the lottery-Smith set  - but also that they have to be comprised entirely of lotteries over the Smith set of the candidate set. Then on top of that, we know that you can tell which lottery-lotteries are maximal by which partitions of unity they admit (that's the "above result"). Note that by "admit" we mean "some subset of the lotteries this lottery-lottery has support over corresponds to it" (this partition of unity).

This is the slightly complicated part. The game I described has a mixed strategy equilibrium; this will take the form of some probability distribution over $\Delta C$. In fact it won't just have one, it'll likely have whole families of them. Much of the time, the lotteries randomized over won't be disjoint - they'll both assign positive probability to some candidate. The key is, the voter doesn't care. As far as a voter's expected utility is concerned, the only thing that matters is the final probability of each candidate.

That's where your collapse of different possible maximal lottery-lotteries to the same partition of unity comes in. Because we know that equivalent candidate-lotteries give voters the same expected utility, the only two ways you get a voter who's indifferent between two candidate-lotteries are 1) they're the same lottery or 2) the voter's utility function is just right to have two very different lotteries tie. Likewise, the only two ways you get a voterbase to be indifferent between two lottery-lotteries is 1) they reduce to the same lottery or 2) the geometric expectation of a voter's utility over candidates sampled from the samples of the lottery-lottery Just Plain Ties.

So: if we can show that whatever equilibrium set of candidate-lotteries Alice and Bob pick always collapses to some convex combination of the Best partitions of unity...? Yeah, I don't think that the second half of the proof holds up as is.

I think I've slightly messed up the definition of lottery-Smith, though not in a fatal way nor (thankfully) in a way that looks to threaten the existence result. The set condition is too strong, in requiring that a lottery-Smith lottery contain all lotteries which correspond to any of the admissible partitions. I'm just going to cut it; it's not actually necessary.

Is this part also supposed to imply the existence of maximal lottery-lotteries? If so, why?

Yes. 

Yes, and in particular, it implies the existence of maximal lottery-lotteries before it even tries to prove that they're also lottery-Smith in the sense I define.

Scott's proof [? · GW] doesn't quite work (as he says there) - it almost works, except for the part where the reward functions for Alice and Bob can quite reasonably be discontinuous. My proof is intended as a patch - the reward functions for Alice and Bob should now be extremely continuous in a way that also corresponds well to "how much better did Alice do at picking a candidate-lottery that V will like than Bob did?".

Hopefully this helped? Reading this is confusing even for me sometimes - the word "lottery/lotteries", which appears 59 times in this comment alone, no longer looks like a real word to me and hasn't since late Wednesday. Your comment was really helpful - I have some editing to do! (update - editing is done.)

I'm probably missing something simple, but what is 356? I was expecting a probability or a percent, but that number is neither.

@henry [LW · GW] (who seems to know Nicky) said on a duplicate link post of this: 

This is an accessible introduction to AI Safety, written by Nicky Case and the teens at Hack Club. So far, part 1/3 is completed, which covers a rough timeline of AI advancement up to this point, and what might come next.

If you've got feedback as to how this can be made more understandable, that'd be appreciated! Reach out to Nicky, or to me and I'll get the message to her.

I agree with you. 

Epsilon machine (and MSP) construction is most likely computationally intractable [I don't know an exact statement of such a result in the literature but I suspect it is true] for realistic scenarios. 

Scaling an approximate version of epsilon reconstruction seems therefore of prime importance. Real world architectures and data has highly specific structure & symmetry that makes it different from completely generic HMMs. This must most likely be exploited. 

The calculi of emergence paper has inspired many people but has not been developed much. Many of the details are somewhat obscure, vague. I also believe that most likely completely different methods are needed to push the program further. Computational Mechanics' is primarily a theory of hidden markov models - it doesn't have the tools  to easily describe behaviour higher up the Chomsky hierarchy. I suspect more powerful and sophisticated algebraic, logical and categorical thinking will be needed here. I caveat this by saying that Paul Riechers has pointed out that actually one can understand all these gadgets up the Chomsky hierarchy as infinite HMMs which may be analyzed usefully just as finite HMMs. 

The still-underdeveloped theory of epsilon transducers I regard as the most promising lens on agent foundations. This is uncharcted territory; I suspect the largest impact of computational mechanics will come from this direction. 

Your point on True Names is well-taken. More basic examples than gauge information, synchronization order are the triple of quantites entropy rate $h$, excess entropy $E$ and Crutchfield's  statistical/forecasting complexity $C$. These are the most important quantities to understand for any stochastic process (such as the structure of language and LLMs!)

Non exhaustive list of reasons one could be interested in computational mechanics: https://www.lesswrong.com/posts/GG2NFdgtxxjEssyiE/dalcy-s-shortform?commentId=DdnaLZmJwusPkGn96 [LW(p) · GW(p)]

@jefftk [LW · GW] comments on the HN thread on this: 

How many people would, if they suddenly died, be reported as a "Boeing whistleblower"? The lower this number is, the more surprising the death.

Another HN commenter says (in a different thread): 

It’s a nice little math problem.

Let’s say both of the whistleblowers were age 50. The probability of a 50 year old man dying in a year is 0.6%. So the probability of 2 or more of them dying in a year is 1 - (the probability of exactly zero dying in a year + the probability of exactly one dying in a year). 1 - (A+B).

A is (1-0.006)^N. B is 0.006N(1-0.006)^(N-1). At 60 A is about 70% and B is about 25% making it statistically insignificant.

But they died in the same 2 month period, so that 0.006 should be 0.001. If you rerun the same calculation, it’s 356.

Does anyone have any takes on the two Boeing whistleblowers who died under somewhat suspicious circumstances? I haven't followed this in detail, and my guess is it is basically just random chance, but it sure would be a huge deal if a publicly traded company now was performing assassinations of U.S. citizens. 

Curious whether anyone has looked into this, or has thought much about baseline risk of assassinations or other forms of violence from economic actors.

Yes, such as: "Anything worth doing is worth doing [well/poorly]" needs more qualifiers to be meaningful. As it is the opposite advice can often be just as useful. I.E. not very.

Better V1: "The cost/utility ratio of beneficial actions at minimum cost are often less favorable than they would be with greater investment."

Better V2: "If an action is beneficial, a flawed attempt may be preferable to none at all."

However, these are too wordy to be pithy and in pop culture transmission accuracy is generally sacrificed in favor of catchiness.

One operationalization is "these AIs are capable of speeding up ML R&D by 30x with less than a 2x increase in marginal costs".

As in, if you have a team doing ML research, you can make them 30x faster with only <2x increase in cost by going from not using your powerful AIs to using them.

With these caveats:

• The speed up is relative to the current status quo as of GPT-4.
• The speed up is ignoring the "speed up" of "having better experiments to do due to access to better models" (so e.g., they would complete a fixed research task faster).
• By "capable" of speeding things up this much, I mean that if AIs "wanted" to speed up this task and if we didn't have any safety precautions slowing things down, we could get these speedups. (Of course, AIs might actively and successfully slow down certain types of research and we might have burdensome safety precautions.)
• The 2x increase in marginal cost is ignoring potential inflation in the cost of compute (FLOP/$) and inflation in the cost of wages of ML researchers. Otherwise, I'm uncertain how exactly to model the situation. Maybe increase in wages and decrease in FLOP/$ cancel out? Idk.
• It might be important that the speed up is amortized over a longer duration like 6 months to 1 year.

I'm uncertain what the economic impact of such systems will look like. I could imagine either massive (GDP has already grown >4x due to the total effects of AI) or only moderate (AIs haven't yet been that widely deployed due to inference availability issues, so actual production hasn't increased that much due to AI (<10%), though markets are pricing in AI being a really, really big deal).

So, it's hard for me to predict the immediate impact on world GDP. After adaptation and broad deployment, systems of this level would likely have a massive effect on GDP.