LessWrong 2.0 Reader

is bitter lesson also applicable to AI alignment? what do you think?

Let's do most of this via the much higher bandwidth medium of voice, but quickly:

1. Yes, qualia^[1] is real, and is a class of mathematical structure.^[2]
2. (placeholder for not a question item)
   1. Matter is a class of math which is ~kinda like our physics.
   2. Our part of the multiverse probably doesn't have special "exists" tags, probably everything is real (though to get remotely sane answers you need a decreasing reality fluid/caring fluid allocation).

   3. Math, in the sense I'm trying to point to it, is 'Structure'. By which I mean: Well defined seeds/axioms/starting points and precisely specified rules/laws/inference steps for extending those seeds. The quickest way I've seen to get the intuition for what I'm trying to point at with 'structure' is to watch these videos in succession (but it doesn't work for everyone):

       

1. ^{^}

   experience/the thing LWers tend to mean, not the original philosophical sense which is pointlessly high complexity stuff about epiphenomena

2. ^{^}

   possibly maybe even the entire class, though if true most qualia would be very very alien to us and not necessarily morally valuable

Here's a podcast with @Slimepriestess [LW · GW] on the whole affair. I'm not particularly involved in any of this, and had some violent disagreements with things said in the podcast^[1], but found it informative on the whole.

I'm quite confused why do you think lined Vanessa's response to something slightly different has much relevance here. 

One of the claims we make paraphrased & simplified models in a way which I hope is closer to your way of thinking about it:

- AIs are mostly not developed and deployed by individual humans
- there is a lot of other agencies or self-interested self-preserving structures/processes in the world
- if the AIs are aligned to the these structures, human disempowerment is likely because these structures are aligned to humans way less than they seem
- there are plausible futures in which these structures keep power longer than humans

Overall I would find it easier to discuss if you tried to formulate what you disagree about in the ontology of the paper. Also some of the points made are subtle enough that I don't expect responses to other arguments to address them.

 

Stock buybacks! Thank you. That is definitely going to be a big part f the "I am missing something here" I was expressing above.

Even if it actually turns out that "Super human AI will run on computers not much more expensive than personal computers" (which deepseek-r1 made marginally more plausible, but I'd say is still unlikely) it remains true that there will be very large returns to running 100 super human AIs instead of 1, or maybe 1 that's 100 times larger and smarter.

In other words, demand for hardware capable of running AIs will be very elastic. I don't see reductions in the costs of running AIs of a given level being bad for expected NVDA future cashflows. They don't mean we'll run the same "amount of AI" in less hardware, it will be closer to more AI in same amount of hardware.

I can think of a few reasons someone might think AI Control research should receive very high priority, apart from what is mentioned in the post or in Buck's comment: [LW(p) · GW(p)]

• You hope/expect early transformative AI to be used for provable safety approaches, using formal verification methods.
• You think AI control research is more tractable than other research agendas, or will have useful results faster, before they are too late to apply.
• Our only chance of aligning a superintelligence is to delegate the problem to AIs, either because it is too hard for humans, or it will arrive sooner than the proper alignment techniques can feasibly be developed.
• You expect a significant fraction of total AI safety research over all time to be done by early transformative AI, so control research has high leverage value in improving the probability of successfully getting the AI to do valuable safety research, even if slop is quite likely.

I agree with basically everything in the post but put enough probability on these points to think that control research has really high expected value anyway.

One quick observation about NVDA dividends that not many people might be aware of: NVDA pays a quarterly dividend of exactly once cent ($0.01) per share. They don't do this for the "usual" reason companies pay dividends (returning money to shareholders) but because by paying a non-zero dividend at all NVDA becomes part of dividend-paying company indexes and that means that ETFs that follow those indexes will buy NVDA shares. So they technically pay a dividend but for the purposes of valuation you should think of it as a non dividend paying stock.

Regarding the more general question of valuation, if you want to value a company based on how much they are currently distributing to shareholders you need to consider not only dividends but also share buybacks. Buybacks are effectively just a more tax-efficient form of paying dividends. I am not sure what the total numbers are for 2024, but in August for instance NVDA announced a $50 billion buyback.

And of course, the proper measure is not current distribution, but total expected discounted distributions over all time. That's hard to estimate, but for a company experiencing explosive growth it is surely higher than current distributions. 

It seems to me that by saying this the authors wanted to communicate "this is not a place to discuss this". But I agree that the phrasing used may inaccurately (?) communicate that the authors are more uncertain/agnostic about this issue than they really are (or that they believe something like "both sides have comparably good arguments"), so I'd suggest to replace it with something like:

The likelihood of loss of control is beyond the scope of this report (for discussion, see: [sources]).

I freely admit to not really understanding how shares are priced. To me it seems like the value of a share should be related to the expected dividend pay-out of that share over the remaining lifetime of the company, with a discount rate applied on pay-outs that are expected to happen further in the future (IE dividend yields 100 years from now are valued much less than equivalent payments this year). By this measure, justifying the current price sounds hard.

Google says that the annual dividend on Nvidia shares is 0.032%. (Yes, the leading digits are 0.0). So, right now, you get a much better rate of return just leaving your money in your bank's current account. So, at least by this measure, Nvidia shares are ludicrously over-priced. You could argue that future Nvidia pay outs might be much larger than the historical ones due to some big AI related profits. But, I don't find this argument convincing. Are future pay outs going to be 100x bigger? It would require a 100-fold yield increase for it to just be competitive with a savings account. If you time discount a little (say those 100-fold increases don't materialise for 3 years) then it looks even worse.

Now, clearly the world doesn't value shares according to the same heuristics that make sense to a non-expert like me. For example, the method "time integrate future expected dividend pay outs with some kind of time discounting" tells us that cryptocurrencies are worthless, because they are like shares with zero dividends. But, people clearly do put a nonzero value on bitcoin - and there is no plausible way that many people are that wrong. So they are grasping something that I am missing, and that same thing is probably what allows company shares to be prices so high relative to the dividends.