LessWrong 2.0 Reader

The question is {who receives the profits from running the 100 super human AI?}. Let's say the 100 super human AI are involved in manufacturing planes or mining. AI hardware is only a small component of the system. I mean metal is important for all sorts of things. But metal producers are only reaping a fraction of that. 

there are lots of smaller, newer companies with varied business models, mostly "too early to tell" for sure if they have the potential to get huge, but I expect in principle many of them should be viable.

will fix

Let's do most of this via the much higher bandwidth medium of voice, but quickly:

1. Yes, qualia^[1] is real, and is a class of mathematical structure.^[2]
2. (placeholder for not a question item)
   1. Matter is a class of math which is ~kinda like our physics.
   2. Our part of the multiverse probably doesn't have special "exists" tags, probably everything is real (though to get remotely sane answers you need a decreasing reality fluid/caring fluid allocation).

   3. Math, in the sense I'm trying to point to it, is 'Structure'. By which I mean: Well defined seeds/axioms/starting points and precisely specified rules/laws/inference steps for extending those seeds. The quickest way I've seen to get the intuition for what I'm trying to point at with 'structure' is to watch these videos in succession (but it doesn't work for everyone):

       

1. ^{^}

   experience/the thing LWers tend to mean, not the original philosophical sense which is pointlessly high complexity stuff about epiphenomena

2. ^{^}

   possibly maybe even the entire class, though if true most qualia would be very very alien to us and not necessarily morally valuable

Here's a podcast with @Slimepriestess [LW · GW] on the whole affair. I'm not particularly involved in any of this, and had some violent disagreements with things said in the podcast^[1], but found it informative on the whole.

I'm quite confused why do you think lined Vanessa's response to something slightly different has much relevance here. 

One of the claims we make paraphrased & simplified models in a way which I hope is closer to your way of thinking about it:

- AIs are mostly not developed and deployed by individual humans
- there is a lot of other agencies or self-interested self-preserving structures/processes in the world
- if the AIs are aligned to the these structures, human disempowerment is likely because these structures are aligned to humans way less than they seem
- there are plausible futures in which these structures keep power longer than humans

Overall I would find it easier to discuss if you tried to formulate what you disagree about in the ontology of the paper. Also some of the points made are subtle enough that I don't expect responses to other arguments to address them.

 

Stock buybacks! Thank you. That is definitely going to be a big part f the "I am missing something here" I was expressing above.

Even if it actually turns out that "Super human AI will run on computers not much more expensive than personal computers" (which deepseek-r1 made marginally more plausible, but I'd say is still unlikely) it remains true that there will be very large returns to running 100 super human AIs instead of 1, or maybe 1 that's 100 times larger and smarter.

In other words, demand for hardware capable of running AIs will be very elastic. I don't see reductions in the costs of running AIs of a given level being bad for expected NVDA future cashflows. They don't mean we'll run the same "amount of AI" in less hardware, it will be closer to more AI in same amount of hardware.

I can think of a few reasons someone might think AI Control research should receive very high priority, apart from what is mentioned in the post or in Buck's comment: [LW(p) · GW(p)]

• You hope/expect early transformative AI to be used for provable safety approaches, using formal verification methods.
• You think AI control research is more tractable than other research agendas, or will have useful results faster, before they are too late to apply.
• Our only chance of aligning a superintelligence is to delegate the problem to AIs, either because it is too hard for humans, or it will arrive sooner than the proper alignment techniques can feasibly be developed.
• You expect a significant fraction of total AI safety research over all time to be done by early transformative AI, so control research has high leverage value in improving the probability of successfully getting the AI to do valuable safety research, even if slop is quite likely.

I agree with basically everything in the post but put enough probability on these points to think that control research has really high expected value anyway.

One quick observation about NVDA dividends that not many people might be aware of: NVDA pays a quarterly dividend of exactly once cent ($0.01) per share. They don't do this for the "usual" reason companies pay dividends (returning money to shareholders) but because by paying a non-zero dividend at all NVDA becomes part of dividend-paying company indexes and that means that ETFs that follow those indexes will buy NVDA shares. So they technically pay a dividend but for the purposes of valuation you should think of it as a non dividend paying stock.

Regarding the more general question of valuation, if you want to value a company based on how much they are currently distributing to shareholders you need to consider not only dividends but also share buybacks. Buybacks are effectively just a more tax-efficient form of paying dividends. I am not sure what the total numbers are for 2024, but in August for instance NVDA announced a $50 billion buyback.

And of course, the proper measure is not current distribution, but total expected discounted distributions over all time. That's hard to estimate, but for a company experiencing explosive growth it is surely higher than current distributions.