LessWrong 2.0 Reader

I can think of a few reasons someone might think AI Control research should receive very high priority, apart from what is mentioned in the post or in Buck's comment: [LW(p) · GW(p)]

• You hope/expect early transformative AI to be used for provable safety approaches, using formal verification methods.
• You think AI control research is more tractable than other research agendas, or will have useful results faster, before they are too late to apply.
• Our only chance of aligning a superintelligence is to delegate the problem to AIs, either because it is too hard for humans, or it will arrive sooner than the proper alignment techniques can feasibly be developed.
• You expect a significant fraction of total AI safety research over all time to be done by early transformative AI, so control research has high leverage value in improving the probability of successfully getting the AI to do valuable safety research, even if slop is quite likely.

I agree with basically everything in the post but put enough probability on these points to think that control research has really high expected value anyway.

One quick observation about NVDA dividends that not many people might be aware of: NVDA pays a quarterly dividend of exactly once cent ($0.01) per share. They don't do this for the "usual" reason companies pay dividends (returning money to shareholders) but because by paying a non-zero dividend at all NVDA becomes part of dividend-paying company indexes and that means that ETFs that follow those indexes will buy NVDA shares. So they technically pay a dividend but for the purposes of valuation you should think of it as a non dividend paying stock.

Regarding the more general question of valuation, if you want to value a company based on how much they are currently distributing to shareholders you need to consider not only dividends but also share buybacks. Buybacks are effectively just a more tax-efficient form of paying dividends. I am not sure what the total numbers are for 2024, but in August for instance NVDA announced a $50 billion buyback.

And of course, the proper measure is not current distribution, but total expected discounted distributions over all time. That's hard to estimate, but for a company experiencing explosive growth it is surely higher than current distributions. 

It seems to me that by saying this the authors wanted to communicate "this is not a place to discuss this". But I agree that the phrasing used may inaccurately (?) communicate that the authors are more uncertain/agnostic about this issue than they really are (or that they believe something like "both sides have comparably good arguments"), so I'd suggest to replace it with something like:

The likelihood of loss of control is beyond the scope of this report (for discussion, see: [sources]).

I freely admit to not really understanding how shares are priced. To me it seems like the value of a share should be related to the expected dividend pay-out of that share over the remaining lifetime of the company, with a discount rate applied on pay-outs that are expected to happen further in the future (IE dividend yields 100 years from now are valued much less than equivalent payments this year). By this measure, justifying the current price sounds hard.

Google says that the annual dividend on Nvidia shares is 0.032%. (Yes, the leading digits are 0.0). So, right now, you get a much better rate of return just leaving your money in your bank's current account. So, at least by this measure, Nvidia shares are ludicrously over-priced. You could argue that future Nvidia pay outs might be much larger than the historical ones due to some big AI related profits. But, I don't find this argument convincing. Are future pay outs going to be 100x bigger? It would require a 100-fold yield increase for it to just be competitive with a savings account. If you time discount a little (say those 100-fold increases don't materialise for 3 years) then it looks even worse.

Now, clearly the world doesn't value shares according to the same heuristics that make sense to a non-expert like me. For example, the method "time integrate future expected dividend pay outs with some kind of time discounting" tells us that cryptocurrencies are worthless, because they are like shares with zero dividends. But, people clearly do put a nonzero value on bitcoin - and there is no plausible way that many people are that wrong. So they are grasping something that I am missing, and that same thing is probably what allows company shares to be prices so high relative to the dividends.

Does Deepseek actually mean that Nvidia is over valued?

FYI I wrote this a few days ago but could not post it due to being rate limited. The one change I made is adding actually to the 1st line.

To be clear I have no intention whatsoever of shorting NVDA

Epistemic status -- I have no idea what I'm talking about

Epistemic status -- very speculative but not quite a DMT hallucination

Super human AI will run on computers not much more expensive than personal computers but perhaps with highly specialized chips maybe even specialized for the task of running a single AI instance

Investment in AI proper will be small relative to AI directed production

There will be a period of increasing marginal returns from AI; but this will eventually become diminishing marginal returns

Even during the period of increasing marginal returns more $$$ will go to AI directed production than AI proper

Companies that most successfully transition to AI will blow the competition away; some of these companies will have a moat & continue to make high profits. But how can such high profits be justified? Maybe the government needs to take 50% of the shares & create trust funds for its citizens.

Companies that buy up the right kinds of land & natural resources will also do well

Companies that are least affected by AI will benefit b/c of the Baumol effect

So what are the get rick quick schemes? Specialized chips, incorporating AI into production systems for non AI goods, strategically buying up the right land & land rights, AI resistant industries!?

Interesting times maybe too interesting

In conclusion I'm agnostic as to whether Nvidia is or is not over valued but other companies may benefit even more as AI advances. I think it's more about leadership & seizing opportunities more so than a few companies having a overwhelmingly dominant position.

Thanks for the reply!

The only general remarks that I want to make

are in regards to your question about

the model of 150 year long vaccine testing

on/over some sort of sample group and control group.

I notice that there is nothing exponential assumed

about this test object, and so therefore, at most,

the effects are probably multiplicative, if not linear.

Therefore, there are lots of questions about power dynamics

that we can overall safely ignore, as a simplification,

which is in marked contrast to anything involving ASI.

If we assume, as you requested, "no side effects" observed,

in any test group, for any of those things

that we happened to be thinking of, to even look for,

then for any linear system, that is probably "good enough".

I am not sure I understand the distinction between linear and exponential in the vaccine context. By linear do you mean that only few people die? By exponential do you mean that a lot of people die?

If so, then I am not so sure that vaccine effects could only be linear. For example, there might be some change in our complex environment that would prompt the vaccine to act differently than it did in the past.

More generally, our vaccine can lead to catastrophic outcomes if there is something about its future behavior that we didn't predict. And if that turns out to be true, then things could go ugly really fast.

And the extent of the damage can be truly big. "Scientifically proven" cancer vaccine that passed the tests is like the holy grail of medicine. "Curing cancer" is often used by parents as an example of the great things their children could achieve. This is combined with the fact that cancer has been with us for a long time and the fact that the current treatment is very expensive and painful.

All of these factors combined tell us that in a relatively short period of time a large percentage of the total population will get this vaccine. At that point, the amount of damage that can be done only depends on what thing we overlooked, which we, by definition, have no control over.

If there is some long future problem that crops up,

the company can say "we never looked for that"

and "we are not responsible for the unexpected",

because the people who made the deployment choices

have taken their profits and their pleasure in life,

and are now long dead. "Not my Job".

"Don't blame us for the sins of our forefathers".

Similarly, no one is going to ever admit or concede

any point, of any argument, on pain of ego death.

This same excuse would surely be used by companies manufacturing the vaccine. They would argue that they shouldn't be blamed for something that the researchers overlooked. They would say that they merely manufactured the product in order to prevent the needless suffering of countless people.

For all we know, by the time that the overlooked thing happens, the original researchers (who developed and tested the vaccine) are long dead, having lived a life of praise and glory for their ingenious invention (not to mention all the money that they received).

Nice post! There can be some surprising language-barriers between early modern writers and today's readers. I remember as an undergrad getting very confused by a passage from Locke in which he often used the word 'sensible.' I took him to mean 'prudent' and only later discovered he meant 'can be sensed'!

does that require you to either have the ability to commit to a plan or the inclination to consistently pick your plan from some prior epistemic perspective

You aren't required to take an action (/start acting on a plan) that is worse from your current perspective than some alternative. Let maximality-dominated mean "w.r.t. each distribution in my representor, worse in expectation than some alternative." (As opposed to "dominated" in the sense of "worse than an alternative with certainty".) Then, in general you would need^[1] to ask, "Among the actions/plans that are not maximality-dominated from my current perspective, which of these are dominated from my prior perspective?" And rule those out.

1. ^{^}

   If you care about diachronic norms of rationality, that is.

What? If every couple had only one child, the population would halve at each generation. That's what they mean. Replacement rate requires more than just one child.

I mean, the whole point was "how can we have fertility but also not be a dystopia". You just described a dystopia. It's also kind of telling that the only way to make people have children, something that is supposedly a joyous experience, you can think of is "have a tyrannical dictator make it very clear that they'll make sure the alternative is even worse". That's pretty telling. Someone thinking this way is part of the problem more than they are of the solution.