LessWrong 2.0 Reader

One lens to view AI is as a prediction engine -- predict what color to make each pixel, predict what word to put next.

Whoever is first to applying this predictive skill to stock markets will probably make immense amounts of money. Then again, people are probably already trying to do this, which creates a situation unlike that from which we derive the historic data to train on, which might render it impossible?

On the gripping hand, large slow and powerful institutions want to make the numbers go up and to the right.

Thanks for putting this together!

I have a vague memory of a post saying that taking zinc early, while virus was replicating in the upper respiratory tract, was much more important than taking it later, because later it would have spread all over the body and thus the zinc can’t get to it, or something like this. So I tend to take a couple early on then stop. But it sounds like you don’t consider that difference important. 

Is it your current (Not asking you to do more research!) impression that it’s useful to take zinc throughout the illness?

In this context, the “resample ablation” used in AI control is like adding more noise into the communication channel 

Ok, gotcha.

It's that she didn't accept the reasoning behind that number enough to really believe it. She added a discount factor based on fallacious reasoning around "if it were that easy, it'd be here already".

Just to clarify: There was no such discount factor that changed the median estimate of "human brain compute". Instead, this discount factor was applied to go from "human brain compute estimate" to "human-brain-compute-informed estimate of the compute-cost of training TAI with current algorithms" — adjusting for how our current algorithm seem to be worse than those used to run the human brain. (As you mention and agree with, although I infer that you expect algorithmic progress to be faster than Ajeya did at the time.) The most relevant section is here.

Comment from @James Chua [LW · GW] in internal slack group

interesting idea! thanks for writing this out. this is my initial reaction, i may have more later on. i agree on this -- this is what i'll expect.
 

we’d expect that a model using its own chain of thought to arrive at an answer is likely to perform better than when using a different chain of thought, even if that chain of thought is provided by a more capable model.

but if the above is true, I'm unsure about  evidence for steganography.
for example, gpt-4o is more capable than gpt-3.5. suppose we provide gpt-3.5 with gpt-4o's CoT.
one reason why gpt-3.5 may not improve is that gpt-4o's CoT skips certain steps since gpt-4o is a more capable model.

e.g. gpt-4o can calculate 4^4 directly, but gpt-3.5 cannot. So gpt-4o's CoT will leave out steps out 4 * 4 * 4* 4.  which gpt-3.5 needs to get the correct answerdo we then say that gpt-3.5 is doing steganography? (edited) 

possible improvement: provide gpt-3.5 with its own version of its own CoT, but paraphrased by another model. gpt-3.5 sees the same steps of "4 * 4 * 4 *4", but paraphrased. Does it still perform as well?

Curious, is the list of advisors public?

I think even the scaling thing doesn't apply here because they're not insuring bigger trips: they're insuring more trips (which makes things strictly better). I'm having some trouble understanding Dennis' point.

In addition to being hauntingly beautiful, this story helped me adjust to the idea of the trans/posthuman future.

14 years ago, I very much did not identify with the Transhuman Vision. It was too alien, too much, and I didn't feel ready for it. I also didn't actively oppose it. I knew that slowly, as I hung out around rationalists, I would probably slowly come to identify more with humanity's longterm future.

I have indeed come to identify more with the longterm future and all of it's weirdness. It was mostly not because of this story, but I did particularly resonate with the framing here – in large part because it met me where I am, instead of jumping into Future Shock. It presents the increasing alienness in gentle increments, from multiple perspectives, and from the perspective of someone currently living in a more Ancestral Human perspective.

This story doesn't tell you what sort of choices are good to make, but it makes it feel easier to wrap my brain around how I (or others) might eventually make such choices.

I don’t think that this is true.

i think this would not happen for the same fundamental reason that an aligned superintelligence can foresee whatever you can, and prevent / not cause them if it agrees they'd be worse than other possibilities. (more generally, "an aligned superintelligence would cause some bad-to-it thing" is contradictory, usually^[1].)

(i wonder if you're using the term 'superintelligence' in a different way though, e.g. to mean "merely super-human"? to be clear i definitionally mean it in the sense of optimal)

(tangentially: the 'nations' framing confuses me)^[2]

I think the main point is that what's worse than other possibilities partially depends on your value system at the start, and there is no non-circular way of resolving deep enough values conflicts such that you can always prevent conflict, so with differing enough values, you can generate conflict on it's own.

(Nitpick that doesn't matter, but when I focus on superintelligence, I don't focus on the AI literally doing optimal actions, because that leads to likely being wrong about what AIs can actually do.)

On the nations point, my point here is that people will program their superintelligences with quite different values, and the superintelligences will disagree about what counts as optimal from their lights, and if the disagreements are severe enough (which I predict is plausible if AI development cannot be controlled at all), conflict can definitely happen between the superintelligences, even if humans no longer are the main players.

Also, it's worth it to read these posts and comments, because I perceive some mistakes that are common amongst rationalists:
 

https://www.lesswrong.com/posts/895Qmhyud2PjDhte6/responses-to-apparent-rationalist-confusions-about-game [LW · GW]

https://www.lesswrong.com/posts/HFYivcm6WS4fuqtsc/dath-ilan-vs-sid-meier-s-alpha-centauri-pareto-improvements#jpCmhofRBXAW55jZv [LW(p) · GW(p)]

i think i wrote before that i agree (trivially) that not all possible values can be maximally satisfied; still, you can have the best possible world, which i think on this axis would look like "there being very many possible environments suited to different beings preferences (as long as those preferences are not to cause suffering to others)" instead of "beings with different preferences going to war with each other" (note there is no coordination problem which must be solved for that to happen. a benevolent superintelligence would itself not allow war (and on that, i'll also hedge that if there is some tragedy which would be worth the cost of war to stop, an aligned superintelligence would just stop it directly instead.))

 

I agree you can have a best possible world (though that gets very tricky in infinite realms due to utility theory breaking at that point), but my point here is that the best possible world is relative to a given value set, and also quite unconstrained, and your vision definitely requires other real-life value sets to lose out on a lot, here.

Are you assuming that superintelligences will have common enough values for some reason? To be clear, I think this can happen, assuming AI is controlled by a specific group that has enough of a monopoly on violence to prevent others from making their own AI, but I don't have nearly the confidence that you do that conflict is always avoidable by ASIs by default.