LessWrong 2.0 Reader

I was scrolling for a while, assuming I'd neared the end, only to look at the position of the scrollbar and find I was barely 5% through. This must have taken a fair bit of effort! I think it's a helpful page and I'm glad I know about it, I encourage you to make a linkpost for it sometime if you haven't already.

Doesn't the "threat" to delete the model have to be DT-credible instead of "credible conditioned on being human-made", given that LW with all its discussion about threat resistance and ignoring is in training sets?

(If I remember correctly, a decision theory must ignore "you're threatened to not do X, and the other agent is claiming to respond in such a way that even they lose in expectation" and "another agent [self-]modifies/instantiates an agent making them prefer that you don't do X".)

One lens to view AI is as a prediction engine -- predict what color to make each pixel, predict what word to put next.

Whoever is first to applying this predictive skill to stock markets will probably make immense amounts of money. Then again, people are probably already trying to do this, which creates a situation unlike that from which we derive the historic data to train on, which might render it impossible?

On the gripping hand, large slow and powerful institutions want to make the numbers go up and to the right.

Thanks for putting this together!

I have a vague memory of a post saying that taking zinc early, while virus was replicating in the upper respiratory tract, was much more important than taking it later, because later it would have spread all over the body and thus the zinc can’t get to it, or something like this. So I tend to take a couple early on then stop. But it sounds like you don’t consider that difference important. 

Is it your current (Not asking you to do more research!) impression that it’s useful to take zinc throughout the illness?

In this context, the “resample ablation” used in AI control is like adding more noise into the communication channel 

Ok, gotcha.

It's that she didn't accept the reasoning behind that number enough to really believe it. She added a discount factor based on fallacious reasoning around "if it were that easy, it'd be here already".

Just to clarify: There was no such discount factor that changed the median estimate of "human brain compute". Instead, this discount factor was applied to go from "human brain compute estimate" to "human-brain-compute-informed estimate of the compute-cost of training TAI with current algorithms" — adjusting for how our current algorithm seem to be worse than those used to run the human brain. (As you mention and agree with, although I infer that you expect algorithmic progress to be faster than Ajeya did at the time.) The most relevant section is here.

Comment from @James Chua [LW · GW] in internal slack group

interesting idea! thanks for writing this out. this is my initial reaction, i may have more later on. i agree on this -- this is what i'll expect.
 

we’d expect that a model using its own chain of thought to arrive at an answer is likely to perform better than when using a different chain of thought, even if that chain of thought is provided by a more capable model.

but if the above is true, I'm unsure about  evidence for steganography.
for example, gpt-4o is more capable than gpt-3.5. suppose we provide gpt-3.5 with gpt-4o's CoT.
one reason why gpt-3.5 may not improve is that gpt-4o's CoT skips certain steps since gpt-4o is a more capable model.

e.g. gpt-4o can calculate 4^4 directly, but gpt-3.5 cannot. So gpt-4o's CoT will leave out steps out 4 * 4 * 4* 4.  which gpt-3.5 needs to get the correct answerdo we then say that gpt-3.5 is doing steganography? (edited) 

possible improvement: provide gpt-3.5 with its own version of its own CoT, but paraphrased by another model. gpt-3.5 sees the same steps of "4 * 4 * 4 *4", but paraphrased. Does it still perform as well?

Curious, is the list of advisors public?

I think even the scaling thing doesn't apply here because they're not insuring bigger trips: they're insuring more trips (which makes things strictly better). I'm having some trouble understanding Dennis' point.

In addition to being hauntingly beautiful, this story helped me adjust to the idea of the trans/posthuman future.

14 years ago, I very much did not identify with the Transhuman Vision. It was too alien, too much, and I didn't feel ready for it. I also didn't actively oppose it. I knew that slowly, as I hung out around rationalists, I would probably slowly come to identify more with humanity's longterm future.

I have indeed come to identify more with the longterm future and all of it's weirdness. It was mostly not because of this story, but I did particularly resonate with the framing here – in large part because it met me where I am, instead of jumping into Future Shock. It presents the increasing alienness in gentle increments, from multiple perspectives, and from the perspective of someone currently living in a more Ancestral Human perspective.

This story doesn't tell you what sort of choices are good to make, but it makes it feel easier to wrap my brain around how I (or others) might eventually make such choices.