LessWrong 2.0 Reader

Yeah it's for the bounty. Hanson suggested that a list of links might be preferred to a printed book, at least for now, since he might want to edit the posts.

No, I just threw that in. But there is the VHEM, and apparently serious people who argue for anti-natalism.

Short of those, there are also advocates for "degrowth".

I suspect the reason that Zvi declined to engage with such arguments is that he thinks they're too batshit insane to be worth giving house room, but these are a few terms to search for.

I mostly expect you start getting more and more into sub-critical intelligence explosion dynamics when you exceed +6std more and more. (E.g. see second half of this other comment i wrote [LW(p) · GW(p)]) I also expect very smart people will be able to better setup computer-augmented note organizing systems or maybe code narrow aligned AIs that might help them with their tasks (in a way it's a lot more useful than current LLMs but hard to use for other people). But idk.

I'm not sure how big the difference between +6 and +6.3std actually is. I also might've confused the actual-competence vs genetical-potential scale. On the scale I used the drive/"how hard one is trying" also plays a big role.

I actually mostly expect this from seeing that intelligence is pretty heavitailed. E.g. alignment research capability seems incredibly heavitailed to me, though it might be hard to judge the differences in capability there if you're not already one of the relatively few people who are good at alignment research. Another example is how Einstein managed to find general relativity where the combined rest of the world wouldn't have been able to do it like that without more experimental evidence.
I do not know why this is the case. It is (very?) surprising to me. Einstein didn't even work on understanding and optimizing his mind. But yeah that's how I guess.

I expect (1) and (2) to go mostly fine (though see footnote^[6]), and for (3) to completely fail. For example, suppose $C_b$ has a bunch of features for things like “true according to smart humans.” How will we distinguish those from features for “true”? I think there’s a good chance that this approach will reduce the problem of discriminating $C_g$ vs. $C_b$ to an equally-difficult problem of discriminating desired vs. undesired features.

However, if we found that the classifier was using a feature for "How smart is the human asking the question?" to decide what answer to give (as opposed to how to then phrase it), that would be a red flag.

Why there isn't more ground-up interest: it's expensive and people can't easily tell if it's worth the cost. Also anything where UV is touching you has to overcome people's safety concerns.

Good question on the large effects are easily measured thing--has to do with the distinction between: 1) in what environments you are cleaning the air, 2) how much you are cleaning the air there, 3) how much pathogen people inhale, and 4) how much pathogen is required to actually make people sick. It's not just a far-UV problem, it'd be a problem for any air cleaner, it's just that far-UV is especially expensive to install and especially faces negative "UV" associations.

Far-UV has a large effect on airborne pathogen concentration, and that large effect is in fact easy to measure in a chamber! But once you add it to a room where people are moving around and talking to each other, how much pathogen are they actually inhaling? Is the air in the room well-mixed? Is the far-UV reaching the infectious air before people inhale it? Even if they inhale air that has living pathogens, are they getting sick? If they get sick, did they get sick from that room or from a different environment that they were in previously? Study endpoints matter a lot. 

Being able to understand intervention efficacy especially becomes a problem if a disease is largely spread via superspreaders/had high variance in infection sources. COVID, at least early in the pandemic, had very high variance, whereas eg seasonal flu doesn't usually. Therefore, if your study intervention is installed in public spaces, it's possible for it not to show much effect on seasonal flu but a large effect on a disease with early-COVID-like dynamics, which means you have to wait for that COVID-like disease to come along to see the effect--but that would be worth it; high-variance diseases are very concerning! 

Another way of saying all this is that it's not the case that the effect will be super hard to measure given enough people and time, it's that the effect is hard to measure given that you need a lot of people in your study to account for the distinctions listed above, and/or you need a highly controlled environment, and that's just expensive.

We know well enough what people mean by "world" - the stuff they care about. The fact that physics keeps on happening if humanity is snuffed out is no comfort at all to me or to most humans.

Arguing epistemology is not going to prevent a nuclear apocalypse or us being wiped out by the new intelligent species we are inventing. The fact that you don't know what's happening on the other side of the world has no bearing on existential dangers facing those people. That's what I mean by saving the world, and I expect what the author meant. This is a different thing than just helping people by your own values and estimates.

I very much agree that pithy mysterious statements for others to argue over is not a good use of the quick takes here.

Is the complaint that you can't do predicate calculus on the probabilities? Because I can certainly use predicate calculus all I want on the expressions within the probabilities.

And if that is the complaint, then my question is: why do we want to do predicate calculus on the probabilities? Like, what would be one concrete application in which we'd want to do that? (Self-reference and things in that cluster would be the obvious use-case, I'm mostly curious if there's any other use-case.)

I have no interest in honor if it's celebrated on a field of the dead. Virtue ethics is fine, as long as it's not an excuse to not figure out what needs doing and how it's going to get done.

Doing ones own part and trusting that the other parts are done by anonymous unknown others is a very silly coordination strategy. We need plans that amount to success, not just everyone doing whatever sounds nice to them.

Edit: I very much agree that saving the world is a team sport. Perhaps it's relevant that successful teams always do some planning and coordinating.

the thing a logician would call a "logic" or possibly a "logic augmented with some probabilities"

The main point of the article is that once you add probabilities you can't do predicate calculus anymore. It's a mathematical operation that's not defined for the entities that you get when you do your augmentation.

Interesting. There's certainly a lot going on in there, and some of it very likely is at least vague models of future word occurrences (and corresponding events). The definition of model-based gets pretty murky outside of classic RL, so it's probably best to just directly discuss what model properties give rise to what behavior, e.g. optimizing for reward.

Model-free systems can produce goal-directed behavior. The do this if they have seen some relevant behavior that achieves a given goal, and their input or some internal representation includes the current goal, and they can generalize well enough to apply what they've experienced to the current context. (This is by the neuroscience definition of habitual vs goal-directed: behavior changes to follow the current goal, usually hungry, thirsty or not).

So if they're strong enough generalizers, I think even a model-free system actually optimizes for reward.

I think the claim should be stronger: for a smart enough RL system, reward is the optimization target.