LessWrong 2.0 Reader

most people today do not appreciate that future computational-life forms will be just as meaningful as the meat-based ones today, and should not sacrifice orders of magnitudes more life-years than will be lived on Earth^[1] for the difference between a big ball of plasma and something else that recreates the same quality of light.

huh i agree with this, but i've been imagining that earth and possibly many galaxies will be in the domain of other preferences because of this syllogism:

1. the lightcone is very much larger than just earth, or even than just earth and the surrounding million galaxies. (wikipedia says "[there are] an estimated 100 billion in all of the observable universe")
2. some people care disproportionately about what happens to earth and its sun. some of them are alignment researchers, or otherwise influencing the trajectory
3. it's better for collaboration if those preferences determine the fate of earth and its sun. i.e. this prevents some values from being incentivized to compete for alignment to them in particular.
   • i can write the expanded version of this part if wanted. notably it doesn't rely on trust.

I think Elizabeth is correct here, and also that vegan advocates would be considerably more effective with higher epistemic standards:

I think veganism comes with trade-offs, health is one of the axes, and that the health issues are often but not always solvable. This is orthogonal to the moral issue of animal suffering. If I’m right, animal EAs need to change their messaging around vegan diets, and start self-policing misinformation. If I’m wrong, I need to write some retractions and/or shut the hell up.

The post unfortunately suffers it's length, detailed explanations, and rebuttal of many motivated misreadings - many of which can be found in the comments, so it's unclear whether this helped. It's also well-researched and cited, well organized, offers cruxes and anticipates objections - vegan advocates are fortunate to have such high-quality criticism.

This could have been a shorter post, which was about rather than engaged in epistemics and advocacy around veganism, with less charitable assumptions. I'd have shared that shorter post more often, but I don't think it would be better.

I think we want to to conceal some facts about ourselves to ourselves in order to maintain a self-image because such self-images affect how we present ourselves to others and thus what we can be socially entitled to.

Some prior work on this topic:

• Robin Hanson:
  • A Theory of Identity
• Paul Christiano:
  • If we can’t lie to others, we will lie to ourselves
  • Epistemic incentives and sluggish updating

In the singer-songwriter example, part of their job is to project confidence that the show is going to be good, another part is to honestly evaluate the flaws in their work in order to improve it, and these can be at odds, creating cognitive dissonance.

which is a reasonable thing for someone to be annoyed about.

No, not really. The most fundamental problem is not the stupid claims about what Google will 'ever' be capable of (although that aspect is highly notable, in case you needed any evidence how Egan blew it on DL, that he wrote this ~2012 and kept doubling down on it), but rather his stubborn insistence on misunderstanding what a search engine does. That is why I am citing it: as an example of his mathematician-like crankery (ie. cognitive rigidity and fanaticism and perfectionism) - he has an idea about what a search engine does, and Google doesn't satisfy it, but he is unable to admit his idea is wrong.

The point of a search engine like Google is to be best-effort and show the best possible Internet results to the user for them to use as they will: it is not to 'show the one true correct answer'. A search engine which showed only 1 correct answer, or nothing, would not be a very useful search engine. (For example, in almost all of my search case-studies a search engine which operated in the way Egan expects Google to operate would be completely useless.)

I wouldn't call that a "complaint", exactly, but a hacky band-aid solution from someone who probably has better things to do with his time than tinker with DNS configuration.

He had that problem for something like a decade, and his solution was to insert spam in every page of his website for a self-inflicted problem that would've taken 5 minutes to fix in his registrar (which I know because I was also running a similar setup c. 2009 in terms of a domain name to content hosted on a shell account), and probably resulted in bugs or emails on a monthly basis the entire time (also based on my experience running a similar site).

Precisely because he had better things to do with his time, his not fixing it and instead complaining about everyone else was an embarrassment and an illustration of his cognitive rigidity & fanaticism. ("Am I wrong? Am I screwing up in a dumb simple fixable way which results in constant problems as people keep linking to the very URLs I insist on redirecting to them and then ordering them to fix my screwup by a nag notice and mysteriously some of them don't do so? Am I out of touch? ...No. The children are wrong.")

Also as an additional example: I have never once emailed, replied to, or tweeted at Egan that I can recall. He nevertheless preemptively blocked me on Twitter so long ago I don't know when.

This is a good take targeted at the wrong example, I think. (Though, I understand why it was the impetus for your take.) I’d argue one wants expressly identity based opinions (like the velentine one) to be filled with all the assumptions that make that opinion actually (or potentially) interesting. If valentine tried to hem and haw and be inclusive he would have provided a less interesting observation. It is then all the other articles that purport not to be expressly identity based that are the problem when they encapsulate all the identity assumptions and implicitly present them as the norm/standard/goes-without-saying-ness that are the problem. 
 

My initial reaction is that at least some of these points would be covered by the Guaranteed Safe AI agenda if that works out, right? Though the "AGIs act much like a colonizing civilization" situation does scare me because it's the kind of thing which locally looks harmless but collectively is highly dangerous. It would require no misalignment on the part of any individual AI. 

Relevant to whether the hot/cold/wet/dry system is a good or a bad idea, from our perspective, is that doctors don't currently use people's star signs for diagnosis. Bogus ontologies can be identified by how they promise to usefully replace more detailed levels of description - i.e., provide a useful abstraction that carves reality at the joints - and yet don't actually do this, from the standpoint of the cultures they're being sold to.

I think that's very probably true, yes. I'm not certain that will continue to be true indefinitely, or that it will or should continue to be the deciding factor for future decision making. I'm just pointing out that we're actually discussing a small subset of a very large space of options, that there are ways of "eating the sun" that allow life to continue unaltered on Earth, and so on. TBH even if we don't do anything like this, I wouldn't be terribly surprised if future humans end up someday building a full or partial shell around Earth anyway, long before there's anything like large-scale starlifting under discussion. Living space, power generation, asteroid deflection, off-world industry, just to name a few reasons we might do something like that. It could end up being easier and cheaper to increase available surface area and energy by orders of magnitude doing something like this than by colonizing other planets and moons.

I like the point about the need for some type of external competitive measure but as you say, they might not be a MMA gym where you need one.

Shifting the metaphor, I think your observation of the sucker punch fits well with the insight that for those with only a hammer, all problems look like nails. The gym would be someone with a screwdriver or riveter as well as the hammer. But even lacking the external check, we should always ask ourselves "Is this really a nail?" I might only have a hammer but if this isn't a nail while the results might be better than nothing (maybe?) I'm not likely to achieve the best that could be done some other way. And,of course, watching just what happens went the other hammer-only people solve the problem and compare results to those when we know the problem is a nail we might learn something from their mistakes.

Yeah, I'm aware of that model. I personally generally expect the "science on model organisms"-style path to contribute basically zero value to aligning advanced AI, because (a) the "model organisms" in question are terrible models, in the sense that findings on them will predictably not generalize to even moderately different/stronger systems (like e.g. this story [LW(p) · GW(p)]), and (b) in practice IIUC that sort of work is almost exclusively focused on the prototypical failure story of strategic deception and scheming, which is a very narrow slice [LW(p) · GW(p)] of the AI extinction probability mass.