LessWrong 2.0 Reader

Admitting and apologizing for being 'only average' often functions as a submission move in dominance hierarchies, i.e. pecking orders.

This move derails attempts to enact more naïve, descriptive-language accountability. When someone has a specific grievance, it corresponds to a claim about the relation between facts and commitments that can be evaluated as true or false. Responding with self-deprecation transforms their concrete complaint into a mere opportunity to either accept or reject the display of submission. This disrupts the sort of language in which object-level accounting can happen, since the original specific issues are neither addressed nor refuted. Rather, they are displaced by the lower-dimensional social dynamics of dominance and submission.

So viewed systemically, such moves are part of a distributed strategy by which pecking orders disrupt and displace descriptive language communities by coordinating to invalidate them. And viewed locally, they erase the specific grievance from common knowledge, preserving the motivating shame.

In particular, even if the LLM were being continually trained (in a way that's similar to how LLMs are already trained, with similar architecture), it still wouldn't do the thing humans do with quickly picking up new analogies, quickly creating new concepts, and generally reforging concepts.

I agree this is a major unsolved problem that will be solved prior to AGI.

However, I still believe "AGI SOON", mostly because of what you describe as the "inputs argument".

In particular, there are a lot of things I personally would try if I was trying to solve this problem, but most of them are computationally expensive.  I have multiple projects blocked on "This would be cool, but LLMs need to be 100x-1Mx faster for it to be practical."  

This makes it hard for me to believe timelines like "20 or 50 years", unless you have some private reason to think Moore's Law/Algorithmic progress will stop.  LLM inference, for example, is dropping by 10x/year, and I have no reason to believe this stops anytime soon.

most people today do not appreciate that future computational-life forms will be just as meaningful as the meat-based ones today, and should not sacrifice orders of magnitudes more life-years than will be lived on Earth for the difference between a big ball of plasma and something else that recreates the same quality of light.

huh i agree with this, but i've been imagining that earth and possibly many galaxies will be in the domain of preferences which want to "live on actual planets" because of this chain-of-logic:

1. the lightcone is very much larger than just earth, or even than just earth and the surrounding million galaxies. (wikipedia says "[there are] an estimated 100 billion in all of the observable universe")
2. some beings care disproportionately about what happens to earth and its sun. some of them are alignment researchers, or otherwise influencing the trajectory
3. it's better for collaboration if those preferences determine the fate of earth and its sun. i.e. this prevents some values from being incentivized to compete for alignment to them in particular.
   • i can write more about this part if wanted.
     • actually, this case is less interesting than the general case i had in mind. because in this case one side has non-linearity in value of this sun versus others, both sides want to make this trade even without a greater chance of both dying if they don't.

I think Elizabeth is correct here, and also that vegan advocates would be considerably more effective with higher epistemic standards:

I think veganism comes with trade-offs, health is one of the axes, and that the health issues are often but not always solvable. This is orthogonal to the moral issue of animal suffering. If I’m right, animal EAs need to change their messaging around vegan diets, and start self-policing misinformation. If I’m wrong, I need to write some retractions and/or shut the hell up.

The post unfortunately suffers it's length, detailed explanations, and rebuttal of many motivated misreadings - many of which can be found in the comments, so it's unclear whether this helped. It's also well-researched and cited, well organized, offers cruxes and anticipates objections - vegan advocates are fortunate to have such high-quality criticism.

This could have been a shorter post, which was about rather than engaged in epistemics and advocacy around veganism, with less charitable assumptions. I'd have shared that shorter post more often, but I don't think it would be better.

I think we want to to conceal some facts about ourselves to ourselves in order to maintain a self-image because such self-images affect how we present ourselves to others and thus what we can be socially entitled to.

Some prior work on this topic:

• Robin Hanson:
  • A Theory of Identity
• Paul Christiano:
  • If we can’t lie to others, we will lie to ourselves
  • Epistemic incentives and sluggish updating

In the singer-songwriter example, part of their job is to project confidence that the show is going to be good, another part is to honestly evaluate the flaws in their work in order to improve it, and these can be at odds, creating cognitive dissonance.

which is a reasonable thing for someone to be annoyed about.

No, not really. The most fundamental problem is not the stupid claims about what Google will 'ever' be capable of (although that aspect is highly notable, in case you needed any evidence how Egan blew it on DL, that he wrote this ~2012 and kept doubling down on it), but rather his stubborn insistence on misunderstanding what a search engine does. That is why I am citing it: as an example of his mathematician-like crankery (ie. cognitive rigidity and fanaticism and perfectionism) - he has an idea about what a search engine does, and Google doesn't satisfy it, but he is unable to admit his idea is wrong.

The point of a search engine like Google is to be best-effort and show the best possible Internet results to the user for them to use as they will: it is not to 'show the one true correct answer'. A search engine which showed only 1 correct answer, or nothing, would not be a very useful search engine. (For example, in almost all of my search case-studies a search engine which operated in the way Egan expects Google to operate would be completely useless.)

I wouldn't call that a "complaint", exactly, but a hacky band-aid solution from someone who probably has better things to do with his time than tinker with DNS configuration.

He had that problem for something like a decade, and his solution was to insert spam in every page of his website for a self-inflicted problem that would've taken 5 minutes to fix in his registrar (which I know because I was also running a similar setup c. 2009 in terms of a domain name to content hosted on a shell account), and probably resulted in bugs or emails on a monthly basis the entire time (also based on my experience running a similar site).

Precisely because he had better things to do with his time, his not fixing it and instead complaining about everyone else was an embarrassment and an illustration of his cognitive rigidity & fanaticism. ("Am I wrong? Am I screwing up in a dumb simple fixable way which results in constant problems as people keep linking to the very URLs I insist on redirecting to them and then ordering them to fix my screwup by a nag notice and mysteriously some of them don't do so? Am I out of touch? ...No. The children are wrong.")

Also as an additional example: I have never once emailed, replied to, or tweeted at Egan that I can recall. He nevertheless preemptively blocked me on Twitter so long ago I don't know when.

This is a good take targeted at the wrong example, I think. (Though, I understand why it was the impetus for your take.) I’d argue one wants expressly identity based opinions (like the velentine one) to be filled with all the assumptions that make that opinion actually (or potentially) interesting. If valentine tried to hem and haw and be inclusive he would have provided a less interesting observation. It is then all the other articles that purport not to be expressly identity based that are the problem when they encapsulate all the identity assumptions and implicitly present them as the norm/standard/goes-without-saying-ness that are the problem. 
 

My initial reaction is that at least some of these points would be covered by the Guaranteed Safe AI agenda if that works out, right? Though the "AGIs act much like a colonizing civilization" situation does scare me because it's the kind of thing which locally looks harmless but collectively is highly dangerous. It would require no misalignment on the part of any individual AI. 

Relevant to whether the hot/cold/wet/dry system is a good or a bad idea, from our perspective, is that doctors don't currently use people's star signs for diagnosis. Bogus ontologies can be identified by how they promise to usefully replace more detailed levels of description - i.e., provide a useful abstraction that carves reality at the joints - and yet don't actually do this, from the standpoint of the cultures they're being sold to.

I think that's very probably true, yes. I'm not certain that will continue to be true indefinitely, or that it will or should continue to be the deciding factor for future decision making. I'm just pointing out that we're actually discussing a small subset of a very large space of options, that there are ways of "eating the sun" that allow life to continue unaltered on Earth, and so on. TBH even if we don't do anything like this, I wouldn't be terribly surprised if future humans end up someday building a full or partial shell around Earth anyway, long before there's anything like large-scale starlifting under discussion. Living space, power generation, asteroid deflection, off-world industry, just to name a few reasons we might do something like that. It could end up being easier and cheaper to increase available surface area and energy by orders of magnitude doing something like this than by colonizing other planets and moons.