LessWrong 2.0 Reader

what do you think about Robin Hanson's culture and value drift?

https://www.overcomingbias.com/p/how-fix-cultural-drift

Admitting and apologizing for being 'only average' often functions as a submission move in dominance hierarchies, i.e. pecking orders.

This move derails attempts to enact more naïve, descriptive-language accountability. When someone has a specific grievance, it corresponds to a claim about the relation between facts and commitments that can be evaluated as true or false. Responding with self-deprecation transforms their concrete complaint into a mere opportunity to either accept or reject the display of submission. This disrupts the sort of language in which object-level accounting can happen, since the original specific issues are neither addressed nor refuted. Rather, they are displaced by the lower-dimensional social dynamics of dominance and submission.

So viewed systemically, such moves are part of a distributed strategy by which pecking orders disrupt and displace descriptive language communities by coordinating to invalidate them. And viewed locally, they erase the specific grievance from common knowledge, preserving the motivating shame.

In particular, even if the LLM were being continually trained (in a way that's similar to how LLMs are already trained, with similar architecture), it still wouldn't do the thing humans do with quickly picking up new analogies, quickly creating new concepts, and generally reforging concepts.

I agree this is a major unsolved problem that will be solved prior to AGI.

However, I still believe "AGI SOON", mostly because of what you describe as the "inputs argument".

In particular, there are a lot of things I personally would try if I was trying to solve this problem, but most of them are computationally expensive.  I have multiple projects blocked on "This would be cool, but LLMs need to be 100x-1Mx faster for it to be practical."  

This makes it hard for me to believe timelines like "20 or 50 years", unless you have some private reason to think Moore's Law/Algorithmic progress will stop.  LLM inference, for example, is dropping by 10x/year, and I have no reason to believe this stops anytime soon.

most people today do not appreciate that future computational-life forms will be just as meaningful as the meat-based ones today, and should not sacrifice orders of magnitudes more life-years than will be lived on Earth for the difference between a big ball of plasma and something else that recreates the same quality of light.

huh i agree with this, but i've been imagining that earth and many galaxies will be in the domain of preferences which want to "live on actual planets" because of this chain-of-logic:

1. the lightcone is very much larger than just earth. (wikipedia says "[there are] an estimated 100 billion in all of the observable universe"). we'd want to give up earth (and some surrounding many galaxies) in return for more good possible futures, because of the good which can be derived from the non-earth parts of those ones.
2. some beings care disproportionately about what happens to earth and its sun. some of them are alignment researchers, or otherwise influencing the trajectory
3. it's better for collaboration if those preferences determine the fate of earth and its sun. i.e. this prevents some values from being incentivized to compete for alignment to them in particular (to the detriment of general success rates).
   • i can write more about this part if wanted. notably (unless i've made a mistake) it doesn't rely on trust, but that's probably not very clear with just this.
     • actually, this case is less interesting than the general case i had in mind. because in this case one side has non-linearity in value of this sun versus others, both sides want to make this trade even without a greater chance of both dying if they don't.

I think Elizabeth is correct here, and also that vegan advocates would be considerably more effective with higher epistemic standards:

I think veganism comes with trade-offs, health is one of the axes, and that the health issues are often but not always solvable. This is orthogonal to the moral issue of animal suffering. If I’m right, animal EAs need to change their messaging around vegan diets, and start self-policing misinformation. If I’m wrong, I need to write some retractions and/or shut the hell up.

The post unfortunately suffers it's length, detailed explanations, and rebuttal of many motivated misreadings - many of which can be found in the comments, so it's unclear whether this helped. It's also well-researched and cited, well organized, offers cruxes and anticipates objections - vegan advocates are fortunate to have such high-quality criticism.

This could have been a shorter post, which was about rather than engaged in epistemics and advocacy around veganism, with less charitable assumptions. I'd have shared that shorter post more often, but I don't think it would be better.

We conceal some facts about ourselves from ourselves to maintain a self-image because such self-images affect how we present ourselves to others and thus what we can be socially entitled to. This is similar to what psychologist Carol Dweck called a "fixed mindset," in contrast with a "growth mindset" where the self-image more explicitly includes the possibility of change.

In the singer-songwriter example, creating a good vibe with the audience generally involves projecting confidence. This confidence can connect to an identity as a competent performer, which maintains entitlement to the audience's approval as well as other perks like booking future shows and charging higher rates. We might think of the performer as implicitly reasoning, "I must have audience approval in order to maintain my identity. I get audience approval by being a good performer. Therefore I must be a good performer. Good performers don't make basic mistakes. Therefore I must have performed flawlessly. Staying out late would cause flaws in my performance. Therefore I must not have stayed out late."

Meanwhile, improving as a performer requires honestly evaluating weaknesses in one's performance - noticing timing issues, pitch problems, or moments where energy flagged. This evaluation process works best with immediate, specific feedback while memories are fresh. Or, in the specific example you gave, the performer's process of improvement needs to include the specific factual memory that they stayed out late, which likely impaired their performance.

When the good vibe with the audience is based on a rigidly maintained self-image, this creates an internal conflict: The same performance needs to be confidently good for maintaining entitlement and specifically flawed to enable improvement. This conflict creates pressure toward shame - the performer must maintain a persona that cannot acknowledge certain facts, while those facts remain active in guiding behavior.

Some prior work on this topic:

• Robin Hanson:
  • A Theory of Identity
• Paul Christiano:
  • If we can’t lie to others, we will lie to ourselves
  • Epistemic incentives and sluggish updating

which is a reasonable thing for someone to be annoyed about.

No, not really. The most fundamental problem is not the stupid claims about what Google will 'ever' be capable of (although that aspect is highly notable, in case you needed any evidence how Egan blew it on DL, that he wrote this ~2012 and kept doubling down on it), but rather his stubborn insistence on misunderstanding what a search engine does. That is why I am citing it: as an example of his mathematician-like crankery (ie. cognitive rigidity and fanaticism and perfectionism) - he has an idea about what a search engine does, and Google doesn't satisfy it, but he is unable to admit his idea is wrong.

The point of a search engine like Google is to be best-effort and show the best possible Internet results to the user for them to use as they will: it is not to 'show the one true correct answer'. A search engine which showed only 1 correct answer, or nothing, would not be a very useful search engine. (For example, in almost all of my search case-studies a search engine which operated in the way Egan expects Google to operate would be completely useless.)

I wouldn't call that a "complaint", exactly, but a hacky band-aid solution from someone who probably has better things to do with his time than tinker with DNS configuration.

He had that problem for something like a decade, and his solution was to insert spam in every page of his website for a self-inflicted problem that would've taken 5 minutes to fix in his registrar (which I know because I was also running a similar setup c. 2009 in terms of a domain name to content hosted on a shell account), and probably resulted in bugs or emails on a monthly basis the entire time (also based on my experience running a similar site).

Precisely because he had better things to do with his time, his not fixing it and instead complaining about everyone else was an embarrassment and an illustration of his cognitive rigidity & fanaticism. ("Am I wrong? Am I screwing up in a dumb simple fixable way which results in constant problems as people keep linking to the very URLs I insist on redirecting to them and then ordering them to fix my screwup by a nag notice and mysteriously some of them don't do so? Am I out of touch? ...No. The children are wrong.")

Also as an additional example: I have never once emailed, replied to, or tweeted at Egan that I can recall. He nevertheless preemptively blocked me on Twitter so long ago I don't know when.

This is a good take targeted at the wrong example, I think. (Though, I understand why it was the impetus for your take.) I’d argue one wants expressly identity based opinions (like the velentine one) to be filled with all the assumptions that make that opinion actually (or potentially) interesting. If valentine tried to hem and haw and be inclusive he would have provided a less interesting observation. It is then all the other articles that purport not to be expressly identity based that are the problem when they encapsulate all the identity assumptions and implicitly present them as the norm/standard/goes-without-saying-ness that are the problem. 
 

My initial reaction is that at least some of these points would be covered by the Guaranteed Safe AI agenda if that works out, right? Though the "AGIs act much like a colonizing civilization" situation does scare me because it's the kind of thing which locally looks harmless but collectively is highly dangerous. It would require no misalignment on the part of any individual AI. 

Relevant to whether the hot/cold/wet/dry system is a good or a bad idea, from our perspective, is that doctors don't currently use people's star signs for diagnosis. Bogus ontologies can be identified by how they promise to usefully replace more detailed levels of description - i.e., provide a useful abstraction that carves reality at the joints - and yet don't actually do this, from the standpoint of the cultures they're being sold to.