LessWrong 2.0 Reader

Upvoted for the song.

What do you mean by "hunger for life"?

What do you mean by "capitalism"?

If you basically mean that machines should help us overcome scarcity, and then everyone should be able to focus on games, friendship, learning, et cetera... sure, why not?

But first we need to make sure the machines won't kill us all when they get smarter than us and start controlling the wold. (Because if they do, it doesn't matter how our corporations and governments were set up.)

However, capitalism, as it stands, obstructs the realization of this vision.

So far, the attempts to replace capitalism often did even worse.

One problem - scarcity. Usually made worse by eliminating capitalism.

Second problem - humans. Psychopaths compete for power, in both capitalism and socialism. We need to solve this. Democracy alone is not a solution; psychopaths are quite successful at getting elected, or getting their people elected.

Is it time to rethink the way corporations and governing systems operate?

I believe people are thinking about this all the time, but do you have a specific proposal that wasn't widely considered yet?

I'm hopeful about it, but preparing the lectures alone will be a lot of work (although the first one will be a repeat of some material presented at ILIAD).

Consider Egan's incentives. "A group of effective altruists collects a ton of money, buys anti-malaria nets, saves million African lives (but other millions still die of malaria)" is an improvement over status quo in real life, but it would be a boring and disappointing story.

Cool fictional villains are at least an improvement over the media narrative "EAs are crypto scammers".

I wonder if there are people who joined the rationalist or effective altruist communities because of recent Egan's stories. A negative advertisement is still advertisement... I can imagine someone reading the story, then trying to find more on the internet, then joining; the question is whether this actually happens.

I'm wondering less if humans will want to date AGIs and more if AGIs will want to date humans.

Sure, if we solve the alignment problem we can build AGIs that want to date humans; but will we decide that's ethical?

The criteria for consciousness and moral worth are varied and debated. The answer to whether AGIs will be conscious and worthy is definitely sort of.

So: is creating a conscious being with a core motivation designed specifically so that it wants to date you a form of slavery? It definitely smacks of grooming or something....

One issue is whether AGIs will want to stay around the human cognitive level. There's an issue with power dynamics in a relationship between a nerd and a demigod.

Sure the humans can cognitively enhance too; what fraction of us will want to become demigods ourselves?

It's going to be wild if we can get there. And fun. Speaking of which, we won't be playing games mostly for status -- we'll mostly be playing for fun.

We won't all have the coolest friends, but we'll all have cool friends because we'll all be cool friends. Humans will no longer be repressed, neurotic messes because we'll have actual understanding of psychology and actual good, safe, supportive childhoods for essentially everyone.

It's gonna be wild if we can get there.

links 1/15/25: https://roamresearch.com/#/app/srcpublic/page/01-15-2025

• https://www.proteinatlas.org/  seems like a good resource. Swedish.
• https://en.m.wikipedia.org/wiki/Human_cloning  human cloning was first discussed by JBS Haldane in a 1969 speech!
• https://en.wikipedia.org/wiki/Protalix_BioTherapeutics they seem pretty successful. enzyme replacement for Gaucher disease. Israeli.
  • https://en.wikipedia.org/wiki/Phillip_Frost interesting guy. "served as a lieutenant commander, U.S. Public Health Service at the National Cancer Institute, from 1963 to 1965." Major pharma investor.
• What happened to Amyris?
  • they used to be a biofuel company but couldn't get production up and costs down:
    • https://web.archive.org/web/20241122084330/https://www.technologyreview.com/2018/05/10/2851/the-scientist-still-fighting-for-the-clean-fuel-the-world-forgot/
    • https://www.technologyreview.com/2012/02/10/20483/amyris-gives-up-making-biofuels-update/
    • https://www.fastcompany.com/1680328/the-rise-and-fall-of-the-company-that-was-going-to-have-us-all-using-biofuels
  • they pivoted to low-volume, high-price beauty & personal care ingredients, which actually generated a bunch of revenue, but not enough to cover costs. and then also bought a ton of celebrity beauty brands, which didn't. 2022 stock plunge, 2023 bankruptcy.
    • https://www.businessoffashion.com/articles/beauty/amyris-filed-for-bankruptcy/
    • https://www.fool.com/investing/2022/11/09/why-amyris-stock-was-driven-into-the-ground-on-wed/
    • https://www.retaildive.com/news/amyris-files-chapter-11-bankruptcy-lays-off-260-workers/690636/
    • https://www.voguebusiness.com/beauty/why-amyris-chapter-11-bankruptcy-is-the-latest-beauty-casualty
    •  
  • they're not terrible at industrial fermentation (compared to other synbio unicorns) and have some lessons learned
    • https://pmc.ncbi.nlm.nih.gov/articles/PMC7695652/
  • they got in trouble with the SEC for recognizing more revenue than they actually made (according to standard accounting)
    • https://www.sec.gov/enforcement-litigation/administrative-proceedings/34-93341-s
  • https://www.science.org/content/article/synthetic-biology-once-hailed-moneymaker-meets-tough-times bad times for biomanufacturing/synbio overall
    • there are kind of...zero large profitable firms founded after 2000 that specialize in industrial fermentation/biomanufacturing, EXCEPT a couple of biotechs that make enzyme drugs.
      • there's plenty of biomanufactured products but pretty much all from very large old boring firms at sorta commodity prices?

Interesting, the prices seemed reasonable overall though I traded the later dates down a little bit because if LLMs haven't won be 2030 the paradigm is probably limited (IMO they hadn't priced in that update). 

I suppose that it's a slightly "unfair" comparison because chess engines are very narrow and humans can't beat them either. How do LLMs compare to top human chess players?

I think that getting too fixated on status games is usually due to some kind of insecurity, e.g. feeling that you need to accumulate status in order to be accepted, respected or something like that. (One can certainly play status games for the fun of it without having such an insecurity, but that seems unlikely to lead to the level of fixation where status game would become the primary activity that humans engage in.) 

If every human can close friends or even lovers with AI systems as you suggest, then I would expect that to provide the kind of deep unconditional feeling of security that makes the need for playing status games fall away. If you feel deeply safe, loved, respected, etc., then status can certainly still feel nice to have, but it's unlikely to feel that important for most people. In the same way that e.g. a loving parent focused on taking care of and spending time with their family may find themselves becoming much less interested in spending their time playing status games.

in some sense that’s just hiring you for any other job, and of course if an AGI lab wants you, you end up with greater negotiating leverage at your old place, and could get a raise (depending on how tight capital constraints are, which, to be clear, in AI alignment are tight).

Actually we don’t even need to train a new byte latent transformer. We can just generate patches using GPT-2 small. 

1. Do the patches correspond to atomic concepts?
2. If we train on the patches generated as such, do we  get a better LM