LessWrong 2.0 Reader

What are Transformers? Like what is concrete but accurate-enough conversational way of describing it that doesn't force me to stop the conversation dead in it's tracks to explain jargon like "Convolutional Neural Network" or "Multi-Head Attention"?

Its weird that I can tell you roughly how the Transformers in a Text Encoder-Decoder like T5 is different from the Autoregressive Transformers that generate the text in ChatGPT (T5 is parallel, ChatGPT sequential), or how I can even talk about ViT and DiT transformers in image synthesis (ViT like Stable Diffusion down and upsample the image performing operations on the entire latent, DiT work on patches). But I don't actually have a clear definition for what is a transformer.

And if I was in a conversation with someone who doesn't know much about compsci (i.e. Me - especially 5 months ago), how would I explain it:
 

"well for text models it is a mechanism that after a stream of words has been tokenized (i.e. blue might be "bl" and "ue" which each have a special id number) and the embeddings retrieved based on those token id numbers which are then used to compute the Query, Key, and Value vectors which often use a similarity measure like a Cosine Similarity Measure to compare the embedding of this key vector to the Qu - HEY, WHERE ARE YOU GOING! I DIDN'T EVEN GET TO THE NORMALIZATION!

Obviously this isn't a definition, this is a "how it works" explanation and what I've just written as an example is heavily biased towards the decoder. But if someone asks me "what is a transformer?" what is a simple way of saying it in conversation?

I tweeted about something a lot like this

https://x.com/robertskmiles/status/1877486270143934881

I don't know if this is true, but I've heard that when the Nazis were having soldiers round up and shoot people (before the Holocaust became more industrial and systematic), as part of the preparation for having them execute civilians, the Nazi officers explicitly offered their soldiers the chance to excuse themselves (on an individual basis) from having to actually perform the executions themselves with no further consequences.

Signal feels overloaded to me and kodo seems like a much more narrow concept.

"I've got pretty good cell signal." "The elite spend a lot of time virtue signalling." "The castaways used smoke signals to get rescued." "Careful configuration is required to prevent signal interference between network nodes." "I think there's a pretty good signal in the polling data." "Ted's not actually being mean to me, he's just counter-signalling."

I'd basically agree that kodo is a subcategory of signal, in the same way that chicken is a subcategory of bird. The narrower concept is useful. Kodo is closer to signal if you just think of it in the sense used by The Signal and the Noise, though there's at least a bit of distinction in my head. There are weaker and stronger signals, which make you more or less confident in (say) which candidate might win based on the poll. As contrast, a poll that was good kodo might tell you either the conservative party is going to win in a landslide or your poll sucks. "Polls say the conservative party is 60% to win" is a signal, but it isn't kodo.

Noise and din seem more like synonyms, though both are kind of defined by being opposites. Noise is everything not signal, din is everything not kodo? I kinda wish din didn't sound like an English word but figured it made more sense to write it up as it was in the lecture.

Another potential crux[^related] is that Ege's world view seemingly doesn't depend at all on AIs which are much faster and smarter than any human. As far as I can tell, it doesn't enter into his modeling of takeoff (or timelines to full automation of remote work which partially depends on something more like takeoff).

On my views this makes a huge difference because a large number of domains would go much faster with much more (serial and smarter) intelligence. My sense is that a civilization where the smartest human was today's median human and also everyone's brain operated 50x slower^[1] would in fact make technological progress much slower. Similarly, if AIs were as much smarter than the smartest humans as the smartest human is smarter than the median human and also ran 50x faster than humans (and operated at greater scale than the smartest humans with hundreds of thousands of copies all at 50x speed for over 10 million parallel worker equivalents putting aside the advantages of serial work and intelligence), then we'd see lots of sectors go much faster.

My sense is that Ege bullet bites on this and thinks that slowing everyone down wouldn't make a big difference, but I find this surprising. Or maybe his views are that parallelism is nearly as good as speed and intelligence and sectors naturally scale up parallel worker equivalents to match up with other inputs, so we're bottlenecking on some other inputs in the important cases.

Yes, I think when it comes to endogenous vs exogenous preferences, you are probably accurate about most of the field believing in exogenous preferences (although I'm not sure since I'm also unfamiliar). Rohin Shah once talked about ambiguous value learning [LW · GW], but my guess is that most people aren't focusing on that direction.

It's possible that people who believe in exogenous preferences will feel confused if they are described as following Mistake Theory, since exogenous preferences sounds like Conflict Theory, and doesn't sound like "pursuing an objective morality."

My personal opinion (and my opinion isn't that important here), is that humans ourselves follow a combination of endogenous and exogenous preferences. Our morals are very strongly shaped by what everyone around us believe, often more than we realize.

But at the same time, our hardcoded biology determines how our morals get shaped. If we hunt and kill animals for food or sport, but observe the animals we kill following various norms in their animal society, we will not adopt their norms by mere exposure, and be indifferent to their norms. This is because our hardcoded biology did not encode any tendency to care about those animals or to respect their norms.

I agree that studying endogenous preferences, and how to make them go right, is valuable!

Ironically, arguably the most important/useful point of the essay is arguing for a rebranded version of the "precisely timed short slow/pause/pivot resources to safety" proposal. Dario's rebranded it as spending down a "security buffer".

(I don't have a strong view on whether this is a good rebrand, seems reasonable to me I guess and the terminology seems roughly as good for communicating about this type of action.)

Population is highly relevant to war between neighbors, particularly conscript war. But is it relevant to colonization? Over the past half century the relative population of Congo to Belgium increased by a factor of 5. A factor of 4 makes a difference in a single battle, but on the scale of a colonization project where a handful of people take over a large territory, it is nothing. At most it would require scaling down ambition by a factor of 5. Wikipedia claims that the colonial population increased by 2 orders of magnitude from 1900 to 1960, while native population did not change. The factor of 100 with no obvious cause is the big story and the factor of 5 historically irrelevant. Belgium did not lose control because the colony became too large to control, for the population grew only after independence. Perhaps it gave up because the colonial population was 1% of the metropolitan population, a real expense, but that was the result of how it governed the colony, not a necessary consequence of relative populations.

The past really was different. The ability of a thousand Belgians to control the whole of the Congo is shocking. Social technologies change much faster than populations.

Nod, this feels a bit at the intersection of what I had in mind with "Cyborgism", and the "Schleppy work in narrow domains" section.

Some thoughts: for this sort of thing, there's a hypothesis ("making it easier to change representations will enable useful thinking in hmath", and a bunch of annoying implementation details you need to test the hypothesis (i.e. actually getting an LLM to do all that work reliably).

So my next question here is "can we test out a version of this sort of thing powered by some humans-in-a-trenchcoat", or otherwise somehow test the ultimate hypothesis without having to build the thing." I'm curious for your intuitions on that

Did anyone predict that we’d see major AI companies not infrequently releasing blatantly misaligned AIs (like Sidney, Claude 3.7, o3)?

Just four days later, X blew up with talk of how GPT-4o has become sickeningly sycophantic in recent days, followed by an admission from Sam Altman that something went wrong (with lots of hilarious examples in replies):

the last couple of GPT-4o updates have made the personality too sycophant-y and annoying (even though there are some very good parts of it), and we are working on fixes asap, some today and some this week.

at some point will share our learnings from this, it's been interesting.