LessWrong 2.0 Reader

At the 2024 LessWrong Community weekend I met somebody who I have been working with for perhaps 50 hours so far. They are better at certain programming related tasks than me, in a way provided utility. Before meeting them they where not even considering working on AI alignment related things. The conversation wen't something like this:

Johannes: What are you working on. Other Person: Web development. What are you working on? Johannes: I am trying to understand intelligence such that we can build a system that is capable enough to prevent other misaligned AI's from being build, and that we understand enough such that we can be sure that it wouldn't kill us. [...] Why are you not working on it? Other Person: (forgot what) Johannes: Oh then now is the perfect time to start working on it. Other Person: So what are you actually doing. Johannes: (Describes some methodologies.) Other Person: (Questions whether these methodologies are actually good, and thinks about how they could be better.) [...]

Actually this all happened after the event when traveling from the venue to the train station.

It doesn't happen that often that I get something really good out of a random meeting. Most of them are bad. However, I think the most important thing I do to get something out is to just immediately talk about the things that I am interested in. This efficiently filters out people, either because they are not interested, or because they can't talk it.

You can overdo this. Starting a conversation with "AI seems very powerful, I think it will likely destroy the world" can make other people feel awkward (I know from experience). However, the above formula of "what do you do" and then "and I do this" get's to the point very quickly without inducing awkwardness.

Oh, I think that also means that section is slightly wrong. You want to take insurance if

and the insurance company wants to offer it if

So define

as you did above. Appendix B suggests that you'd take insurance if $V\left(W_{you}\right)>0$ and they'd offer it if $V\left(W_{them}\right)<0$. But in fact they'd offer it if $V(W_{them}+P)<0$.

I find it suspicious that a lot of the criticisms I read online of Indian-Americans (nepotism, obsequiousness, "dual loyalty", scheming) are very similar to the criticisms I hear of Jews.

It doesn't need to be a singular high-value conversation. I'd say the long-term value of conversations is heavy tailed and so it may pay to have lots of conversations of low expected value. https://www.sciencedirect.com/science/article/abs/pii/B9780124424500500250

One example of what I am talking about is the middle chapters of the book Genesis, where it discusses applications of AI in military and general governance.

I don't necessarily agree with the book's predictions, but it really got me thinking of a near term pre-AGI world.

https://www.axios.com/2024/11/19/henry-kissinger-ai-book-released

Someone recommended I create the community, maybe I will in the new year.

I feel your points are very intelligent. I also agree that specializing AI is a worthwhile direction.

It's very uncertain if it works, but all approaches are very uncertain, so humanity's best chance is to work on many uncertain approaches.

Unfortunately, I disagree it will happen automatically. Gemini 1.5 (and probably Gemini 2.0 and GPT-4) are Mixture of Experts models. I'm no expert, but I think that means that for each token of text, a "weighting function" decides which of the sub-models should output the next token of text, or how much weight to give each sub-model.

So maybe there is an AI psychiatrist, an AI mathematician, and an AI biologist inside Gemini and o1. Which one is doing the talking depends on what question is asked, or which part of the question the overall model is answering.

The problem is they they all output words to the same stream of consciousness, and refer to past sentences with the words "I said this," rather than "the biologist said this." They think that they are one agent, and so they behave like one agent.

My idea [LW · GW]—which I only thought of thanks to your paper—is to do the opposite. The experts within the Mixture of Experts model, or even the same AI on different days, do not refer to themselves with "I" but "he," so they behave like many agents.

:) thank you for your work!

I'm not disagreeing with your work, I'm just a little less optimistic than you and don't think things will go well unless effort is made. You wrote the 100 page paper so you probably understand effort more than me :)

Happy holidays!

The thing with NVIDIA though is that the IV is so high and so are premiums. I spent a few hours looking for a better trade than that, though I think it's pretty solid.

I think SPY calls can possibly be much better than NVIDIA calls. The market doesn't expect the stock market to go up significantly in the next few years, but I think theres a chance it will assuming timelines are short. Here's the SPY YoY growth during the internet boom in the 90s.

Year 2000 saw a -9.7% return ($86.54) 1999: +20.4% ($95.88) 1998: +28.7% ($79.65) 1997: +33.5% ($61.89) 1996: +22.5% ($46.37) 1995: +38.0% ($37.85) 1994: +0.4% ($27.42)

Here we see that from any two year period from 1995-1999, the stock market went up anywhere from 50% to 70%.

Thus, I don't think it's unreasonable to think SPY has a good chance of going up 50% - 70% by Jan 15 2027 (to be fair, the past two years had a YoY growth of ~25%)

If you buy a $855 Strike price call for that date and SPY increases 50% by then you get a 12x return. If SPY increases 70% you get a 62x return.

If you buy the highest Strike Price call for that day at $910 and SPY increase by 70%, you get an 83x return.

Something to think about at least. At this time I’m going to buy long dated SPY calls for 2-3 years out at the 800$-910$ range. Nvidia calls still look good but the premiums are just so expensive because of the companies recent massive growth and volatility, so I think SPY calls are the better option.I'm still thinking about how to hedge incase the upcoming chaos turns the market sour (perhaps a Taiwanese blockade, or NVIDIA profits being hurt by increasing government interference)

This is looking like a February 2020 moment.

Eric Drexler wrote two essays that seem related, which I really loved.

The first is How to Understand Everything (and why). It's short enough to be quoted essentially whole, so if you don't mind I'll do so:

In science and technology, there is a broad and integrative kind of knowledge that can be learned, but isn’t taught. It’s important, though, because it makes creative work more productive and makes costly blunders less likely.

Formal education in science and engineering centers on teaching facts and problem-solving skills in a series of narrow topics. It is true that a few topics, although narrow in content, have such broad application that they are themselves integrative: These include (at a bare minimum) substantial chunks of mathematics and the basics of classical mechanics and electromagnetism, with the basics of thermodynamics and quantum mechanics close behind.

Most subjects in science and engineering, however, are narrower than these, and advanced education means deeper and narrower education. What this kind of education omits is knowledge of extent and structure of human knowledge on a trans-disciplinary scale. This means understanding — in a particular, limited sense — everything.

To avoid blunders and absurdities, to recognize cross-disciplinary opportunities, and to make sense of new ideas, requires knowledge of at least the outlines of every field that might be relevant to the topics of interest. By knowing the outlines of a field, I mean knowing the answers, to some reasonable approximation, to questions like these:

What are the physical phenomena?
What causes them?
What are their magnitudes?
When might they be important?
How well are they understood?
How well can they be modeled?
What do they make possible?
What do they forbid?

And even more fundamental than these are questions of knowledge about knowledge:

What is known today?
What are the gaps in what I know?
When would I need to know more to solve a problem?
How could I find what I need?

It takes far less knowledge to recognize a problem than to solve it, yet in key respects, that bit of knowledge is more important: With recognition, a problem may be avoided, or solved, or an idea abandoned. Without recognition, a hidden problem may invalidate the labor of an hour, or a lifetime. Lack of a little knowledge can be a dangerous thing.

Looking back over the last few decades, I can see that I’ve invested considerably more than 10,000 hours in learning about the structures, relationships, contents, controversies, open problems, limitations, capabilities, developing an understanding of how the fields covered in the major journals fit together to constitute the current state of science and technology. In some areas, of course, I’ve dug deeper into the contents and tools of a field, driven by the needs of problem solving; in others, I know only the shape of the box and where it sits.

This sort of knowledge is a kind of specialty, really — a limited slice of learning, but oriented crosswise. Because of this orientation, though, it provides leverage in integrating knowledge from diverse sources. I am surprised by the range of fields in which I can converse with scientists and engineers at about the level of a colleague in an adjacent field. I often know what to ask about their research, and sometimes make suggestions that light their eyes.

The follow-up essay is How to Learn About Everything. It's again short enough to quote wholesale:

Note that the title above isn’t “how to learn everything”, but “how to learn about everything”. The distinction I have in mind is between knowing the inside of a topic in deep detail — many facts and problem-solving skills — and knowing the structure and context of a topic: essential facts, what problems can be solved by the skilled, and how the topic fits with others.

This knowledge isn’t superficial in a survey-course sense: It is about both deep structure and practical applications. Knowing about, in this sense, is crucial to understanding a new problem and what must be learned in more depth in order to solve it. The cross-disciplinary reach of nanotechnology almost demands this as a condition of competence. 

Studying to learn about everything

To intellectually ambitious students I recommend investing a lot of time in a mode of study that may feel wrong. An implicit lesson of classroom education is that successful study leads to good test scores, but this pattern of study is radically different. It cultivates understanding of a kind that won’t help pass tests — the classroom kind, that is.

1. Read and skim journals and textbooks that (at the moment) you only half understand. Include Science and Nature.
2. Don’t halt, dig a hole, and study a particular subject as if you had to pass a test on it.
3. Don’t avoid a subject because it seems beyond you — instead, read other half-understandable journals and textbooks to absorb more vocabulary, perspective, and context, then circle back.
4. Notice that concepts make more sense when you revisit a topic.
5. Notice which topics link in all directions, and provide keys to many others. Consider taking a class.
6. Continue until almost everything you encounter in Science and Nature makes sense as a contribution to a field you know something about.

Why is this effective?

You learned your native language by immersion, not by swallowing and regurgitating spoonfuls of grammar and vocabulary. With comprehension of words and the unstructured curriculum of life came what we call “common sense”.

The aim of what I’ve described is to learn an expanded language and to develop what amounts to common sense, but about an uncommonly broad slice of the world. Immersion and gradual comprehension work, and I don’t know of any other way.

This process led me to explore the potential of molecular nanotechnology as a basis for high-throughput atomically precise manufacturing. If broad-spectrum common sense were more widespread among scientists, there would be no air of controversy around the subject, milestones like the U.S. National Academies report on molecular manufacturing would have been reached a decade earlier, and today’s research agenda and perception of global problems would be very different.

I think I prefer either of Drexler's approach, Sarah Constantin's / Scott's fact-posting [LW · GW], and Holden Karnofsky's learning by writing, all of which can start with endless breadth but also require (quoting Drexler) deep structure and practical applications as focusing mechanisms, to the sort of learning that I think might be incentivised by budding panologists having to maximise their minimum score across some standardised battery of tests. I also liked Sarah's suggestion at the end:

Ideally, a group of people writing fact posts on related topics, could learn from each other, and share how they think. I have the strong intuition that this is valuable. It's a bit more active than a "journal club", and quite a bit more casual than "research". It's just the activity of learning and showing one's work in public.

One problem I've seen around consilience is that some people say "well, clearly you don't think that there is one thing" when what they actually have evidence for is "you don't agree with my analogies across distant fields". So "believing in the unity of science" can get conflated with "agreeing with a particular set of analogies."

Andy: "Evolutionary biology proves that capitalism is the correct economic system, because competition produces fitness among firms as it does among organisms."
Betty: "I'm not sure you can directly draw inferences from biology to economics like that, without dealing with a whole bunch of disanalogies between the objects of those fields."
Andy: "What, do you deny that biology and economics are both studying the same world?"

Some casual conversations with strangers that were high instrumental value:

At my first (online) LessWrong Community Weekend in 2020, I happened to chat with Linda Linsefors. That was my first conversation with anyone working in AI Safety. I’d read about the alignment problem for almost a decade at that point and thought it was the most important thing in the world, but I’d never seriously considered working on it. MIRI had made it pretty clear that the field only needed really exceptional theorists, and I didn’t think I was one of those. That conversation with Linda started the process of robbing me of my comfortable delusions on this front. What she said made it seem more like the field was pretty inadequate, and perfectly normal theoretical physicists could maybe help just by applying the standard science playbook for figuring out general laws in a new domain. Horrifying. I didn't really believe it yet, but this conversations was a factor in me trying out AI Safety Camp a bit over a year later.

At my first EAG, I talked to someone who was waiting for the actual event to begin along with me. This turned out to be Vivek Hebbar, who I'd never heard of before. We got to talking about inductive biases of neural networks. We kept chatting about this research area sporadically for a few weeks after the event. Eventually, Vivek called me to talk about the idea that would become this post [LW · GW]. Thinking about that idea led to me understanding the connection between basin broadness and representation dimensionality in neural networks, which ultimately resulted in this [LW · GW] research. It was probably the most valuable conversation I’ve had at any EAG so far, and it was unplanned.

At my second EAG, someone told me that an idea for comparing NN representations I’d been talking to them about already existed, and was called centred kernel alignment. I don’t quite remember how that conversation started, but I think it might have been a speed friending event.

My first morning in the MATS kitchen area in Berkeley, someone [LW · GW] asked me if I’d heard about a thing called Singular Learning Theory. I had not. He went through his spiel on the whiteboard. He didn’t have the explanation down nearly as well back then, but it still very recognisably connected to how I’d been thinking about NN generalisation and basin broadness, so I kept an eye on the area.