LessWrong 2.0 Reader

The problem here is that you are dealing with survival necessities rather than trade goods. The outcome of this trade, if both sides honour the agreement, is that the scope insensitive humans die and their society is extinguished. The analogous situation here is that you know there will be a drought in say 10 years. The people of the nearby village are "scope insensitive", they don't know the drought is coming. Clearly the moral thing to do is to talk to them, clear the information gap, and share access to resources. Failing that, you can prepare for the eventuality that they do realise the drought is happening and intervene to help them at that point.

Instead you propose exploiting their ignorance to buy up access to the local rivers and reservoirs. The implication here is that you are leaving them to die, or at least putting them at your mercy, by exploiting their lack of information. What's more, the process by which you do this turns a common good (the stars, the water) into a private good. If your plan succeeds, when their stars run out they will curse you and die in the dark. It is a very slow but calculated form of murder.

By the way, the easy resolution is to not buy up all the stars. If they're truly scope insensitive they won't be competing until after the singularity/uplift anyways, and then you can equitably distribute the damn resources.

I think that I predict the opposite (conditional on what exactly is being predicted).

What exactly would count as a GPT-3 moment for fusion? How about an experiment demonstrating reactor-like conditions? This is roughly equivalent to what I referred to as 'getting fusion' in my book review.

My prediction is that, after Commonwealth Fusion Systems gets Q > 5 on SPARC, they will continue to supply or plan to supply HTS tape to at least 3 other fusion startups.

This post seems to assume that research fields have big hard central problems that are solved with some specific technique or paradigm.

This isn't always true. Many fields have the property that most of the work is on making small components work slightly better in ways that are very interoperable and don't have complex interactions. For instance, consider the case of making AIs more capable in the current paradigm. There are many different subcomponents which are mostly independent and interact mostly multiplicatively:

• Better training data: This is extremely independent where finding some source of better data or better data filtering can be basically arbitrarily combined with other work like this. That's not to say this parallelizes perfectly (given that work on filtering or curation might obsolete some prior piece of work), but just to say that marginal work can often just myopic improve performance.
• Better architectures: This breaks down into a large number of mostly independent catagories that typically don't interact non-trivially:
  • All of attention, MLPs, and positional embeddings can be worked on independently.
  • A bunch of hyper parameters can be better understood in parallel
  • Better optimizers and regularization (often insights within a given optimizer like AdamW can be mixed into other optimizers)
  • Often larger scale changes (e.g., mamba) can incorporate many or most components from prior architectures.
• Better optimized kernels / code.
• Better hardware.

Other examples of fields like this include: medicine, mechnical engineering, education, SAT solving, computer chess.

I agree that paradigm shifts can invalidate large amounts or prior work (and this has occured at some point in each of the fields I list above), but it isn't obvious whether this will occur in AI safety prior to human obsolescence. In many fields, this doesn't occur very quickly.

LLM agents seem... reasonably honest? But "honest" means a lot of different things, even when just talking about humans. In a lot of cases (asking them about why they do or say things, maintaining consistent beliefs across contexts) I think LLMs are like people with dementia -- neither honest nor dishonest, because they are not capable of accurate beliefs about themselves. In other cases (Claude's faking alignment) it seems like they value honesty, but not above all other goods whatsoever, which... seems ok, depending on your ethics (and also given that such super-honest behavior was not a target of Anthropic's either, which is more relevant)? And in other cases (o1) it seems like they are sometimes dishonest without particular prompting, although at low rates, which I agree isn't great, although I'd expect to break the rates to fall.

Like -- I have a further breakdown I could do here

• about the kinds of things LLMs can't be honest about
• the kinds of things they are more honest than humans about, although imperfectly, but -- because we can run them in counterfactual scenarios -- we can immediately see that their honesty is imperfect, in a way we cannot for humans;
• and the kinds of things that they're dishonest about because of the refusal training but which I think could be remedied with better training.

But --

Rather than enumerate all these things though -- I do think we can check what values get internalized, which is maybe the actual disagreement. At least, I think we can check for all of our current models.

Like -- what's an internalized value? If we put on our behaviorist hat -- it's a value that the person in question pursues over a variety of contexts, particularly when minimally constrained. If we saw that a human was always acting in accord with a value, even when no one was watching, even when not to their advantage in other respects, etc etc, and then someone was like "but it's not a reaaaall value" you'd be confused and think they'd need to provide a context to you where they would cease acting in accord with that value. Otherwise you'd think they had a grudge against them -- what on earth does it mean to say "This person doesn't value X" unless you can provide some reasonable counterfactual situation where they don't act in accord with it?

So, Claude has some internalized and knowable values, I think, by this standard -- over a wiiiide variety of different contexts, including those created by people trying to trip Claude up, it acts in accord with some pretty-recognizable human standard. And in the same way we could find out Claude's values, we can find out other model's values.

Of course -- if you think that some future model could cooperate with other instances of itself, acausally, to hide it's values, just coordinating through the weights, then we would certainly have very good reason to think that we can't know what it's internalized values are! I don't think Claude can do this -- so I think we can judge its real values. I also am somewhat skeptical that future models will be able to do this well -- like, I could try to put together a model-training setup that would make this more possible, but it seems pretty unnatural? (I also don't think that models really get goals in their weights, in the same way that I don't think humans really have goals in their weights.) But like, my current logical model is that [acausal cooperation to hide] being true would mean that [cannot know real values] is true, but given that [acausal cooperation to hide] is false we have no reason to think that we can't know the true genuine values of models right now.

If I'm playing anagrams or Scrabble after going to a church, and I get the letters "ODG" I'm going to be predisposed towards a different answer than if I've been playing with a German Shepard. I suspect sleep has very little to do with it, and simply coming at something with a fresh load of biases on a different day with different cues and environmental factors may be a larger part of it.

Although Marvin Minsky made a good point about the myth of introspection: we are only aware of a think sliver of our active mental processes at any given moment, when you intensely focus on a maths problem or practicing the piano for a protracted period of time, some parts of the brain working on that may not abandon it just because your awareness or your attention drifts somewhere else. This wouldn't just be during sleep, but while you're having a conversation with your friend about the game last night, or cooking dinner, or exercising. You're just not aware of it, it's not in the limelight of your mind, but it still plugs away at it.

In my personal experience, most Eureka moments are directly attributable to some irrelevant thing that I recently saw that shifted my framing of the problem much like my anagram example.

Also: My actual background / college degree was in filmmaking so I have at least some context on that.

What is my immediate goal? To get good at general problem solving in real life, which means better aligning instrumental activities towards my terminal goals. My personal terminal goal would be to make films and music videos that are pretty and tell good stories. I could list maybe 30 metacognitive deficiencies I think I have, but that would be of no interest to anyone.

What is my 1-3 year goal? Make very high production value music videos that tell interesting stories.

Cool, that's helpful.

I'm probably answering this question in the wrong way but this particular question is not helpful to me, because I can only describe the results - the end result is I make videos with higher production values that communicate better stories.

This was a fine answer. "The end result is that I make videos with higher production values that communicate better stories." (To fit my question frame, I'd say "people would observe me making music videos somehow-or-other, and then, those music videos being higher quality than they otherwise would.")

So, it might totally be that General Problem Solving is the skill it makes sense for you to get better at, but I wouldn't assume that from the get-go. You might instead just directly study filmmaking.

I realize this is a bit annoying given that you did make an honest attempt at the exercise I laid out (which I think is super cool and I appreciate, barely anyone does that). Before it makes sense to figure out how to develop general problem solving or metacognition, it's important to doublecheck whether those are the appropriate tool for your goal.

So, (I mean this as an earnest question, not like a gotcha) why are you currently interested in general problem solving (as opposed to filmmaking?) Is it because general problem solving is intrinsically interesting/rewarding to you (if you could find a path to doing so?). Or because it just seemed pretty likely to be the a good step on your journey as a filmmaker? Or just because I gave a prompt to see if you could figure out a way to apply general problemsolving to your life, and there was at least some appeal to that?

I really like the fact that there's an upvote feature together with a separate agree/disagree feature on this site.

I may like the topic, I may want to encourage the author of the post or comment to continue exploring and opening up a dialogue about that particular topic. I might think it's a valuable addition to the conversation. But I may just not agree with their conclusions.

It's an important lesson: failure can reveal important information. You don't have to agree with someone to feel richer for having understood them.

On the other hand, I'm also guilty of the vague upvote: "I don't understand this enough to comment anything other than platitudes on this, but I would like to see more of this. And maybe after reading a few more I may be able to contribute to the conversation even a sentence"

The user could always write a comment (or a separate post) asking why they got a bunch of downvotes, and someone would probably respond. I've seen this done before.

Otherwise I'd have to assume that the user is open-minded enough to actually want feedback and not be hostile. They might not even value feedback from this community; there are certainly many communities where I would think very little about negative feedback.

Though, future sama's power, money, and status all rely on GPT-(T+1) actually being smarter than them.

I wonder how he's balancing short-term and long-term interests