LessWrong 2.0 Reader

All good points, many I agree with. If nothing else, I think that humanity should pre-commit to following this strategy whenever we find ourselves in the strong position. It's the right choice ethically, and may also be protective against some potentially hostile outside forces.

However, I don't think the acausal trade case is strong enough that I would expect all sufficiently powerful civilizations to have adopted it. If I imagine two powerful civilizations with roughly identical starting points, one of which expanded while being willing to pay costs to accommodate weaker allies while the other did not and instead seized whatever they could, then it is not clear to me who wins when they meet. If I imagine a process by which a civilization becomes strong enough to travel the stars and destroy humanity, it's not clear to me that this requires it to have the kinds of minds that will deeply accept this reasoning. 

It might even be that the Fermi paradox makes the case stronger - if sapient life is rare, then the costs paid by the strong to cooperate are low, and it's easier to hold to such a strategy/ideal.

I struggle a bit to remember what ASI is but I'm gonna assume it's Artificial Super Intelligence. 

Let's say that that's markedly cleverer than 1 person. So it's capable of running very successful trading strategies or programming extremely well. It's not clear to me that such a being:

• Has been driven towards being agentic, when its creators will prefer something more docile
• Can cooperate well enough with itself to manage some massive secret takeover
• Is competent enough to recursively self improve (and solve the alignment problems that creates)
• Can beat everyone else combined

Feels like what such a being/system might do is just run some terrifically successful trading strategies and gather a lot of resources while frantically avoiding notice/trying to claim it won't take over anything else. Huge public outcry, continuing regulation but maybe after a year it settles to some kind of equilibrium. 

Chance of increasing capabilities and then some later jump, but seems plausible to me that that wouldn't happen in one go. 

Hi everyone! Skylar (ACX meetup czar) has asked me to circulate the following short survey about the meetup: https://docs.google.com/forms/d/e/1FAIpQLSeTUw3pHQK0oGEKZwHjPFar3PyWkEry-X5M6hD6cGrJpgplBQ/viewform 

If you have a moment to fill it out, that would be awesome. Thanks!

It's pretty sad to call all of these end states you describe alignment as alignment is an extremely natural word for "actually terminally has good intentions".

Aren't there a lot of clearer words for this? "Well-intentioned", "nice", "benevolent", etc.

(And a lot of terms, like "value loading" and "value learning", that are pointing at the research project of getting good intentions into the AI.)

To my ear, "aligned person" sounds less like "this person wishes the best for me", and more like "this person will behave in the right ways".

If I hear that Russia and China are "aligned", I do assume that their intentions play a big role in that, but I also assume that their circumstances, capabilities, etc. matter too. Alignment in geopolitics can be temporary or situational, and it almost never means that Russia cares about China as much as China cares about itself, or vice versa.

And if we step back from the human realm, an engineered system can be "aligned" in contexts that have nothing to do with goal-oriented behavior, but are just about ensuring components are in the right place relative to each other.

Cf. the history of the term "AI alignment" [LW(p) · GW(p)]. From my perspective, a big part of why MIRI coordinated with Stuart Russell to introduce the term "AI alignment" was that we wanted to switch away from "Friendly AI" to a term that sounded more neutral. "Friendly AI research" had always been intended to subsume the full technical problem of making powerful AI systems safe and aimable; but emphasizing "Friendliness" made it sound like the problem was purely about value loading, so a more generic and low-content word seemed desirable.

But Stuart Russell (and later Paul Christiano) had a different vision in mind for what they wanted "alignment" to be, and MIRI apparently failed to communicate and coordinate with Russell to avoid a namespace collision. So we ended up with a messy patchwork of different definitions.

I've basically given up on trying to achieve uniformity on what "AI alignment" is; the best we can do, I think, is clarify whether we're talking about "intent alignment" vs. "outcome alignment" when the distinction matters.

But I do want to push back against those who think outcome alignment is just an unhelpful concept — on the contrary, if we didn't have a word for this idea I think it would be very important to invent one. 

IMO it matters more that we keep our eye on the ball (i.e., think about the actual outcomes we want and keep researchers' focus on how to achieve those outcomes) than that we define an extremely crisp, easily-packaged technical concept (that is at best a loose proxy for what we actually want). Especially now that ASI seems nearer at hand (so the need for this "keep our eye on the ball" skill is becoming less and less theoretical), and especially now that ASI disaster concerns have hit the mainstream (so the need to "sell" AI risk has diminished somewhat, and the need to direct research talent at the most important problems has increased).

And I also want to push back against the idea that a priori, before we got stuck with the current terminology mess, it should have been obvious that "alignment" is about AI systems' goals and/or intentions, rather than about their behavior or overall designs. I think intent alignment took off because Stuart Russell and Paul Christiano advocated for that usage and encouraged others to use the term that way, not because this was the only option available to us.

“How many Oxford dons does it take to change a lightbulb?”

CHANGE‽‽‽

Besides math and programming, what are your other skills and interests?

*

I have an idea of a puzzle game, not sure if it would be good or bad, I haven't done even a prototype. So if anyone is interested, feel free to try... I hope I can explain it sufficiently clearly in words...

The game plan is divided into squares; I imagine a typical level to be between 10x10 and 30x30 squares large. Each square is either empty, or contains an immovable wall, or contains a movable block. The game consists of moving the blocks. Each move = you click a specific block, and try dragging it in one of the 4 directions, and either it is possible or not.

A block cannot move into a wall. A block can push another block. A block does not pull another block. For example, if there are 3 blocks in a horizontal line, and you click the middle one and try dragging it to the left, two blocks will move and the third one (the one on the right) will stay there. So far, it should be completely obvious, like what you would happen if you moved some actual objects.

In addition, each side of a block (or a wall) may be empty, or may contain a colored "magnet" (or perhaps a "lock" is a better metaphor). These add the following constraints for the movement of blocks:

• Magnets of different colors can never touch each other. If one block has a green magnet on the right side, and another has a blue magnet on the left side, you cannot put them next to each other so that the magnets would touch. (If you try to do that, the block refuses to move. Graphically, I imagine that it would move like half the way, and then you would get a visual indicator where is the problem, and when you stop dragging, it will return to its original place.) Though it is okay if the blocks touch on their other sides, where they don't have magnets.
• Magnets of the same color cannot be connected or disconnected by a move in a perpendicular direction. If one block has a green magnet on the right side, and another has a green magnet on the left side, if you move them next to each other, then when you try moving one of them up or down, it drags the other block along with it. Either both blocks move (in a direction perpendicular to their magnetic connection) or neither does. In a direction parallel to the magnetic connection, either one block pushes the other, or they disconnect if you pull them apart (i.e. the magnets do nothing when moving in a parallel direction).
• A magnet can touch a side without a magnet, doing so has no effect as if the magnet is not there.

Or, to describe it more like a programmer:

1. You choose a block and a direction to move. Now we create a set of "blocks that will move one step in given direction" like this: At first, the set contains the selected block. For each block in the set, a block next to it in the selected direction is also added to the set (pushed by the previous block). For each block in the set, a block next to it in a perpendicular direction is also added to the set if they are connected by magnets of the same color. We keep applying these two rules until we can add no more blocks to the set.
2. Now we check what would happen if blocks in the set moved one step in given direction, and all other blocks stayed at their place. If any block would move into a wall, the entire move is cancelled. (A block cannot move into another block, because by the set creation algorithm, that other block would also be in the set, and thus it would also move.) If two blocks -- one that moved, and one that didn't move -- would end up next to each other so that their magnets would touch each other (regardless of their colors), the entire move is cancelled. In both cases, the place that causes the problem is visually indicated to the player. (That is, even if you already know that the move is cancelled, keep checking which other places you also need to highlight. Then move all blocks in the set a few pixels in a given direction, so the player sees which blocks would be pushed along.)
3. If there is no problem, the blocks in the set all simultaneously move one step the given direction.

I think that these rules are time-reversible; whatever move you make, you can revert it by one or more moves. This is a desirable property, because it means you can never get stuck in the game. (It also means you can automatically generate levels by generating a solution and then making a few hundred random moves.)

A magnet can also be on the side of a wall. (The wall is basically a block that cannot be moved.)

The puzzle is solved when each magnet is connected to a magnet of the same color.

For bonus points, include a visual editor, and maybe an export/import of levels to a text file.

Also here are some sessions tentatively scheduled (some may change):

1. Fighting Moloch in Politics talk/Q&A with Martin Sustrik [LW · GW]
2. One-Shot 'Baba Is You' Rationality Exercises activity with Raymond Arnold [LW · GW]
3. Write Your First Fact-Post activity led by Sarah Constantin
4. Currently Untitled Sequel to And All the Shoggoths Merely Players [LW · GW] narrated by Zack Davis [LW · GW] and John Wentworth [LW · GW]
5. Write Your First Glowfic activity led by Alicorn [LW · GW]
6. Podcast and Q&A with Alexander Wales (author of Worth the Candle) and Daystar Eld, moderated by Jamie Wahls
7. Wanted: People Who Want by Jacob Falkovich
8. Magic-The-Gathering Color Wheel for Writers talk by Duncan Sabien

What would a "qualia-first-calibration" app would look like?

Or, maybe: "metadata-first calibration"

The thing with putting probabilities on things is that often, the probabilities are made up. And the final probability throws away a lot of information about where it actually came from.

I'm experimenting with primarily focusing on "what are all the little-metadata-flags associated with this prediction?". I think some of this is about "feelings you have" and some of it is about "what do you actually know about this topic?"

The sort of app I'm imagining would help me identify whatever indicators are most useful to me. Ideally it has a bunch of users, and types of indicators that have been useful to lots of users can promoted as things to think about when you make predictions.

Braindump of possible prompts:

– is there a "reference class" you can compare it to?

– for each probability bucket, how do you feel? (including 'confident'/'unconfident' as well as things like 'anxious', 'sad', etc)

– what overall feelings do you have looking at the question?

– what felt senses do you experience as you mull over the question ("my back tingles", "I feel the Color Red")

...

My first thought here is to have various tags you can re-use, but, another option is to just do totally unstructured text-dump and somehow do factor analysis on word patterns later?

topic: economics

idea: when building something with local negative externalities, have some mechanism to measure the externalities in terms of how much the surrounding property valuation changed (or are expected to change based, say, through a prediction market) and have the owner of that new structure pay the owners of the surrounding properties.

Thanks!

• one way to construct an HMM is by finding all past histories of tokens that condition the future tokens with the same probablity distribution, and make that equivalence class a hidden state in your HMM. Then the conditional distributions determine the arrows coming out of your state and which state you go to next. This is called the "epsilon machine" in Comp Mech, and it is unique. It is one presentation of the data generating process, but in general there are an infinite number of HMM presntations that would generate the same data. The epsilon machine is a particular type of HMM presentation - it is the smallest one where the hidden states are the minimal sufficient statistics for predicting the future based on the past. The epsilon machine is one of the most fundamental things in Comp Mech but I didn't talk about it in this post. In the future we plan to make a more generic Comp Mech primer that will go through these and other concepts.
• The interpretability of these simplexes is an issue that's in my mind a lot these days. The short answer is I'm still wrestling with it. We have a rough experimental plan to go about studying this issue but for now, here are some related questions I have in my mind:
  • What is the relationship between the belief states in the simplex and what mech interp people call "features"?
  • What are the information theoretic aspects of natural language (or coding databases or some other interesting training data) that we can instantiate in toy models and then use our understanding of these toy systems to test if similar findings apply to real systems.

For something like situational awareness, I have the beginnings of a story in my head but it's too handwavy to share right now. For something slightly more mundane like out-of-distribution generaliztion or transfer learning or abstraction, the idea would be to use our ability to formalize data-generating structure as HMMs, and then do theory and experiments on what it would mean for a transformer to understand that e.g. two HMMs have similar hidden/abstract structure but different vocabs. 

Hopefully we'll have a lot more to say about this kind of thing soon!