@nostalgebraist [LW · GW] bites that bullet here:

“Reinforcement learning” (RL) is not a technique.  It’s a problem statement, i.e. a way of framing a task as an optimization problem, so you can hand it over to a mechanical optimizer.

What’s more, even calling it a problem statement is misleading,  because it’s (almost) the most general problem statement possible for any arbitrary task.  If you try to formalize a concept like “doing a task well,“ or even “being an entity that acts freely and wants things,” in the most generic terms with no constraints whatsoever, you end up writing down “reinforcement learning.”

and so does Russell & Norvig 3rd edition

Reinforcement learning might be considered to encompass all of AI.

…That’s not my perspective though.

For my part, there’s a stereotypical core of things-I-call-RL which entails:

• (A) There’s a notion of the AI’s outputs being better or worse
• (B) But we don’t have ground truth (even after-the-fact) about what any particular output should ideally have been
• (C) Therefore the system needs to do some kind of explore-exploit.

By this (loose) definition, both model-based and model-free RL are central examples of “reinforcement learning”, whereas LLM self-supervised base models are not reinforcement learning (cf. (B), (C)), nor are ConvNet classifiers trained on ImageNet (ditto), nor are clustering algorithms (cf. (A), (C)), nor is A* or any other exhaustive search within a simple deterministic domain (cf. (C)), nor are VAEs (cf. (C)), etc.

(A,B,C) is generally the situation you face if you want an AI to win at videogames or board games, control bodies while adapting to unpredictable injuries or terrain, write down math proofs, design chips, found companies, and so on.

This (loose) definition of RL connects to AGI safety because (B-C) makes it harder to predict the outputs of an RL system. E.g. we can plausibly guess that an LLM base model, given internet-text-like prompts, will continue in an internet-text-typical way. Granted, given OOD prompts, it’s harder to say things a priori about the output. But that’s nothing compared to e.g. AlphaZero or AlphaStar, where we’re almost completely in the dark about what the trained model will do in any nontrivial game-state whatsoever. (…Then extrapolate the latter to human-level AGIs acting in the real world!)

(That’s not an argument that “we’re doomed if AGI is based on RL”, but I do think that a very RL-centric AGI would need tailored approaches to thinking about safety and alignment that wouldn’t apply to LLMs; and I likewise think that likewise a massive increase in the scope and centrality of LLM-related RL (beyond the RLHF status quo) would raise new (and concerning) alignment issues, different from the ones we’re used to with LLMs today.)

highly convergent

Huh? A hyperphone is a two-player tool. Loom is a one-player tool.

Generally the hypothesis is that most people will get more sodium in their diet than they crave with their natural desire, if they just eat the food of least resistance (cheapest or easiest, most shelf stable, whatnot). A lot of the sodium that gets into your diet is not so richly activating your taste buds as table salt applied to taste.

What we want overall with salinity is to preserve it at a level that's correct for us, because we take it in through our diet and excrete it through various processes like sweat. Excessive salt consumption doesn't directly affect your overall salt and water balance that much, because the body has hormonal regulation of various mechanisms to keep it stable - it's presumably the overworking of these mechanisms that causes health issues, which is much preferable than it causing issues directly if you've seen the effects of the wrong salinity on cells in a petri dish under a microscope.

(The effects on whatever cells I was looking at, which started at a neutral salinity: Raising the salinity (saltier) caused them to shrivel up and dessicate like raisins; lowering the salinity (less salty) caused them to explode.)

What are you Doing? What did you Plan?

[Suno]

What are you doing? What did you plan? Are they aligned? If not then comprehend, if what you are doing now is better than the original thing. Be open-minded about, what is the optimal thing.

Don't fix the bottom line too: "Whatever the initial plan was is the best thing to do."

There are sub-agents in your mind. You don't want to fight, with them, as usually they win in the end. You might then just feel bad and don't even understand why. As a protective skin your sub-agent hides, the reasons for why, you feel so bad right now.

At that point, you need to pack the double crux out.

But ideally, we want to avoid, any conflict that might arise. So don't ask yourself if you followed your consequentialist reasoner's plan. Instead just ask: "What is the best thing for me to do right now?" while taking all the sub-agents into account.

To do it set a timer for 1 minute, and spend that time reflecting about: What do you want to get out of this session of work, why is this good, how does this help?

You can wirte notes in advance, then document your plans, and then read them out loud.

to remember the computations your brain did before, such that you don't need to repeat some of these chores.

Ideally, the notes would talk about, the reasons for why something seemed like a good thing to try.

But then as you evaluate what next step you could take, drop that bottom line. Treat it as evidence for what your brain computed in the past as an optimal policy, but nothing more. It's now your new goal to figure out again for yourself, using all the subagents within your shell.

And to do this regularly you of course use a timer you see. Every 30 minutes to an hour it should ring out loud reminding you to evaluate, what would be the next step to take.

If you let everybody influence the decision process that will commence, the probability is high that after you decide there will be no fight, in your mind.

On the politics part : one thing I like very much with the Roman republic system was the concept of the "cursus honorum". Basically if you wanted to go for a politician career you had to start at the bottom, get elected to a first position, do well, get elected to something more prestigious, etc. And it worked very well - a significant part of Roman success was that their government (and generals) were way better than competing powers, in this was mainly due to having a lot of experienced, competent politicians and generals with somewhat well aligned incentives.

It was whelming.

So, sure, there is a threshold effect in whether you get value from bike lanes on your complex journey from point A to point G.  But other people throughout the city have different threshold effects:

• Other people are starting and ending their trips from other points; some people are even starting and ending their trip entirely on Naito Parkway.
• People have a variety of different tolerances for how much they are willing to bike in streets, as you mention.
• Even people who don't like biking in streets often have some flexibility.  You say that you personally are flexible, "but for the sake of argument, let's just assume that there is no flexibility".  But in real life, even many people who absolutely refuse to bike in traffic might be happy to walk their bike on the sidewalk (or whatever) for a single block, in order to connect two long stretches of beautiful bike path.

When you add together a million different possible journeys across thousands of different people, each with their own threshold effects, the sum of the utility provided by each new bike lane probably ends up looking much more like a smooth continuum of incremental benefits per each new bike lane that is added to a city's network, with no killer threshold effects.  This is very different from a bridge, where indeed half a bridge is not useful to any portion of the city's residents.

Therefore, I don't think "if you're going to start building [a bike lane network], you better m ake sure you have plans to finish it".  Rather, I think adding random pieces of the network piecemeal (as random roads undergo construction work for other reasons, perhaps) is a totally reasonable thing for cities to do.

Another example of individual threshold effects adding up to continuous benefit from the provider's perspective:  suppose I only like listening to thrash-metal songs on Spotify.  Whenever Spotify adds a song from a genre I don't care for -- pop, classical, doom-metal, whatever -- it provides literally ZERO value to me!  There's a huge threshold effect, where I only care when spotify adds thrash-metal songs!  But of course, since everyone has different preferences, the overall effect of adding songs from Spotify's perspective is to provide an incremental benefit to the quality of their product each time they add a song.  (Disclaimer: I am not actually a thrash-metal fanatic.)

Personally I think this would be pretty cool!

I wouldn't call this "AI lab watch." "Lab" has the connotation that these are small projects instead of multibillion dollar corporate behemoths.

Disagree on "lab". I think it's the standard and most natural term now. As evidence, see your own usage a few sentences later:

They've all committed to this in the WH voluntary commitments and I think the labs are doing things on this front.

I'm interested in what people think of are the strongest arguments against this view. Here are a few counterarguments that I'm aware of: 

1. Empirically the AI-focused scaling labs seem to care quite a lot about safety, and make credible commitments for safety. If anything, they seem to be "ahead of the curve" compared to larger tech companies or governments.

2. Government/intergovernmental agencies, and to a lesser degree larger companies, are bureaucratic and sclerotic and generally less competent. 

3. The AGI safety issues that EAs worry about the most are abstract and speculative, so having a "normal" safety culture isn't as helpful as buying in into the more abstract arguments, which you might expect to be easier to do for newer companies.

4. Scaling labs share "my" values. So AI doom aside, all else equal, you might still want scaling labs to "win" over democratically elected governments/populist control.