LessWrong 2.0 Reader

I would certainly agree with part of what you are saying. Especially the point that many important lessons are taught by pain [correct me if this is misinterpreting your comment]. Indeed, as a parent for example, if your goal is for your child to gain the capacity for self sufficiency, a certain amount of painful lessons that reflect the inherent properties of the world are necessary to achieve such a goal.

On the other hand, I do not agree with your framing of pain as being the main motivator [again, correct me if required]. In fact, a wide variety of systems in the brain are concerned with calculating and granting rewards. Perhaps pain and pleasure are the two sides of the same coin, and reward maximisation and regret minimisation are identical. In practice however, I think they often lead to different solutions.

I also do not agree with your interpretation that chronic pain does not reduce agency. For family members of mine suffering from arthritis, their chronic pain renders them unable to do many basic activities, for example, access areas for which you would need to climb stairs. I would like to emphasise that it is not the disease which limits their "degrees of freedom" [at least in the short term], and were they to take a large amount of painkillers, they could temporarily climb stairs again. 

Finally, I would suggest that your framing as a "contrast between the current state and the goal state" is basically an alternative way of talking about the transition probability from the current state to the goal state. In my opinion, this suggests that our conceptualisations pain are overwhelmingly similar.

A few things stood out to me here; namely, a steelman and a strawman.

You make the sweeping claim that "AI can bring - already brings - lots of value, and general improvements to human lives", but don't substantiate that claim at all. (Maybe you think it's obvious, as a daily user. I think there's lots of room to challenge AI utility to human beings.) Much of the "benefits of AI" talk boils down to advertising and hopeful hype from invested industries. I would understand claiming, as an example, that "AI increases the productivity or speed of certain tasks, like writing cover letters". That might be an improvement in human lives, though it depends on things like whether the quality also decreases, and who or what is harmed as part of the cost of doing it. 

But this should be explored and supported; it is not at all obvious. Claiming that there is "lots of value" isn't very persuasive by itself -- especially since you include "improvements to human lives" in your statement. I'd be very curious to know which improvements to human lives AI has brought, and whether they actually stand up against, not just the dangers, but the actual already-existing downsides of AI, as well.

Those are, I feel, the strawmen in this argument. And I apologize if I make any inadvertently heated word-choices here; the tactic you're using is a theme I've been seeing that's getting under my skin: To weigh AI's "possible gains" against its "potential dangers" in a very sci-fi, what-might-happen-if-it-wakes-up way, while failing to weigh its actual, demonstrated harms and downsides as part of the equasion at all. This irks me. I think it particularly irks me when an argument (such as this one) claims all the potential upsides for AI as benefits for humans in one sweeping sentence, but then passes over the real harms to humans that even the fledgling version of AI we have has already caused, or set in motion. 

I understand that the "threat" of sentient supercomputer is sexier to think about -- and it serves as a great humblebrag for the industry, too. They get to say "Yes yes, we understand that the little people are worried our computers are TOO smart, hahaha, yes let's focus on that" -- but it's disingenuous to call the other problems "boring dangers", even though I'm sure there's no interest in discussing them at AI tech conventions. But many of these issues aren't dangers; they're already-present, active problems that function as distinct downsides to allowing AI (with agency or not) unfettered access to our marketplaces. 

Three of many possible examples, open to argument, but worthy of consideration, of already-a-downside "dangers" are: Damage to the environment / wasting tons of resources in an era where we should definitely be improving on efficiency (and you know, feeding the poor and stuff, maybe rather than "giving" Altman 7 trillion dollars). Mass-scale theft from artists and craftspeople that could harm or even destroy entire industries or areas of high human value. (And yes, that's an example of "the tech industry being bad guys" and not inherent to AI as a concept, but it is also how the real AI is built and used by the people actually doing it, and currently no-one seems able to stop them. So it's the same type of problem as having an AI that was designed poorly with regards to safety: Some rich dudes could absolutely decide to release it in spite of technicalities like human welfare. I mention this to point out that the mechanism for greedy corporations to ignore safety and human lives is already active in this space, so maybe "we're stopping all sorts of bad guys already" isn't such an ironclad reason to ignore those dangers. Or any dangers, because um, that's just kind of a terrible argument for anything; sorry.) And a third one, the scrambling of fact and fiction[1] to the point where search-engines are losing utility and people who need to know the difference for critical reasons -- like courts, scientists and teachers -- are struggling to do their work. 

All of which is a bit of a long way to say that I see a steelman and a strawman here, making this argument pretty weak overall. I also see ways you could improve both of those, by looking into the details (if they exist) of your steelman, and broadening your strawmen to include not just theoretical downsides but real ones. 

But you made me think, and elucidate something that's been irritating me for many months; thank you for that!

[1] I object to the term "hallucination"; it's inaccurate and offensive. I don't love "fiction" either, but at least it's accurate. 
 

It works only if it fly above your territory - and similar systems are used for drones detection now. Actually, they use people eyes and smartphones and instant messaging. But during recent attack on Iran a single F35 flied over Iraq and fired a missile from like 200 km distance on a target in Iran. 

You're intuition is correct when the jet has already passed ahead - those are very hard to catch and shoot down. But usually you detect an aircraft when it's heading towards you, and all the missile has to do is intercept. It doesn't even have to be faster than the jet (unless the jet detected it in time and does a 180).

The way I'd phrase the theoretical problem when you fit a model to a distribution (e.g. minimizing KL-divergence on a set of samples), you can often prove theorems of the form "the fitted-distribution has such-and-such relationship to the true distribution", e.g. you can compute confidence intervals for parameters and predictions in linear regression.

Often, all that is sufficient for those theorems to hold is:

1. The model is at an optimum
2. The model is flexible enough
3. The sample size is big enough

... because then if you have some point X you want to make predictions for, the sample size being big enough means you have a whole bunch of points in the empirical distribution that are similar to X. These points affect the loss landscape, and because you've got a flexible optimal model, that forces the model to approximate them well enough.

But this "you've got a bunch of empirical points dragging the loss around in relevant ways" part only works on-distribution, because you don't have a bunch of empirical points from off-distribution data. Even if technically they form an exponentially small slice of the true distribution, this means they only have an exponentially small effect on the loss function, and therefore being at an optimal loss is exponentially weakly informative about these points.

(Obviously this is somewhat complicated by overfitting, double descent, etc., but I think the gist of the argument goes through.)

I guess it depends on whether one makes the cut between theory and practice with or without assuming that one has learned the distribution? I.e. I'm saying if you have a distribution D, take some samples E, and approximate E with Q, then you might be able to prove that samples from Q are similar to samples from D, but you can't prove that conditioning on something exponentially unlikely in D gives you something reasonable in Q. Meanwhile you're saying that conditioning on something exponentially unlikely in D is tantamount to optimization.

https://www.richardhanania.com/p/if-scott-alexander-told-me-to-jump

Just some random thoughts:

• are the some kind of summer seasonal jobs? perhaps you could try looking for those
• find opportunities to meet local people, then ask them if they know about a job
• is there anything you could make at home and try to sell?

Let D be the distribution of (reward, trajectory) pairs for every possible trajectory.

Split D into two subsets: D1 where reward > 7 and D2 where reward ≤ 7.

Suppose that, in D, 1-in-a-googol samples is in the subset D1, and all the rest are in D2.

(For example, if a videogame involves pressing buttons for 20 minutes, you can easily have less than 1-in-a-googol chance of beating even the first mini-boss if you press the buttons randomly.)

Now we randomly pick a million samples from D in an attempt to learn the distribution D. But (as expected with overwhelming probability), it turns out that every single one of those million samples are more specifically in the subset D2.

Now consider a point X in D1 (its reward is 30). Question: Is X “out of distribution”?

Arguably no, because we set up a procedure to learn the distribution D, and D contains X.

Arguably yes, because when we ran the procedure, all the points actually used to learn the distribution were in D2, so we were kinda really only learning D2, and D2 doesn’t contain X.

(In the videogame example, if in training you never see a run that gets past the first mini-boss, then certainly intuitively we'd say that runs that do get past it, and that thus get to the next stage of the game, are OOD.)

Anyway, I was gonna say the opposite of what you said—sufficiently hard optimization via conditional sampling works in theory (i.e., if you could somehow learn D and conditionally sample on reward>7, it would work), but not in practice (because reward>7 is so hard to come by that you will never actually learn that part of D by random sampling).

I guess my question would be 'how else did you think a well-generalising sequence model would achieve this?' Like, what is a sufficient world model but a posterior over HMM states in this case? This is what GR theorem asks. (Of course, a poorly-fit model might track extraneous detail or have a bad posterior.)

From your preamble and your experiment design, it looks like you correctly anticipated the result, so this should not have been a surprise (to you). In general I object to being sold something as surprising which isn't (it strikes me as a lesser-noticed and perhaps oft-inadvertent rhetorical dark art and I see it on the rise on LW, which is sad).

That said, since I'm the only one objecting here, you appear to be more right about the surprisingness of this!

The linear probe is new news (but not surprising?) on top of GR, I agree. But the OP presents the other aspects as the surprises, and not this.

Note that I agree with your sentiment here, although my concrete argument is basically what LawrenceC wrote as a reply to this post [LW(p) · GW(p)].