LessWrong 2.0 Reader

Video games also have potential legal advantages over IQ tests for companies. You could argue that "we only hire people good at video games to get people who fit our corporate culture of liking video games" but that argument doesn't work as well for IQ tests.

IANAL but unless you work for a videogame company(or a close analogue like chess.com) , I think this is just false. If your job is cognitively demanding, having IQ tests (or things like IQ tests with a mildly plausible veneer) probably won't get you in legal trouble^[1], whereas I think employment lawyers would have a field day if you install culture fit questions with extreme disparate impact, especially when it's hard to directly link the games to job performance. 

1. ^{^}

   The US Army has something like an IQ test.  So does the US Postal Service. So does the NFL. I've also personally worked in a fairly large tech company (not one of the top ones, before I moved to the Bay Area) that had ~IQ tests as one of the entrance criteria.  AFAIK there has never been any uproar about it. 

And it is important to notice that o1 is an attempt to use tons of inference as a tool, to work around its G (and other) limitations, rather than an increase in G or knowledge.

This is a rather strange statement.

o1 is basically a "System 2" addition (in terms of "Thinking, fast and slow") on top of a super-strong GPT-4o "System 1". As far as "System 1" entities go, GPT-4-level systems seem to me to be rather superior to the "System 1" "fast thinking" components of a human being^[1].

It seems to be the case that the "System 2" part is a significant component of G of a human, and it seems to be the case that o1 does represent a "System 2" addition on top of a GPT-4-level "System 1". So it seems appropriate to attribute an increase of G to this addition (given that this addition does increase its general problem-solving capabilities).

Basically, "System 2" thinking still seems to be a general capability to reason and deliberate, and not a particular skill or tool.

The post is mostly arguing that desires can shift around as we learn and think, which I agree with. But a couple parts of the post (including the title and some of the subheadings) seems to suggest something more than that: it’s not just desire-shifts, but desire convergence, towards a thing called “our values”.

(In other words, if I say that “I’m learning about blah”, then I’m strongly implying that there’s a fact-of-the-matter about blah, and that my beliefs about blah are approximately-monotonically converging towards that fact-of-the-matter. Right?)

Do you think there’s a thing (“human values”) to which desires gradually converge, via the kinds of processes described in this post? (I don’t, see §2.7 here [? · GW].)

the track record of people trying to broadly ensure that humanity continues to be in control of the future

What track record?

It seems like it's possible right?

Interestingly a similar tool run by a very smart guy (https://www.creatorml.com/) is apparently about to shut down, so it might not be a financially sustainable thing to try.

Steve uses the phrase “model-based RL” here, though that’s a pretty vague term and I’m not sure how well the usual usage fits Steve’s picture.

Here [LW · GW]’s an elaboration of what I mean:

The human brain has a model-based RL system that it uses for within-lifetime learning. I guess that previous sentence is somewhat controversial, but it really shouldn’t be:

• The brain has a model—If I go to the toy store, I expect to be able to buy a ball.
• The model is updated by self-supervised learning (i.e., predicting imminent sensory inputs and editing the model in response to prediction errors)—if I expect the ball to bounce, and then I see the ball hit the ground without bouncing, then next time I see that ball heading towards the ground, I won’t expect it to bounce.
• The model informs decision-making—If I want a bouncy ball, I won’t buy that ball, instead I’ll buy a different ball.
• There’s reinforcement learning—If I drop the ball on my foot just to see what will happen, and it really hurts, then I probably won’t do it again, and relatedly I will think of doing so as a bad idea.

…And that’s all I mean by “the brain has a model-based RL system”.

I emphatically do not mean that, if you just read a “model-based RL” paper on arxiv last week, then I think the brain works exactly like that paper you just read. On the contrary, “model-based RL” is a big tent comprising many different algorithms, once you get into details. And indeed, I don’t think “model-based RL as implemented in the brain” is exactly the same as any model-based RL algorithm on arxiv.

So, how can we improve this further?

Some things I'm going to look into, please tell me if it's a waste of time:

• Seeing if there are any REITs that specialize in server farms or chip fabs and have long-term options
• Apparently McKinsey has a report about what white-collar jobs are most amenable to automation. Tracking down this report (they have lots) if it's not paywalled or at least learning enough about it to get the gist of which (non-AI) companies would save the most money by "intelligent automation".
  • From first principles I'd expect companies/industries which have a large proportion of their operating expenses going to salaries and benefits as the first in line to automate.
  • Industries that are essentially aggregators and resellers of labor would have to do this to survive at all
  • ...and the ones among them that lag in AI adoption would be candidates for short positions

what I want for rationality techniques is less a tutor and more of an assertive rubber duck walking me through things when capacity is scarce. 

In general, Intercom is the best place to send us feedback like this, though we're moderately likely to notice a top-level shortform comment.  Will look into it; sounds like it could very well be a bug.  Thanks for flagging it.

How accurate is the summary I have presented above?

Basically accurate.

Where do values, as opposed to beliefs-about-values, come from?

That is the right next question to ask. Humans have a map of their values, and can update that map in response to rewards in order to "learn about values", but still leaves the question of when/whether there's any "real values" which the map represents, and what kind-of-things those "real values" are.

A few parts of an answer:

• "human values" are not one monolithic thing; we value lots of different stuff, and different parts of our value-estimates can separately represent "a real thing" or fail to represent "a real thing".
• we don't yet understand what it means for part of our value-estimates to represent "a real thing", but it probably works pretty similarly to epistemic representation more generally - e.g. my belief about the position of the dog in my apartment represents a real thing (even if the position itself is wrong) exactly when there is in fact a dog in my apartment at all.