LessWrong 2.0 Reader

So, how can we improve this further?

Some things I'm going to look into, please tell me if it's a waste of time:

• Seeing if there are any REITs that specialize in server farms or chip fabs and have long-term options
• Apparently McKinsey has a report about what white-collar jobs are most amenable to automation. Tracking down this report (they have lots) if it's not paywalled or at least learning enough about it to get the gist of which (non-AI) companies would save the most money by "intelligent automation".
  • From first principles I'd expect companies/industries which have a large proportion of their operating expenses going to salaries and benefits as the first in line to automate.
  • Industries that are essentially aggregators and resellers of labor would have to do this to survive at all
  • ...and the ones among them that lag in AI adoption would be candidates for short positions

what I want for rationality techniques is less a tutor and more of an assertive rubber duck walking me through things when capacity is scarce. 

In general, Intercom is the best place to send us feedback like this, though we're moderately likely to notice a top-level shortform comment.  Will look into it; sounds like it could very well be a bug.  Thanks for flagging it.

How accurate is the summary I have presented above?

Basically accurate.

Where do values, as opposed to beliefs-about-values, come from?

That is the right next question to ask. Humans have a map of their values, and can update that map in response to rewards in order to "learn about values", but still leaves the question of when/whether there's any "real values" which the map represents, and what kind-of-things those "real values" are.

A few parts of an answer:

• "human values" are not one monolithic thing; we value lots of different stuff, and different parts of our value-estimates can separately represent "a real thing" or fail to represent "a real thing".
• we don't yet understand what it means for part of our value-estimates to represent "a real thing", but it probably works pretty similarly to epistemic representation more generally - e.g. my belief about the position of the dog in my apartment represents a real thing (even if the position itself is wrong) exactly when there is in fact a dog in my apartment at all.

Given both my personal experience with LLMs and my reading of the role that empirical engagement has historically played in non-paradigmatic research, I tend to advocate for a methodology which incorporates immediate feedback loops with present day deep learning systems over the classical "philosophy -> math -> engineering" deconfusion/agent foundations paradigm.

I'm curious what your read of the history is, here? My impression is that most important paradigm-forming work so far has involved empirical feedback somehow, but often in ways exceedingly dissimilar from/illegible to prevailing scientific and engineering practice.

I have a hard time imagining scientists like e.g. Darwin, Carnot, or Shannon describing their work as depending much on "immediate feedback loops with present day" systems. So I'm curious whether you think PIBBSS would admit researchers like these into your program, were they around and pursuing similar strategies today?

If you include Facebook & Google (i.e. the entire orgs) as "frontier AI companies", then 6-figures.  If you only include Deepmind and FAIR (and OpenAI and Anthropic), maybe order of 10-15k, though who knows what turnover's been like.  Rough current headcount estimates:

Deepmind: 2600 (as of May 2024, includes post-Brain-merge employees)

Meta AI (formerly FAIR): ~1200 (unreliable sources; seems plausible, but is probably an implicit undercount since they almost certainly rely a lot of various internal infrastructure used by all of Facebook's engineering departments that they'd otherwise need to build/manage themselves.)

OpenAI: >1700

Anthropic: >500 (as of May 2024)

So that's a floor of ~6k current employees.

I discussed something similar in the "Human brains don't seem to neatly factorize" section of the Obliqueness [LW · GW] post. I think this implies that, even assuming the Orthogonality Thesis, humans don't have values that are orthogonal to human intelligence (they'd need to not respond to learning/reflection to be orthogonal in this fashion), so there's not a straightforward way to align ASI with human values by plugging in human values to more intelligence.

Over 125 current & former employees of frontier AI companies have called on @CAGovernor to #SignSB1047.

I know this is a political statement that isn't optimizing for such things, but, I am pretty interested in knowing "what actually is the denominator of people who meaningfully count as 'employees of frontier AI companies?". If the answer is 10s of thousands then, well, that is indeed a tiny number. But I think the number might be something more like 1000-3000?

Well, yes. But scientists need to have optimism that their experiments will lead somewhere, entrepeneurs have to be optimistic about there projects (and I'm optimistic that this remark will not get me kicked off this site).  Without optimism great projects would not be undertaken.

I don’t know if it’s a place for this, but at some point it became impossible to open an article in new tab from Chrome on IPhone - clicking on article title from “all posts” just opens the article. Really ruins my LW reading experience. Couldn’t quickly find a way to send this feedback to a right place either, so I guess this is a quick take now.