LessWrong 2.0 Reader

If developments like this continue, could open weights models be made into a case for not racing? E.g. if everyone's getting access to the weights, what's the point in spending billions to get there 2 weeks earlier?

this can be done more scalably in a text game, no? 

Physicalist epiphenomenalism is the only philosophy that is compatible with the autonomy of matter and my experience of consciousness, so it has not competitors as a cosmovision

No, identity theory and illusionism are competitors. As I have ponied out before.

I appreciate the discussion, but I'm disappointed by the lack of rigor in proposals, and somewhat expect failure for the entire endeavor of quantifying empathy (which is the underlying drive for discussing consciousness in these contexts, as far as I'm concerned).
 

Of course, we do not measure computers by mass, but by speed, number of processors and information integration. But if you directly do not have enough computing capacity, your neural network is simply small and information processing is limited.

It's worth going one step further here - how DO we measure computers, and how might that apply to consciousness?  Computer benchmarking is a pretty complex topic, with most of the objective trivial measures (FlOps, IOPS, data throughput, etc.) being well-known to not tell the important details, and specific usage benchmarks being required to really evaluate a computing system.  Number of transistors is a marketing datum, not a measure of value for any given purpose.

Until we get closer to actual measurements of cognition and emotion, we're unlikely to get any agreement on relative importance of different entities' experiences.

If you consider the "true ability" to be the exponential of the subtest scores, then the extent to which the problem I mention applies depends on the base of the exponential. In the limiting case where the base goes to infinity, only the highest ability matter, whereas in the limiting case where the base goes to 1, you end up with something basically linear.

As for whether it's a crux, approximately nobody has thought about this deeply enough that they would recognize it, but I think it's pretty foundational for a lot of disagreements about IQ.

Possible bug report: today I've been seeing errors of the form

Error: Cannot query field "givingSeason2024VotedFlair" on type "User". Did you mean "givingSeason2024DonatedFlair"?

that tend to go away when the page is refreshed. I don't remember if all errors said this same thing.

https://www.lesswrong.com/tag/recursive-self-improvement [? · GW]

DeepSeek is Chinese. I'm not really familiar with the company.

DeepSeek is the best Chinese DL research group now and have been for at least a year. If you are interested in the topic, you ought to learn more about them.

I thought Chinese companies were at least a year behind the frontier.

This seems roughly consistent with what you would expect. People usually say half a year to a year behind. Q* was invented somewhere in summer 2023, according to the OA coup reporting; ballpark June-July 2023, I got the impression since it seemed to already be a topic of discussion with the Board c. August 2023 pre-coup. Thus, we are now (~20 Nov 2024) almost at December 2024, which is about a year and a half. o1-preview was announced 12 September 2024, 74 days ago, and o1-preview's benchmarks were much worse than the true o1 which was still training then (and of course, OA has kept improving it ever since, even if we don't know how - remember, time is always passing, and what you read in a blog post may already be ancient history). Opensource/competitor models (not just Chinese or DeepSeek specifically) have a long history of disappointing in practice when they turn out to be much narrower, overfit to benchmarks, or otherwise somehow lacking in quality & polish compared to the GPT-4s or Claude-3s.

So, if a competing model claims to match o1-preview from almost 3 months ago, which itself is far behind o1, with additional penalties from compensating for the hype and the apples-to-oranges comparisons, and where we still don't know if they are actually the same algorithm at core (inasmuch as neither OA nor DeepSeek, AFAIK, have yet to publish any kind of detailed description of what Q*/Strawberry/r1 is), and possibly as much as >1.5 years behind if DS has gone down a dead end... Overall, this still seems roughly what you would expect now: half a year to a year behind. It's always a lot easier to catch up [LW(p) · GW(p)] with an idea after someone else has proven it works and given you an awful lot of hints about how it probably works, like the raw sample transcripts. (I particularly note the linguistic tics in this DS version too, which I take as evidence for my inner-monologue splicing guess of how the Q* algorithm works.)

Shouldn't we expect that ultimately the only thing selected for is mostly caring about long run power?

I was attempting to address that in my first footnote, though maybe it's to important a consideration to be relegated to a footnote. 

To say it differently, I think we'll see selection evolutionary fitness, which can take two forms:

• Selection on AIs' values, for values that are more fit, given the environment.
• Selection on AIs' rationality and time preference, for long-term strategic VNM rationality.

These are "substitutes" for each other. An agent can either have adaptive values, adaptive strategic orientation, or some combination of both. But agents that fall below the Pareto frontier described by those two axes^[1], will be outcompeted.

Early in the singularity, I expect to see more selection on values, and later in the singularity (and beyond), I expect to see more selection on strategic rationality, because I (non-confidently) expect the earliest systems to be myopic and incoherent in roughly similar ways to humans (though probably the distribution of AIs will vary more on those traits than humans). 

The fewer generations there are before strong, VNM agents with patient values / long time preferences, the less I expect small amounts of caring for human in AI systems will be eroded. 

1. ^{^}

   Actually, "axes" are a bit misleading since the space of possible values is vast and high dimensional. But we can project it onto the scalar of "how fit are these values (given some other assumptions)?"

You're probably right. I neglected check how effective this would be in any quantitative sense.

I think you underestimate the cost of street-level murals ($100K / mi is about $60 / m), and neglect the benefit of tunnels' inevitable insulation, but the decision would probably end up the same.