LessWrong 2.0 Reader

I agree with basically everything in the post, and especially that simple linear models are way undervalued. I've also come across cases where experts using literally 100x more data in there models get a worse outcome than other experts because they made a single bad assumption and didn't sanity check it properly. And I've seen cases where someone builds a linear model on the reciprocal of the variable they should have used, or where they didn't realize they were using a linear approximation of an exponential too far from the starting point. Modeling well is itself a skill that requires expertise and judgment. Other times, I see people build a simple linear model, which is built well, and then fail to notice or understand what it's telling them.

There's a Feynmann quote I love about talking simple models seriously:

As they're telling me the conditions of the theorem, I construct something which fits all the conditions. You know, you have a set (one ball)—disjoint (two balls). Then the balls turn colors, grow hairs, or whatever, in my head as they put more conditions on. Finally they state the theorem, which is some dumb thing about the ball which isn't true for my hairy green ball thing, so I say, 'False!'

And a Wittgenstein quote about not thinking enough about what model predictions and observations imply:

“Tell me,” the great twentieth-century philosopher Ludwig Wittgenstein once asked a friend, “why do people always say it was natural for man to assume that the sun went around the Earth rather than that the Earth was rotating?” His friend replied, “Well, obviously because it just looks as though the Sun is going around the Earth.” Wittgenstein responded, “Well, what would it have looked like if it had looked as though the Earth was rotating?”

Literally last week I was at an event listening to an analyst from a major outlet that produces model-based reports that people pay a lot of money for. They were telling an audience of mostly VCs that their projections pretty much ignore the future impact of any technology that isn't far enough along to have hard data. Like for energy, they projections about nuclear, but exclude SMRs, and about hydrogen, but exclude synthetic hydrocarbons. Thankfully most of the room immediately understood (based on conversations I had later in the day) that this meant the model was guaranteed to be wrong in the most important cases, even though it looks like a strong, well-calibrated track record.

The solution to that, of course, is to put all the speculative possibilities in the model, weight them at zero for the modal case, and then do a sensitivity analysis. If your sensitivity analysis shows that simple linear models vary by multiple orders of magnitude in response to small changes in weights, well, that's pretty important. But experts know if they publish models like that, most people will not read the reports carefully. They'll skim, and cherry-pick, and misrepresent what you're saying, and claim you're trying to make yourself unfalsifiable. They'll ignore the conditionality and probabilities of the different outcomes and just hear them all as "Well it could be any of these things." I have definitely been subject to all of those, and at least once (when the error bars were >100x the most likely market size for a technology) chose not to publish the numerical outcomes of my model at all.

Kai-Fu Lee, CEO of 01 AI, posted on LinkedIn:

Yi-Lightning is a small MOE model that is extremely fast and inexpensive. Yi-Lightning costs only $0.14 (RMB0.99 ) /mil tokens [...] Yi-Lightning was pre-trained on 2000 H100s for 1 month, costing about $3 million, a tiny fraction of Grok-2.

Assuming it's traned in BF16 with 40% compute utilization, that's a 2e24 FLOPs model (Llama-3-70B is about 6e24 FLOPs, but it's not MoE, so the FLOPs are not used as well). Assuming from cost that it has 10-20B active parameters, it's trained on 15-30T tokens. So not an exercise in extreme compute scaling, just excellent execution.

I see. There's a really nice post here (maybe several) that touches on that idea in a manner similar to the Ship of Theseus, but I can't find it. The basic idea was that if we take for granted the idea that mind uploads are fully conscious but then start updating the architecture to optimize for various things, is there a point where we are no longer "sentient".

oh I understood you weren't agreeing.  I was just responding that I don't know what aspects of 'firing patterns' specifically cause sentience to emerge, or how it would or wouldn't apply to your alternate scenarios.  

That’s useful analysis. Focusing so heavily on evals seems like a mistake given how AI Safety Institutes are focused on evals.

Thank you, this information was useful for a project I'm working on.

Your reasoning makes sense with regards to how a more authoritarian government would make it more likely that we can avoid x-risk, but how do you weigh that against the possibility that an AGI that is intent-aligned (but willing to accept harmful commands) would be more likely to create s-risks in the hands of an authoritarian state, as the post author has alluded to?

Also, what do you make of the author's comment below [LW(p) · GW(p)]?

• In general, the public seems pretty bought-in on AI risk being a real issue and is interested in regulation. Having democratic instincts would perhaps push in the direction of good regulation (though the relationship here seems a little less clear).

The thing I would bet is "your 'build a lifeboat for some people-like-you to move to somewhere other than EA' plan will work at least a bit, and, one of the important mechanisms for it working will be those effortful posts you wrote."

A thing I wanted to check: were you grokking the general premise that calculus and much of physics haven't been invented yet, and the metaphor here is more about an early stage physicist who has gotten a sense of how "I feel confused here, and I might need to invent [something that will turn out to be calculus]", but, it's at an early enough stage that crisp physics to easily explain it doesn't exist yet?

(If you did get that part, I'm interested in hearing a little bit more about what felt annoying, and if you didn't get that, I'm interested in what sort of things might have helped make the pre-physics/calculus part more clear)

I definitely wouldn't bet money that EA will have evolved into something you can live with (Neither EA nor the threads of rationality that he affeted evolved into things Ben Hoffman could live with)

But, I do think there is something important about the fact that, despite that, it is inaccurate to say "the critiques dropped like a stone through water" (or, what I interpret that poetry to mean, which is something like "basically nobody listened at all". I don't think I misunderstood that part but if I did then I do retract my claim)