LessWrong 2.0 Reader

Grabby distributed backup computronium that can't be overtaken in the expansion chase.

AGI-powered defense

You mention both "local" and "cosmic" unfairness, but the body of the post appears to focus solely on the "cosmic", to its detriment. The challenges of Dostoevsky (or Qureshi-Hurst, but I am not familiar with her work) are not about whether "cosmic" unfairness can have some rationale, but about this suffering person here -- and for that person, notions of some "Divine Plan" (in whatever terms we may conceive of such) do not provide any relief. Religions that include belief in such things as angels or Divine incarnations face an even more stark problem from the lack of intervention; or rather, the lack of inconsistent intervention, by those spiritual powers.

While transformers are training, there may be certain quantities which are close to constant with high probability over the test distribution because of a concentration of measure phenomenon.

It's natural for the transformer to learn algorithms which rely on these quantities being constant, because they essentially are over the test distribution. 

One strategy for handling this is to first identify these quantities, and then prove that they are indeed close to constant with high probability over the test distribution using concentration inequalities.

A large source of these quantities is the denominator of any softmaxes. These are sums of many independent variables, and so concentrate around a constant. The softmax denominator can often be factored into (components that aren't constant) * (a constant), where this choice of factorization is often problem-specific. Once you've identified this constant, it simplifies the analysis of the softmax considerably.

There are lots of useful concentration inequalities which help prove that sums of independent variables concentrate around a constant.

Humanity has only ever eradicated two diseases (and one of those, rinderpest, is only in cattle not humans). The next disease on the list is probably Guinea worm (though polio is also tantalizingly close).

At its peak Guinea worm infected ~900k people a year. In 2024 we so far only know of 7 cases. The disease isn't deadly, but it causes significant pain for 1-3 weeks (as a worm burrows out of your skin!) and in ~30% of cases that pain persists afterwards for about a year. In .5% of cases the worm burrows through important ligaments and leaves you permanently disabled. Eradication efforts have already saved about 2 million DALYs.^[1]

I don't think this outcome was overdetermined; there's no recent medical breakthrough behind this progress. It just took a herculean act of international coordination and logistics. It took distributing millions of water filters, establishing village-based surveillance systems in thousands of villages across multiple countries, and meticulously tracking every single case of Guinea worm in humans or livestock around the world. It took brokering a six-month ceasefire in Sudan (the longest humanitarian ceasefire in history!) to allow healthcare workers to access the region. I've only skimmed the history, and I'm generally skeptical of historical heroes getting all the credit, but I tentatively think it took Jimmy Carter for all of this to happen.

Rest in peace, Jimmy Carter.

1. ^{^}

   I'm compelled to caveat that top GiveWell charities are probably in the ballpark of $50/DALY, and the Carter Center has an annual budget of ~$150 million a year, so they "should" be able to buy 2 million DALYs every single year by donating to more cost-effective charities. But c'mon this worm is super squicky and nearly eradicating it is an amazing act of agency.

Very cool tool; thank you for making it!

A tangential question:

Recent systems with Python interpreters offer more computational power but aren't optimized for intuitive probabilistic modeling.

Are you thinking here of the new-ish canvases built into the chat interfaces of some of the major LLMs (Claude, ChatGPT)? Or are there tools specifically optimized for this that you think are good? Thanks!

Yeah, to be more clear, it's not so much that she got that wrong. It's that she didn't accept the reasoning behind that number enough to really believe it. She added a discount factor based on fallacious reasoning around "if it were that easy, it'd be here already".

Some hypotheses predict that we should have already been able to afford to train a transformative model with reasonable probability; I think this is unlikely, so I execute an update against low levels of FLOP...

In my opinion the correct takeaway from an estimate showing that "a low number of FLOPs should be enough if the right algorithm is used" combined with "we don't seem close, and are using more FLOPs than that already" is "therefore, our algorithms must be pretty crap and there must be lots of room for algorithmic improvement." Thus, that we are in a compute hardware overhang and that algorithmic improvement will potentially result in huge capabilities gains, speed gains, and parallel inference instances. Of course, if our algorithms are really that far from optimal, why should we not expect to continue to be bottlenecked by algorithms? The conclusion I come to is that if we can compensate for inefficient algorithms with huge over-expenditure in compute, then we can get a just-barely-good-enough ML research assistant who can speed up the algorithm progress. So we should expect training costs to drop rapidly after automating ML R&D.

Then she also blended the human brain compute estimate with three other estimates which were multiple orders of magnitude larger. These other estimates were based on, in my opinion, faulty premises.

I first lay out four hypotheses about 2020 training computation requirements, each of which anchors on a key quantity estimated from biology: total computation done over evolution, total computation done over a human lifetime, the computational power of the human brain, and the amount of information in the human genome...

Since the time of her report she has had to repeatedly revise her timelines down. Now she basically agrees with me. Props to her for correcting.

https://www.lesswrong.com/posts/K2D45BNxnZjdpSX2j/ai-timelines?commentId=hnrfbFCP7Hu6N6Lsp [LW(p) · GW(p)]

This cleared it up for me. So this isn't really about a preference for "warfighting" or "danger", this is about a preference for death in battle, as in Norse Vikings, Japanese Samurai, and Spartan and Aztec Warriors. This is more clearly analogous to a preference for chastity in terms of direct reproductive success. Preferences for death by martyrdom could be another example.

It has basically significantly accelerated my ability to build fully functional websites very quickly. To the point where it was basically a phase transition between me building my org’s website and not building it (waiting for someone with web dev experience to do it for me).

I started my website by leveraging the free codebase template he provides on his github and covers in the course.

It seems to me that AIs have remained stubbornly terrible at this from GPT-3 to GPT-4 to Sonnet 3.5.1 to o1^[2]; that the improvement on this hard-to-specify quality has been ~0. I guess we'll see if o3 (or an o-series model based on the next-generation base model) change that. AI does feel right on the cusp of getting good at this...

... just as it felt at the time of GPT-3.5, and GPT-4, and Sonnet 3.5.1, and o1. That just the slightest improvement along this axis would allow us to plug the outputs of AI cognition into its inputs and get a competent, autonomous AI agent.

Boy do I disagree with this take! Excited to discuss.

Can you say more about what skills you think the GPT series has shown ~0 improvement on?

Because if it's "competent, autonomous agency" then there has been massive progress over the last two years and over the last few months in particular. METR has basically spent dozens of FTE-years specifically trying to measure progress in autonomous agency capability, both with formal benchmarks and with lots of high-surface-area interaction with models (they have people building scaffolds to make the AIs into agents and do various tasks etc.) And METR seems to think that progress has been rapid and indeed faster than they expected.

Has there been enough progress to automate swathes of jobs? No, of course not -- see the benchmarks. E.g. RE-bench shows that even the best public models like o1 and newsonnet are only as good as professional coders on time horizons of, like, an hour or so. (give or take, depends on how you measure, the task, etc.) Which means that if you give them the sort of task that would take a normal employee, like, three hours, they are worse than a competent human professional. Specifically they'd burn lots of tokens and compute and push lots of buggy code and overall make a mess of things, just like an eager but incompetent employee.

And I'd say the models are unusually good at these coding tasks compared to other kinds of useful professional tasks, because the companies have been trying harder to train them to code and it's inherently easier to train due to faster feedback loops etc.

This (the Sun is the  only important local source of ... anything) has been an obvious conclusion for decades.  Freeman Dyson described one avenue to capturing all the energy in 1960.  The recent changes that make it more salient (and framed as "eating or inhabiting the sun" rather than "capturing the sun's output") is the recent progress in AI, which does two things:

1. Adds weight to the computational theory of mind.  If everything important is "just" computation, then all this expense and complexity of human bodies and organic brains is temporary and will be unnecessary in the future.  This simplifies the problem of HOW to eat the sun into just how to make computronium out of it.
2. Provides a more believable path for solving very hard engineering problems, by using smarter engineers than we can currently birth and train.  It does NOT actually solve the problems, or even prove that they're solvable.  We don't actually know what computation is (for this purpose), or how to optimize the entropy problem of "making one thing more ordered always makes other things less ordered".

That said, I don't know how to make beliefs on this scale pay any rent.  "within 10,000 years" and "that's just science fiction" are identical labels to me.