LessWrong 2.0 Reader

IMO the criterion for selecting the positive training examples should be that the chatbot won the argument, under standard debating rules (plus Godwin's Law, of course): it net-shifted a vote of humans towards its position. If the aim is to evoke USENET, I think we should allow the chatbot to use more then one persona holding more than one viewpoint, even ones that also argue with each other.

That said, I do share the impression that others here have expressed of it heading "downhill". Watered down. Lower standards. Less serious. Stuff like that. I find it a little annoying and disappointing, but nothing too crazy.

I don't know, looking back at older posts (especially on LW 1.0) current LW is less "schizo" and more rigorous/boring—though maybe that's because I sometimes see insanely long & detailed mechinterp posts?

Trying this prompting approach briefly on GPT-4, if you just venture a clearly-mistaken opinion, it does politely but informatively correct you (distinctly not USENET-style). On some debatable subjects it was rather sycophantic to my viewpoint, though with a bit of on-the-other-hand push-back in later paragraphs. So I'm gradually coming to the opinion this is only about as humorous as Grok. But it still might be a thought-provoking change of pace.

I've strong-upvoted it to -1, because I agree.

What assumptions is this making about scaling laws for these benchmarks? I wouldn't know how to convert laws for losses into these kind of fuzzy benchmarks.

I'm almost tempted to Cunningham-tune a Mistral 7B base model. We'd only need O(10,000) good examples and O($100). And it would be funny as hell.

The question in point 2 is whether an ASI could sustain itself without humans and without new types of hardware such as Drexler style nanomachinery, which to a significant portion of people (me not included) seems to be too hypothetical to be of actual concern. I currently don't see why the answer to that question should be a highly certain no, as you seem to suggest. Here are some thoughts:

• The world economy is largely catering to human needs, such as nutrition, shelter, healthcare, personal transport, entertainment and so on. Phenomena like massive food waste and people stuck in bullshit jobs, to name just two, also indicate that it's not close to optimal in that. An ASI would therefore not have to prevent a world economy from collapsing or pick it up afterwards, which I also don't think is remotely possible with existing hardware. I think the majority of processes running in the only example of a world economy we have is irrelevant to the self-preservation of an ASI.
• An ASI would presumably need to keep it's initial compute substrate running long enough to transition into some autocatalytic cycle, be it on the original or a new substrate. (As a side remark, it's also thinkable that it might go into a reduced or dormant state for a while and let less energy- and compute-demanding processes act on its behalf until conditions have improved on some metric). I do believe that conventional robotics is sufficient to keep the lights on long enough, but to be perfectly honest, that's conditioned on a lack of knowledge about many specifics, like exact numbers of hardware turnover and energy requirements of data centers capable of running frontier models, the amount and quality of chips currently existing on the planet, the actual complexity of keeping different types of power plants running for a relevant period of time, the many detailed issues of existing power grids, etc. I weakly suspect there is some robustness built into these systems that stems not only from the flexible bodies of human operators or from practical know how that can't be deduced from the knowledge base of an ASI that might be built.
• The challenge would be rendered more complex for an ASI if it were not running on general-purpose hardware but special-purpose circuitry that's much harder to maintain and replace. It may additionally be a more complex task if the ASI could not gain access to its own source code (or relevant parts of it), since that presumably would make a migration onto other infrastructure considerably more difficult, though I'm not fully certain that's actually the case, given that the compiled and operational code may be sufficient for an ASI to deduce weights and other relevant aspects.
• Evolution presumably started from very limited organic chemistry and discovered autocatalytic cycles based on biochemistry, catalytically active macromolecules and compartmentalized cells. That most likely implies that a single cell may be able to repopulate an entire planet that is sufficiently earth-like and give rise to intelligence again after billions of years. That fact alone certainly does not imply that thinking sand needs to build hypothetical nanomachinery to win the battle against entropy over a long period of time. Existing actuators and chips on the planet, the hypothetical absence of humans, and a HLAI or ASI moderately above it may be sufficient in my current opinion.

Very interesting! Happy to have a chat about this / possible collaboration.

That is a surprisingly excellent idea.

The basic idea was simple: if hardware suffers more transient failures as it gets smaller, why not allow software to detect erroneous computations and re-execute them? This idea seemed promising until John realized THAT IT WAS THE WORST IDEA EVER. Modern software barely works when the hardware is correct, so relying on software to correct hardware errors is like asking Godzilla to prevent Mega-Godzilla from terrorizing Japan. THIS DOES NOT LEAD TO RISING PROPERTY VALUES IN TOKYO.

Having worked at Google for several years, they are legendary masters of "allow software to detect erroneous computations and re-execute them", they execute it with great skill at enormous scale, and have very conclusively demonstrated that it does in fact lead to rising property values (at least in Silicon Valley and everywhere else they have research offices or a datacenter). However, their internal name for it is 'Borg', not 'Godzilla'. [Please note that this is a criticism of James Mickens book, not johnswentworth's excellent post.]