LessWrong 2.0 Reader

Counter-counterpoint: big groups like bureaucracies are not composed of randomly selected individuals from their respective countries. I strongly doubt that say, 100 randomly selected Google employees (the largest plausible bureaucracy that might potentially develop AGI in the very near term future?) would answer extremely similarly to 100 randomly selected Americans.

Of course, in the only moderately near term or median future, something like a Manhatten Project for AI could produce an AGI. This would still not be identical to 100 random Americans, but averaging across the US security & intelligence apparatus, the current political facing portion of the US executive administration, and the leadership + relevant employee influence from a (mandatory?) collaboration of US frontier labs would be significantly closer on average. I think it would at least be closer to average Americans than a CCP Centralized AGI Project would be to average Chinese people, although I admit I'm not very knowledgeable on the gap between Chinese leadership and average Chinese people other than basics like (somewhat) widespread VPN usage.

One factor is how compute-constrained capabilities research is compared to safety research. 

There's been lots of ink spilled recently on the degree to which capabilities research is compute-bottlenecked in a scenario where you have a surge of AI research labor caused by automated AI researchers, but everyone agrees that capability research requires a ton of compute. For safety research, I don't feel well informed about how important compute is, but I think a lot of safety research being done right now is not very compute-intensive. I'm not aware off the top of my head of safety work that involve doing experiments approaching the scale of training a frontier model. So you could have a scenario where the large majority of resources is going towards capabilities, but the big increase in safety labor in absolute terms differentially helps safety.

I use LLMs daily yet I still am not sure they really help all that much with the core productivity bottlenecks. I worry they lower the barrier to excessive perfectionism and “vibe coding” or “vibe learning.” They seem to short-circuit the theory-practice gap by giving users instant but unreliable and often inextensible results.

My fear is that they’ll raise expectations about productivity gains (because AI-assisted workers can bring immediate results in more quickly to a higher apparent standard of polish), while drastically reducing the knowledge gain by the workers about the problem domain. For example, workers may be able to whip up a codebase more quickly but have less familiarity with it at the end of the process, making it much more difficult to make modifications efficiently. Essentially, I suspect AI will generate massive technical debt in exchange for short-term wins, and that bad incentives will tend to perpetuate this in organizations. People will quickly set up new systems using AI, take credit, and exit those projects before serious problems become apparent.

I agree that there is some ontological mismatch here, but I think your position is still in pretty clear conflict to what Neel said, which is what I was objecting to: 

My understanding is that eg Jaime is sincerely motivated by reducing x risk (though not 100% motivated by it), just disagrees with me (and presumably you) about various empirical questions about how to go about it, what risks are most likely, what timelines are, etc.

"Not 100% motivated by it" IMO sounds like an implication that "being motivated by reducing x-risk would make up something like 30%-70% of the motivation". I don't think that's true, and I think various things that Jaime has said make that relatively clear.

You can explicitly set windowLog with --zstd=windowLog=...

It's sometimes useful to combine low-ish compression level with high window size. E.g. when the input data contains multiple similar large chunks that do not fit into the low-compression-level window.

Interesting find! Thanks for sharing. Curious to see what related training data could be contributing

Been a while since I posted here. https://medium.com/@jeffj4a/personal-agents-will-enable-direct-democracy-9413f5607c15 

Fixed link: https://medium.com/@jeffj4a/personal-agents-will-enable-direct-democracy-9413f5607c15

I've actually been recently thinking about randomness in ML, and I've come to a compelling case for it's specific role. The insights do seem to generalize to all problem-solving mechanisms in a way. I can expand if you want

Sure

I developed this idea here: https://medium.com/@jeffj4a/personal-agents-will-enable-direct-democracy-9413f5607c15. Pretty much the same byline as yours. :-)