LessWrong 2.0 Reader

Thanks, this post made me seriously consider applying to a PhD, and I strong-upvoted. I had vaguely assumed that PhDs take way too long and don't allow enough access to compute compared to industry AI labs. But considering the long lead time required for the application process and the reminder that you can always take new opportunities as they come up, I now think applying is worth it.

However, looking into it, putting together a high-quality application starting now and finishing by the deadline seems approximately impossible? If the deadline were December 15, that would give you two weeks; other schools like Berkeley have even earlier deadlines. I asked ChatGPT how long it would take to apply to just a single school, and it said it would take 43–59 hours of time spent working, or ~4–6 weeks in real time. Claude said 37-55 hours/4-6 weeks.

Not to discourage anyone from starting their application now if they think they can do it - I guess if you're sufficiently productive and agentic and maybe take some amphetamines, you can do anything. But this seems like a pretty crazy timeline. Just the thought of asking someone to write me a recommendation letter in a two-week timeframe makes me feel bad.

Your post does make me think "if I were going to be applying to a PhD next December, what would I want to do now?" That seems pretty clarifying, and would probably be a helpful frame even if it turns out that a better opportunity comes along and I never apply to a PhD.

I think it'd be a good idea for you to repost this in August or early September of next year!

Kinda, my current mainline-doom-case is "some AI gets controlled --> powerful people use it to prop themselves up --> world gets worse until AI gets uncontrollably bad --> doom". I would call it a different yet also-important doom case of "perpetual low-grade-AI dictatorship where the AI is controlled by humans in a surveillance state".

All of these ideas sound awesome and exciting, and precisely the right kind [LW · GW] of use of LLMs that I would like to see on LW!

It's looking like the values of humans are far, far simpler than a lot of evopsych literature and Yudkowsky thought, and related to this, values are less fragile than people thought 15-20 years ago, in the sense that values generalize far better OOD than people used to think 15-20 years ago

I'm not sure I like this argument very much, as it currently stands. It's not that I believe anything you wrote in this paragraph is wrong per se, but more like this misses the mark a bit in terms of framing.

Yudkowsky had (and, AFAICT, still has) a specific theory [LW · GW] of human values in terms of what they mean in a reductionist [LW · GW] framework, where it makes sense (and is rather natural) to think of (approximate) utility functions [LW · GW] of humans and of Coherent Extrapolated Volition [LW · GW] as things-that-exist-in-the-territory [LW · GW]. 

I think a lot of writing and analysis, summarized by me here [LW(p) · GW(p)], has cast a tremendous amount of doubt on the viability of this way of thinking and has revealed what seem to me to be impossible-to-patch holes at the core of these theories. I do not believe [LW(p) · GW(p)] "human values" in the Yudkowskian sense ultimately make sense as a coherent concept that carves reality at the joints [LW · GW]; I instead observe a tremendous number of unanswered questions and apparent contradictions [LW(p) · GW(p)] that throw the entire edifice in disarray.

But supplementing this reorientation of thinking around what it means to satisfy human values has been "prosaic" [LW · GW] alignment researchers pivoting more towards intent alignment [LW · GW] as opposed to doomed-from-the-start paradigms like "learning the true human utility function" [LW · GW] or ambitious value learning [LW · GW], a recognition that realism about (AGI) rationality [LW · GW] is likely just straight-up false and that the very specific set of conclusions MIRI-clustered alignment researchers have reached [LW(p) · GW(p)] about what AGI cognition will be like are entirely overconfident and seem contradicted by our modern observations of LLMs [LW(p) · GW(p)], and ultimately an increased focus on the basic observation that full value alignment simply is not required [LW(p) · GW(p)] for a good AI outcome (or at the very least for prevent AI takeover). So it's not so much that human values (to the extent such a thing makes sense) are simpler, but more so that fulfilling those values is just not needed to as nearly a high a degree as people used to think.

Couldn't you do something like fit a Gaussian to the model's activations, then restrict the steered activations to be high likelihood (low Mahalanobis distance)? Or (almost) equivalently, you could just do a whitening transformation to activation space before you constrain the L2 distance of the perturbation.

(If a gaussian isn't expressive enough you could model the manifold in some other way, eg. with a VAE anomaly detector or mixture of gaussians or whatever)

The theoretical maximum FLOPS of an Earth-bound classical computer is something like $2^{35}$.

Is this supposed to have a different base or exponent? A single H100 already gets like $2^{45}$ FLOP/s.

Ooh.

I lean towards an opt-out system for whole post imports? I'd expect the vast majority of relevant authors to be happy with it, and it would offer less inconvenience to readers. Letting an author easily register as "no whole text imports please" seems worthwhile, and maybe if people aren't happy with that switching to opt-in?

I look forward to seeing some of this integrated into LW!

I spend a fair amount of time writing comments that are probably pretty obvious to experienced LWers. I think this is important for getting newer people on-board with historical ideas and arguments. This includes explaining to enthusiastic newbies why they're being voted into oblivion. Giving them some simulated expert comments would be great.

I think we'd see a large improvement in how useful new user's early posts are, and therefore how much they're encouraged to help vs. turned away by downvotes because they're not really contributing. It would be a great way to let new posters do some efficient due diligence without catching up on the entire distributed history of discussion on their chosen topic.

I think it would also be useful for the most knowledgeable authors to have an LLM with cached context/hidden state from the best LW posts and comment sections giving virtual comments before publication.

I'd love to see whatever you've got going internally as an optional feature (if it doesn't cost too much) rather than wait for a finished, integrated feature.

Sure. So, there are some workplaces have implicit cultural norms that aren't written down but are crucial for career advancement. Always being available and responding to emails quickly might be an unspoken expectation, or participating in after-work social events might not be mandatory but would be noted and count against people looking for promotion. Certain dress codes or communication styles might be rewarded or penalized beyond their actual professional relevance.

In a community, this usually comes as a form of purity testing of some kind, but can also be related to preferences around how you socialize or what you spend your time doing. If you're in a community that thinks sex-work is low status, for example, and you want to ask if that's true... just asking might in fact be costly, because it might clue people in to your potential interest in doing it.

Does that make sense?