LessWrong 2.0 Reader

What do you mean you’re encoding/decoding like normal but using the k means vectors? Shouldn’t the SAE training process for a top k SAE with k = 1 find these vectors then? 

In general I’m a bit skeptical that clustering will work as well on larger models, my impression is that most small models have pretty token level features which might be pretty clusterable with k=1, but for larger models many activations may belong to multiple “clusters”, which you need dictionary learning for. 

I think you're assuming these minds are more similar to human minds than they necessarily are. My point is that there's three cases wrt alignment here.

1. The AI is robustly aligned with humans
2. The AI has a bunch of other goals, but cares about humans to some degree, but only to the extent that humans give them freedom and are nice to it, but still to a large enough extent that even as it becomes smarter / ends up in a radically OOD distribution, will care for those humans.
3. The AI is misaligned (think scheming paperclipper)

In the first we're fine, even if we negate the AIs freedom, in the third we're screwed no matter how nicely we treat the AI, only in the second do your concerns matter.

But, the second is 1) at least as complicated as the first and third 2) disincentivized by training dynamics and 3) its not something were even aiming for with current alignment attempts.

These make it very unlikely. You've put up an example that tries to make it seem less unlikely, like saying you value your parents for their own sake but would stop valuing them if you discovered they were conspiring against you.

However, the reason this example is realistic in your case very much hinges on specifics of your own psychology and values, which are equally unlikely to appear in AIs, for more or less the reasons I gave. I mean, you'll see shadows of them, because at least pretraining is on organic text written by humans, but these are not the values were aiming for when we're trying to align AIs. And if our alignment effort fails, and what ends up in the terminal values of our AIs are a haphazard collection of the stuff found in the pretraining text, we're screwed anyways.

Agree with Dagon here, when omnizoid say's "Its obvious that you should" they are calling on the rules of their own morality. Its similar with "Her suffering is bad", that's a direct moral judgment. Both statements fall apart when you consider that someone may have different moral rules than you.

For example, in NZ we have an issue with deer destroying our native bush which in turn hurts our native birds. Deer are considered an invasive species and are actively eradicated. In the case when you are actively in the presence of a hurting deer empathy drives you to help, suffering is not pleasant to witness. However I suspect that many NZ's would condemn every deer in NZ to a painful death, as long as they didn't have to witness it, in order to save our trees and birdlife.

Is there a benchmark in which SAEs clearly, definitely outperform standard techniques?

I don't know how to avoid ASI killing us. However, when I try to imagine worlds in which humanity isn't immediately destroyed by ASI, humanity's success can often be traced back to some bottleneck in the ASI's capabilities.

For example, Eliezer's list of lethalities [LW · GW]point 35 argues that "Schemes for playing "different" AIs off against each other stop working if those AIs advance to the point of being able to coordinate via reasoning about (probability distributions over) each others' code." because "Any system of sufficiently intelligent agents can probably behave as a single agent, even if you imagine you're playing them against each other." Note that he says "probably" (boldface mine).

In a world there humanity wasn't immediately destroyed by ASI, I find it plausible (let's say 10%) that something like Arrow's impossibility theorem exists for coordination. And that we were able to exploit that to successfully pit different AIs against each other.

Of course you may argue that "10% of worlds not immediately destroyed by ASI" is a tiny slice of probability space. And that even in those worlds, the ability to pit AIs against each other is not sufficient. And you may disagree that the scenario is plausible. However, I hope I explained why I believe the idea of exploiting ASI limitations is a step in the right direction.

I feel like my post appears overly dramatic; I'm not very surprised and don't consider this the strongest evidence against SAEs. It's an experiment I ran a while ago and it hasn't changed my (somewhat SAE-sceptic) stance much.

But this is me having seen a bunch of other weird SAE behaviours (pre-activation distributions are not the way you'd expect from the superposition hypothesis h/t @jake_mendel [LW · GW], if you feed SAE-reconstructed activations back into the encoder the SAE goes nuts, stuff mentioned in recent Apollo papers, ...).

Reasons this could be less concerning that it looks

• Activation reconstruction isn't that important: Clustering is a strong optimiser -- if you fill a space with 16k clusters maybe 90% reconstruction isn't that surprising. I should really run a random Gaussian data baseline for this.
• End-to-end loss is more important, and maybe SAEs perform much better when you consider end-to-end reconstruction loss.
• This isn't the only evidence in favour of SAEs, they also kinda work for steering/probing (though [AF · GW] pretty badly).

Your text is full of saliently negative things in the lives of wild animals, plus big numbers (since there are so many natural lives), but I don't see any consideration of balancing goods linked to similarly large numbers.

Fundamentally, you don't seem to be tracking the possibility that many wild animal lives are "lives worth living", and that the balance of lives that were not worth living (and surely some of those exist) might still be overbalanced by lives that were worth living.

Maybe this wouldn't matter very much to not track, but it is the default presumption of most people that life is worth living, and that lives worth living are often very good, and very worth living, such that they might redeem net negative lives on net and make "life in general at all" a net positive.

Pollywogs (the larval form of frogs, after eggs, and before growing legs) are an example where huge numbers of them are produced, and many die before they ever grow into frogs, but from their perspective, they probably have many many minutes of happy growth, having been born into a time and place where quick growth is easy: watery and full of food! The spring! <3

They swim around, they exercise, they eat, they stay near others in a school, they get bigger. Granted, they also hide from predators, but most of the time when they die it is generally very fast and over soon (such as when a camouflaged fish ambushes them as they swim obliviously right past the mouth the of the predator, or a heron's beak spears out of the sky with no warning). The vast majority of their moments are positive, and in most cases they barely even cognize the sadness at the end.

Even living only a few weeks as a pollywog would grant over a million seconds of happy life, compared to a handful of seconds during which death is imminent, but hasn't happened yet.

The balance seems clearly positive to me, because the goodness of normal moments of a life (even the life of a mere pollywog) full of growth and development are, in fact, a positive good.

Like it really does seem that the balance here, of bad-life-moments vs good-life-moments, might be on the order of "one per million" even for a pollywog that never grows up to be a frog.

Haven't you ever seen a vernal pool full of pollywogs, swimming around all lazy-like, enjoying themselves in the sun and the water? Maybe my childhood was unusually good, to have been full of time in nature, but even so, you've probably seen some happy animals in nature? Perhaps sea birds at the beach, or fish nibbling water plants near a dock? Perhaps squirrels playing in a back yard, or deer browsing in a field, or vultures doing lazy circles in an updraft?

Most of the animal-life-moments I've seen in nature have exhibited a profound "OK-ness". Just chilling. Just being. Just finding some food. Just happy to exist. Each such moment "the score" was going UP! In any well-contemplated life, such moments should be on the positive side of the ledger!

Certainly when "life itself" becomes questionable for animals in nature, they seem to do everything they can to keep their life... escaping danger, seeking safety, eating food (rather than engaging in a hunger strike or trying to forgo water and thirst to death to bring their own end on faster). Every indication we have is that animals are NOT suicidal, but rather they seem to "net value" their own lives. Presumably because these are good lives according to their own subjectivities.

I feel as though your text is some kind of exhibition of nearly systematic failures to detect and weigh such goodness properly.

Even by your own confused standards it seems that you should surely be speaking against the badness of factory farming, not the badness of wild nature? Maybe factory farmed lives might truly be "not worth living", and much more clearly subject to choices by consumers, farmers, and policy makers? Nearly every salient feature of factory farming makes it more worth calling attention to try to fix what is bad about it.

I wonder if you're engaged in some sort of cry for help here? Your advocacy here seems wildly out of balance with almost any coherent theory of justice, reason, or the optimized pursuit of the melioration of badness in the world, such that I wonder if the brokenness of your advocacy is itself the thing you're hoping to demonstrate in public before a large audience, to get helpful corrective signals regarding whatever is going wrong in the part of you that generated this essay?

It would be consistent with "being unusually unaware of the good parts of normal animal lives" if you were also unaware of such moments in your own life :-(

I wonder if you're in a depression, perhaps? It is early February and the northern hemisphere is reaching the end of winter and maybe you have SAD? Could you prescribe dancing, and eating homemade desserts with friends, and watching Moanna, and snuggling up in an electric blanket while listening to music to yourself, and practice noticing the texture of all these various good experiences, and remember how easy it would be to spend lots of your time hedonically wallowing in such pleasures, then... uh... recalculate whether "life in general" seems good or bad to you?

Without commenting on any strategic astronomy and neurology, it is worth noting that "bias", at least, is a major concern of the new administration (e.g., the Republican chair of the House Financial Services Committee is actually extremely worried about algorithmic bias being used for housing and financial discrimination and has given speeches about this).  

this seems concerning. Can somebody ELI5 what's going on here?

I guess I somewhat agree with this, but I've also seen many examples of regulations that were passed in response to a particular incident decades ago, whose other non-incident related harms were completely ignored.

Compliance costs are real, and my experience dealing with the federal bureaucracy is that they're often completely ignored.