LessWrong 2.0 Reader

I typically consume my greens with ground flax seeds in a smoothie.

I feel very confident that adding refined oil to vegetables shouldn't be considered healthy, in the sense that the opportunity cost of 1 Tablespoon of olive oil is 120 calories, which is over a pound of spinach for example. Certainly it's difficult to eat that much spinach and it's probably unwise, but I just say that to illustrate that you can get a lot more nutrition from 120 calories than the oil will be adding, even if it makes the greens more bioavailable.

That said "healthy" is a complicated concept. If adding some oil to greens helps something eat greens they otherwise wouldn't eat for example, that's great.

For the first point, there's also the question of whether 'slightly superhuman' intelligences would actually fit any of our intuitions about ASI or not. There's a bit of an assumption in that we jump headfirst into recursive self-improvement at some point, but if that has diminishing returns, we happen to hit a plateau a bit over human, and it still has notable costs to train, host and run, the impact could still be limited to something not much unlike giving a random set of especially intelligent expert humans the specific powers of the AI system. Additionally, if we happen to set regulations on computation somewhere that allows training of slightly superhuman AIs and not past it ...

Those are definitely systems that are easier to negotiate with, or even consider as agents in a negotiation. There's also a desire specifically not to build them, which might lead to systems with an architecture that isn't like that, but still implementing sentience in some manner. And the potential complication of multiple parts and specific applications a tool-oriented system is likely to be in - it'd be very odd if we decided the language processing center of our own brain was independently sentient/sapient separate from the rest of it, and we should resent its exploitation.

I do think the drive/just a thing it does we're pointing at with 'what the model just does' is distinct from goals as they're traditionally imagined, and indeed I was picturing something more instinctual and automatic than deliberate. In a general sense, though, there is an objective that's being optimized for (predicting the data, whatever that is, generally without losing too much predictive power on other data the trainer doesn't want to lose prediction on).

I wonder what he would have thought was the downside of worshiping a longer list of things...

For the things mentioned, it feels like he thinks "if you worship X then the absence of X will be constantly salient to you in most moments of your life".

It seems like he claims that worshiping some version of Goodness won't eat you alive, but in my experiments with that, I've found that generic Goodness Entities are usually hungry for martyrs, and almost literally try to get would-be saints to "give their all" (in some sense "eating" them). As near as I can tell, it is an unkindness to exhort the rare sort of person who is actually self-editing and scrupulous enough to even understand or apply the injunction in that direction without combining it with an injunction that success in this direction will lead to altruistic self harm unless you make the demands of Goodness "compact [LW · GW]" in some way.

Zvi mentions ethics explicitly so I'm pretty sure readings of this sort are "intended". So consider (IF you've decided to try to worship an ethical entity) that one should eventually get ready to follow Zvi's advice in "Out To Get You" for formalized/externalized ethics itself so you can enforce some boundaries on whatever angel you summon (and remember, demons usually claim to be angels (and in the current zeitgeist it is SO WEIRD that so many "scientific rationalists" believe in demons [LW · GW] without believing in angels as well)).

Anyway. Compactification (which is possibly the same thing as "converting dangerous utility functions into safe formulas for satisficing"):

Get Compact when you find a rule you can follow that makes it Worth It to Get Got.

The rule must create an acceptable max loss. A well-chosen rule transforms Out to Get You for a lot into Out to Get You for a price you find Worth It. You then Get Got.

This works best using a natural point beyond which lies clear diminishing returns. If no such point exists, be suspicious.

A simple way is a budget. Spend at most $25,000 on this car, or $5,000 on this vacation package. This creates an obvious max dollar loss.

Many budgets should be $0. Example: free to play games. Either it’s worth playing for free or it isn’t. It isn’t.

The downside of budgets is often spending exactly your maximum, especially if others figure out what it is. Do your best to avoid this. Known bug.

An alternative is restriction on type. Go to a restaurant and avoid alcohol, desert and appetizers. Pay in-game only for full game unlocks and storage space.

Budgets can be set for each purchase. Hybrid approaches are good.

Many cap their charitable giving at 10%. Even those giving more reserve some amount for themselves. Same principle.

For other activities, max loss is about time. Again, you can use a (time) budget or limit your actions in a way that restricts (time) spent, or combine both.

Time limits are crude but effective. Limiting yourself to an hour of television or social media per day maxes loss at an hour. This risks making you value the activity more. Often time budgets get exactly spent same as dollar budgets. Try to let unspent time roll over into future periods, to avoid fear or ‘losing’ unspent time.

When time is the limiting factor, it is better where possible to engineer your environment and options to make the activity compact. You’ll get more out of the time you do spend and avoid feeling like you’re arbitrarily cutting yourself off.

Decide what’s worth watching. Watch that.

For Facebook, classify a handful of people See First. See their posts. No others. Look at social media only on computers. Don’t comment. Or post.

A buffet creates overeating. Filling up one plate (or one early to explore, then one to exploit) ends better.

Unlimited often requires limitation.

Outside demands follow the pattern. To make explanation and justification easier, choose good enough rules that sound natural, simple and reasonable.

Experiments need a chance, but also a known point where you can know to call it quits. Ask whether you can get a definitive negative result in reasonable time. Will I worry I did it wrong? Will others claim or assume I did it wrong or didn’t give it a fair chance?

For myself, I have so far found it much easier to worship wisdom than pure benevolence.

Noticing ways that I am a fool is kinda funny. There are a lot of them! So many that patching each such gap would be an endless exercise! The wise thing, of course, would be to prioritize which foolishnesses are most prudent to patch, at which times. A nice thing here is that wisdom basically assimilates all valid criticism as helpful, and often leads to teaching unskilled critics to criticize better, and this seems to make "living in the water" more pleasant (at least in my experience so far).

The axioms of U are recursively enumerable. You run all M(i,j) in parallel and output a new axiom whenever one halts. That's enough to computably check a proof if the proof specifies the indices of all axioms used in the recursive enumeration.

I am perhaps not speaking as precisely as I should be. I appreciate your comments.

I believe it's correct to say that if you consider all of the food/energy we consumed in the past 50+ million years, it's virtually all plants.

The past 2-2.5 million years had us introducing more animal products to greater or lesser extents. Some were able to subsist on mostly animal products. Some consumed them very rarely.

In that sense it is a relatively recent introduction. My main point is that given our evolutionary history, the idea that plants would be healthier for us than animal products when we have both in abundance, and the idea that plants are more suitable to maintaining health long past reproductive age, aren't immediately/obviously unreasonable ideas.

I think your intuitions are generally correct, and as I say, it's usually a good heuristic to avoid overly processed food. In the absence of other evidence, if you're in a food market where everything is edible, you should probably opt for the less processed option. I also don't disagree with it playing a role in national health guidelines.

But it's a very imprecise heuristic, and I think LessWrong-ers with aspirations to understand the world more accurately should feel a bit uncomfortable with it, especially when benign and beneficial processes are lumped together with those with much clearer mechanisms for harm. 

performance gap of trans women over women

The post is about the performance gap of trans women over men, not women.

I guess (but don't know) that most people who downvote Garrett's comment overupdated on intuitive explanations of singular learning theory, not realizing that entire books with novel and nontrivial mathematical theory have been written on it. 

Maybe by the time we cotton on properly, they're somewhere past us at the top end.

Great point. I agree that there are lots of possible futures where that happens. I'm imagining a couple of possible cases where this would matter:

1. Humanity decides to stop AI capabilities development or slow it way down, so we have sub-ASI systems for a long time (which could be at various levels of intelligence, from current to ~human). I'm not too optimistic about this happening, but there's certainly been a lot of increasing AI governance momentum in the last year.
2. Alignment is sufficiently solved that even > AGI systems are under our control. On many alignment approaches, this wouldn't necessarily mean that those systems' preferences were taken into account.

We can't "just ask" an LLM about its interests and expect the answer to soundly reflect its actual interests.

I agree entirely. I'm imagining (though I could sure be wrong!) that any future systems which were sentient would be ones that had something more like a coherent, persistent identity, and were trying to achieve goals.

LLMs specifically have a 'drive' to generate reasonable-sounding text

(not very important to the discussion, feel free to ignore, but) I would quibble with this. In my view LLMs aren't well-modeled as having goals or drives. Instead, generating distributions over tokens is just something they do in a fairly straightforward way because of how they've been shaped (in fact the only thing they do or can do), and producing reasonable text is an artifact of how we choose to use them (ie picking a likely output, adding it onto the context, and running it again). Simulacra like the assistant character can be reasonably viewed (to a limited degree) as being goal-ish, but I think the network itself can't.

That may be overly pedantic, and I don't feel like I'm articulating it very well, but the distinction seems useful to me since some other types of AI are well-modeled as having goals or drives.

Newton's Universal Law of Gravitation was the first highly accurate model of things falling down that generalized beyond the earth, and it is also the second-most computationally applicable model of things falling down that we have today.

Are you saying that singular learning theory was the first highly accurate model of breadth of optima, and that it's one of the most computationally applicable ones we have?