LessWrong 2.0 Reader

Welcome!

The short and informal version is that epistemics covers all the stuff surrounding the direct claims. Things like credence levels, confidence intervals, probability estimates, etc are the clearest indicators. It also includes questions like where the information came from, how it is combined with other information, what other information we would like to have but don't, etc.

The most popular way you'll see this expressed on LessWrong is through Bayesian probability estimates and a description of the model (which is to say the writer's beliefs about what causes what).

The epistemic status statement you see at the top of a lot of posts is for setting the expectation. This lets the OP write complete thoughts without the expectation that they demonstrate full epistemic rigor, or even that they endorse the thought per se.

Other proponents of the bill (longform, 1-3h)

[...]

Charles Foster

Note: I wouldn't personally call myself a proponent, but I'm fine with Michaël putting me in that bucket for the sake of this post.

That's common for beginners. If you want to give this a go, you should start by writing down fleeting, vague associations. "Something a bit sad or disappointing. A car. School and also not school. The texture of cinnamon rolls."

It doesn't matter that you can't remember anything concrete at first. Eventually, you'll remember more and more.

Takes on a few more important questions:

Should safety-focused people support the advancement of FMA capabilities?

Probably. The advantages of a system without goal-directed RL (RL is used, but only to get the "oracle" to answer questions as the user intended them) and with a legible train-of-thought seem immense. I don't see how we close the floodgates of AGI development now. Given that we're getting AGI, it really seems like our best bet is FMA AGI.

But I'm not ready to help anyone develop AGI until this route to alignment and survival has been more thoroughly worked through in the abstract. I really wish more alignment skeptics would engage with specific plans isntead of just pointing to general arguments about how alignment would be difficult, some of which don't apply to the ways we'd really align FMAs (see my other comment on this post). We may be getting close; Shut It All Down isn't a viable option AFAICT so we need to get together our best shot.

1. Will the first transformative AIs be FMAs?

Probably, but not certainly. I'd be very curious to get a survey of people who've really thought about this. People who are sure they won't give reasons I find highly dubious. At the least it seems likely enough that we should be thinking about aligning them in more detail, because we can see their general shape better than other possible first AGIs

2. Will narrow FMAs for a variety of specific domains be transformatively useful before we get transformatively useful general FMAs?

No. There are advantages to creating FMAs for specific domains, but there are also very large advantages to working on general reasoning. Humans are not limited to narrow domains, but can learn but anything through instruction or self-instruction. Language models trained on human "thought" can do the same as soon as they have any sort of useful persistent memory. Existing memory systems don't work well, but they will be improved, probably rapidly. 

3. If FMAs are the first transformative AIs [? · GW] (TAIs), how long will FMAs remain the leading paradigm?

This is a really important question. I really hope they remain the leading paradigm long enough to become useful in aligning other types of AGI. And that they remain free of goal-directed RL adequately to remain alignable.

I don't agree with any of this. When I was really into lucid dreaming, I discovered that the best approach is two-fold: keep a detailed dream journal, and make a habit of performing reality checks. That's it. If you don't keep a dream journal, you'll likely have lucid dreams and just ... forget about them. And as for reality checks, my preferred one is trying to push my thumb through my palm. You can do it casually anywhere and it's an instant confirmation.

When I was actively trying to induce them, I often had periods where I had several lucid dreams per night. Gradually, I mastered dream flight (it's so weird how it's a skill), and I became better able to maintain my lucid state, which is often the most tricky part.

I was never interested in erotic, oneironautic adventures. I spent most of my time flying, which doesn't really get old.

In your climate, defection from the natural gas and electric grid is very far from being economical, because the peak energy demand for the year is dominated by heating, and solar peaks in the summer, so you would need to have extreme oversizing of the panels to provide sufficient energy in the winter.

I think the prediction here is that people will detach only from the electric grid, not from the natural gas grid. If you use natural gas heat instead of a heat pump for part of the winter, then you don't need to oversize your solar panels as much.

I tend to disagree, although I could be wrong. My epistemic status for the claim is at 65% ;). I have not spent as much time on lesswrong so it wouldn't surprise me. 

Before posting that comment I double checked with this article [EA · GW].  Here is a quote which the author got from urban dictionary (the author said the definition is accurate, and from my experience I agreed):

The epistemic status is a short disclaimer at the top of a post that explains how confident the author is in the contents of the post, how much reputation the author is willing to stake on it, what sorts of tests the thesis has passed.

It should give a reader a sense of how seriously they should take the post.

I do agree with you that people do not typically list probabilities when listing their epistemic status, although I have seen at least several people list probabilities in their epistemic status section. It is not unheard of, and I still think it is good practice for especially important claims. 

You are not wrong that the epistemic status often includes reasons for claims, but an epistemic status usually includes the level of confidence in a claim. 

Of course, it's just a definition, so it's not the end of the world if we disagree over it. It's not a crux 🤷‍♂️. 

If you set aside the pricing structure and just look at the underlying economics, the power grid will still be definitely needed for all the loads that are too dense for rooftop solar, ie industry, car chargers, office buildings, apartment buildings, and some commercial buildings. If every suburban house detached from the grid, these consumers would see big increases in their transmission costs, but they wouldn't have much choice but to pay them. This might lead to a world where downtown areas and cities have electric grids, but rural areas and the sparser parts of suburbs don't.

There's an additional backup-power option not mentioned here, which is that some electric cars can feed their battery back to a house. So if there's a long string of cloudy days but the roads are still usable, you can transport power from the grid to an off-grid house by charging at a public charger, and discharging at home. This might be a better option than a natural-gas generator, especially if it only comes up rarely.

If rural areas switch to a regime where everyone has solar+batteries, and the power grid only reaches downtown and industrial areas... that actually seems like it might just be optimal? The price of disributed generation and storage falls over time, but the cost of power lines doesn't, so there should be a crossover point somewhere where the power lines aren't worth it. Maybe net-metering will cause the switchover to happen too soon, but it does seem like a switchover should happen eventually.

True if you don't count the training process as part of the optimizer (which is a choice that sometimes makes sense and sometimes doesn't). If you count the training process as part of the optimizer, then you can of course just flip your loss function or RL signal most of the time.

Seems like if you're working with neural networks there's not a simple map from an efficient (in terms of program size, working memory, and speed) optimizer which maximizes X to an equivalent optimizer which maximizes -X. If we consider that an efficient optimizer does something like tree search, then it would be easy to flip the sign of the node-evaluating "prune" module. But the "babble" module is likely to select promising actions based on a big bag of heuristics which aren't easily flipped. Moreover, flipping a heuristic which upweights a small subset of outputs which lead to X doesn't lead to a new heuristic which upweights a small subset of outputs which lead to -X. Generalizing, this means that if you have access to maximizers for X, Y, Z, you can easily construct a maximizer for e.g. 0.3X+0.6Y+0.1Z but it would be non-trivial to construct a maximizer for 0.2X-0.5Y-0.3Z. This might mean that a certain class of mesa-optimizers (those which arise spontaneously as a result of training an AI to predict the behaviour of other optimizers) are likely to lie within a fairly narrow range of utility functions.