LessWrong 2.0 Reader

1. Will the first transformative AIs be FMAs?

Probably, but not certainly. I'd be very curious to get a survey of people who've really thought about this. People who are sure they won't give reasons I find highly dubious. At the least it seems likely enough that we should be thinking about aligning them in more detail, because we can see their general shape better than other possible first AGIs

2. Will narrow FMAs for a variety of specific domains be transformatively useful before we get transformatively useful general FMAs?

No. There are advantages to creating FMAs for specific domains, but there are also very large advantages to working on general reasoning. Humans are not limited to narrow domains, but can learn but anything through instruction or self-instruction. Language models trained on human "thought" can do the same as soon as they have any sort of useful persistent memory. Existing memory systems don't work well, but they will be improved, probably rapidly. 

3. If FMAs are the first transformative AIs [? · GW] (TAIs), how long will FMAs remain the leading paradigm?

This is a really important question. I really hope they remain the leading paradigm long enough to become useful in aligning other types of AGI. And that they remain free of goal-directed RL adequately to remain alignable.

I don't agree with any of this. When I was really into lucid dreaming, I discovered that the best approach is two-fold: keep a detailed dream journal, and make a habit of performing reality checks. That's it. If you don't keep a dream journal, you'll likely have lucid dreams and just ... forget about them. And as for reality checks, my preferred one is trying to push my thumb through my palm. You can do it casually anywhere and it's an instant confirmation.

When I was actively trying to induce them, I often had periods where I had several lucid dreams per night. Gradually, I mastered dream flight (it's so weird how it's a skill), and I became better able to maintain my lucid state, which is often the most tricky part.

I was never interested in erotic, oneironautic adventures. I spent most of my time flying, which doesn't really get old.

In your climate, defection from the natural gas and electric grid is very far from being economical, because the peak energy demand for the year is dominated by heating, and solar peaks in the summer, so you would need to have extreme oversizing of the panels to provide sufficient energy in the winter.

I think the prediction here is that people will detach only from the electric grid, not from the natural gas grid. If you use natural gas heat instead of a heat pump for part of the winter, then you don't need to oversize your solar panels as much.

I tend to disagree, although I could be wrong. My epistemic status for the claim is at 65% ;). I have not spent as much time on lesswrong so it wouldn't surprise me. 

Before posting that comment I double checked with this article [EA · GW].  Here is a quote which the author got from urban dictionary (the other says the definition is accurate):

The epistemic status is a short disclaimer at the top of a post that explains how confident the author is in the contents of the post, how much reputation the author is willing to stake on it, what sorts of tests the thesis has passed.

It should give a reader a sense of how seriously they should take the post.

I do agree with you that people do not typically list probabilities when listing their epistemic status, although I have seen at least several people list probabilities in their epistemic status section. It is not unheard of, and I still think it is good practice for especially important claims. 

You are not wrong that the epistemic status often includes reasons for claims, but an epistemic status usually includes the level of confidence in a claim. 

Of course, it's just a definition, so it's not the end of the world if we disagree over it. It's not a crux 🤷‍♂️. 

If you set aside the pricing structure and just look at the underlying economics, the power grid will still be definitely needed for all the loads that are too dense for rooftop solar, ie industry, car chargers, office buildings, apartment buildings, and some commercial buildings. If every suburban house detached from the grid, these consumers would see big increases in their transmission costs, but they wouldn't have much choice but to pay them. This might lead to a world where downtown areas and cities have electric grids, but rural areas and the sparser parts of suburbs don't.

There's an additional backup-power option not mentioned here, which is that some electric cars can feed their battery back to a house. So if there's a long string of cloudy days but the roads are still usable, you can transport power from the grid to an off-grid house by charging at a public charger, and discharging at home. This might be a better option than a natural-gas generator, especially if it only comes up rarely.

If rural areas switch to a regime where everyone has solar+batteries, and the power grid only reaches downtown and industrial areas... that actually seems like it might just be optimal? The price of disributed generation and storage falls over time, but the cost of power lines doesn't, so there should be a crossover point somewhere where the power lines aren't worth it. Maybe net-metering will cause the switchover to happen too soon, but it does seem like a switchover should happen eventually.

True if you don't count the training process as part of the optimizer (which is a choice that sometimes makes sense and sometimes doesn't). If you count the training process as part of the optimizer, then you can of course just flip your loss function or RL signal most of the time.

Seems like if you're working with neural networks there's not a simple map from an efficient (in terms of program size, working memory, and speed) optimizer which maximizes X to an equivalent optimizer which maximizes -X. If we consider that an efficient optimizer does something like tree search, then it would be easy to flip the sign of the node-evaluating "prune" module. But the "babble" module is likely to select promising actions based on a big bag of heuristics which aren't easily flipped. Moreover, flipping a heuristic which upweights a small subset of outputs which lead to X doesn't lead to a new heuristic which upweights a small subset of outputs which lead to -X. Generalizing, this means that if you have access to maximizers for X, Y, Z, you can easily construct a maximizer for e.g. 0.3X+0.6Y+0.1Z but it would be non-trivial to construct a maximizer for 0.2X-0.5Y-0.3Z. This might mean that a certain class of mesa-optimizers (those which arise spontaneously as a result of training an AI to predict the behaviour of other optimizers) are likely to lie within a fairly narrow range of utility functions.

To nitpick on your nitpick, in the US, 1000x safer would be 42 deaths yearly. https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in_U.S._by_year

For the whole world, it would just be above 1k. https://en.wikipedia.org/wiki/List_of_countries_by_traffic-related_death_rate#List, but 2032 seems like an ambitious deadline for that.

In addition, it does seem against the spirit of the question to resolve positively solely because of reducing traffic deaths.

Just wanted to mention that if anyone liked my submissions (3rd prize,  An Overview of “Obvious” Approaches to Training Wise AI Advisors, Some Preliminary Notes on the Promise of a Wisdom Explosion), 
I'll be running a project related to this work as part of AI Safety Camp.

Isn't it basically a policy choice over there to require net metering and thus make it not economical to get your power over the grid?

Many African countries like Nigeria have a problem that nobody builds power plants and an electric grid because there are laws that forbid that limit businesses to provide that service in a profitable way. 

It seems like this would mean that Puerto Rico is going to move into having third-world country energy reliability and a requirement for everyone to deal with buying their own generators like people in Nigeria have to do while in total paying more for energy than they would have to pay if energy would be provided in a more centralized fashion.