LessWrong 2.0 Reader

Richard Carrier has written an entire rebuttal of the argument on his blog. He always answers comments under his posts, so if you disagree with some part of the rebuttal, you can just leave a comment there explaining which claim makes no sense to you. Then, he will usually defend the claim in question or provide the necessary clarification.

Oh I meant a (2, 0) tensor.

By "possible worlds" I mean all worlds that are consistent with laws of logic, such as the law of non-contradiction.

For example, it might be the case that for some reason alignment would only have been solved if and only if Abraham Lincoln wasn't assasinated in 1865. That means that humans in 2024 in our world will not be able solve alignment, despite it being solvable in principle.

My answer is kind of similar to @quila [LW · GW]'s. I think that he means roughly the same thing by "space of possible mathematical things."

I don't think that my definition of alignment is particularly important here, because I was mostly clarifying the semantic part of the sentence, as opposed to defending that claim. Alignment is a broad word and I don't really have the authority to choose one specific meaning.

For example, one article managed to find six distinct interpretations of the word:

P1: Avoiding takeover from emergent optimization in AI agents

P2: Ensuring that AI’s information processing (and/or reasoning) is intelligible to us

P3: Ensuring AIs are good at solving problems as specified (by user or designer)

P4: Ensuring AI systems enhance, and don’t erode, human agency

P5: Ensuring that advanced AI agents learn a human utility function

P6: Ensuring that AI systems lead to desirable systemic and long term outcomes

Thanks a lot for writing this! Some clarifying questions:

• In this context, is QFT roughly a shorthand for "statistical field theory, studied via the mathematical methods of Euclidean QFT"? Or do you expect intuitions from specifically quantum phenomena to play a role?
• There is a community of statistical physicists who use techniques from statistical mechanics of disordered systems and phase transitions to study ML theory, mostly for simple systems (linear models, shallow networks) and simple data distributions (Gaussian data, student-teacher model with a similarly simple teacher). What do you think of this approach? How does it relate to what you have in mind?
• Would this approach, at least when applied to the whole network, rely on an assumption that trained DNNs inherit from their initialization a relatively high level of "homogeneity" and relatively limited differentiation, compared say to biological organisms? For instance, as a silly thought experiment, suppose you had the same view into a tiger as you have a DNN: something like all the chemical-level data as a collection of time-series indexed by (spatially randomized) voxels, and you want to understand the behaviour of the tiger as function of the environment. How would you expect a QFT-based approach to proceed? What observables would it encoder first? Would it be able to go beyond the global thermodynamics of the tiger and say something about cell and tissue differentiation? How would it "put the tiger back together"? (Those are not gotcha questions - I don't really know if any existing interpretability method would get far in this setting!)

In "The Autodidactic Universe", the authors try to import concepts from machine learning into physics. In particular, they want to construct physical models in which "the Universe learns its own physical laws". In my opinion they're not very successful, but one might wish to see whether their physicalized ML concepts can be put to work back in ML, in the context of a program like yours. 

"Speedrun" projects. Write papers with hypothetical data and decide whether they'd be interesting. If not then move on to something else.

Writing hypothetical paper abstracts has been a good quick way for me to figure out if things would be interesting.

I mean, one of them’s math built bombs and computers & directly influenced pretty much every part of applied math today, and the other one’s math built math. Not saying he wasn’t smart, but no question are bombs & computers more flashy. 

So if we ended up with some advanced AI’s replacing humans, then we made some sort of mistake

Again, I'm glad that we agree on this. I notice you want to do what I consider the right thing, and I appreciate that.

The way I currently envision the “typical” artificial conscience is that it would put a pretty strong conscience weight on not doing what its user wanted it to do, but this could be over-ruled by the conscience weight of not doing anything to prevent catastrophes. So the defensive, artificial conscience-guard-railed AI I’m thinking of would do the “last resort” things that were necessary to avoid s-risks, x-risks, and major catastrophes from coming to fruition, even if this wasn’t popular with most people, at least up to a point.

I can see the following scenario occur: the AI, with its AC, decided rightly that a pivotal act needs to be undertaken to avoid xrisk (or srisk). However, the public mostly doesn't recognize the existence of such risks. The AI will proceed sabotaging people's unsafe AI projects against public will. What happens now is: the public gets absolutely livid at the AI, that is subverting human power by acting against human will. Almost all humans team up to try to shut down the AI. The AI recognizes (and had already recognized) that if it looses, humans risk going extinct, so it fights this war against humanity and wins. I think in this scenario, an AI, even one with artificial conscience, could become the most hated thing on the planet.

I think people underestimate the amount of pushback we're going to get once you get into pivotal act territory. That's why I think it's hugely preferred to go the democratic route and not count on AI taking unilateral actions, even if it would be smarter or even wiser, whatever that might mean exactly.

All that said, if we could somehow pause development of autonomous AI’s everywhere around the world until humans got their act together, developing their own consciences and senses of ethics, and were working as one team to cautiously take the next steps forward with AI, that would be great.

So yes definitely agree with this. I don't think lack of conscience or ethics is the issue though, but existential risk awareness.

I for one warmly welcome Drunk Edward Tufte to LW!

Data scientist (frowning) “this here are the culture war edgelords, these (gestures) are people who repost memes wantonly, but THESE (points at much taller larger treemap cell, on a log scale) are normies who had a bad breakup and said something like “f* men”. Oh, and these guys over here are sex positive and used the same words while meaning more or less the opposite”