LessWrong 2.0 Reader

One and a half years later it seems like AI tools are able to sort of help humans with very rote programming work (e.g. changing or writing code to accomplish a simple goal, implementing versions of things that are well-known to the AI like a textbook algorithm or a browser form to enter data, answering documentation-like questions about a system) but aren't much help yet on the more skilled labor parts of software engineering.

I was expecting earlier choices of foraging location to have a much stronger impact, and mistook some of the randomness for affects of earlier choices. In retrospect it would have been better to spend longer exploreing various possibilites rather than settling on an exploit strategy so soon. Adding an explicit target was a big improvement as it gave some idea of "how good a strategy" we should be searching for.

Some other examples:

1. Agency and embeddedness are fundamentally at odds with each other. Decision theory and physics are incompatible approaches to world-modeling, with each making assumptions that are inconsistent with the other. Attempting to build mathematical models of embedding agency will fail as an attempt to understand advanced AI behavior.
2. Reductionism is false. If modeling a large-scale system in terms of the exact behavior of its small-scale components would take longer than the age of the universe, or would require a universe-sized computer, the large-scale system isn't explicable in terms of small-scale interactions even in principle. The Sequences are incorrect to describe non-reductionism as ontological realism about large-scale entities [LW · GW] -- the former doesn't inherently imply the latter.
3. Relatedly, nothing is ontologically primitive. Not even elementary particles: if, for example, you took away the mass of an electron, it would cease to be an electron and become something else. The properties of those particles, as well, depend on having fields to interact with. And if a field couldn't interact with anything, could it still be said to exist?
4. Ontology creates axiology and axiology creates ontology. We aren't born with fully formed utility functions in our heads telling us what we do and don't value. Instead, we have to explore and model the world over time, forming opinions along the way about what things and properties we prefer. And in turn, our preferences guide our exploration of the world and the models we form of what we experience. Classical game theory, with its predefined sets of choices and payoffs, only has narrow applicability, since such contrived setups are only rarely close approximations to the scenarios we find ourselves in.

I wonder if it would be a good idea to put editor's notes after likely typos, like:

but we want [editor's note: Elon likely meant “won’t” here] do any contract or agree to “evangelize”.

Yes, it sounds that he put too much stock into Andrej's paper-counting argument, and then even left the board because he didn't want to be associated with a failing company?

This sounds like metacognitive concepts and models. Like past, present, future, you can roughly align them with three types of metacognitive awareness: declarative knowledge, procedural knowledge, and conditional knowledge.

#1 - What do you think you know, and how do you think you know it?

Content knowledge (declarative knowledge) which is understanding one's own capabilities, such as a student evaluating their own knowledge of a subject in a class. It is notable that not all metacognition is accurate.

#2 - Do you know what you are doing, and why you are doing it?

Task knowledge (procedural knowledge) refers to knowledge about doing things. This type of knowledge is displayed as heuristics and strategies. A high degree of procedural knowledge can allow individuals to perform tasks more automatically.

#3 - What are you about to do, and what do you think will happen next?

Strategic knowledge (conditional knowledge) refers to knowing when and why to use declarative and procedural knowledge. It is one's own capability for using strategies to learn information.

Another somewhat tenuous alignment is with metacognitive skills: evaluating, monitoring, and planning.

#1 - What do you think you know, and how do you think you know it?

Evaluating: refers to appraising the final product of a task and the efficiency at which the task was performed. This can include re-evaluating strategies that were used.

#2 - Do you know what you are doing, and why you are doing it?

Monitoring: refers to one's awareness of comprehension and task performance

#3 - What are you about to do, and what do you think will happen next?

Planning: refers to the appropriate selection of strategies and the correct allocation of resources that affect task performance.

Quotes are adapted from https://en.wikipedia.org/wiki/Metacognition

I sometimes feel we spend too much time on philosophy and communication in the x-risk community. But thinking through the OpenAI drama suggests that it's crucial.

Now the world is in more and more immediate danger because a couple of smart guys couldn't get their philosophy or their communication right-enough, and didn't spend the time necessary to clarify. Instead Musk followed his combative and entrepreneurial instincts. The result was dramatically heating up the race for AGI, which previously had no real competition to DeepMind.

OpenAI wouldn't have launched without Musk's support, and he gave it because he was afraid of Larry Page being in charge of a successful Google AGI effort.

From Musk's interview with Tucker Carlson (automated transcript, sorry!):

I mean the the reason open AI exists at all is that um Larry Paige and I used to be close friends and I would stay at his house in pal Alto and I would talk to him late into the night about uh AI safety and at least my (01:12) perception was that Larry was not taking uh AI safety seriously enough um and um what did he say about it he really seemed to be um one want want sort of digital super intelligence basically digital God if you will and at one point uh I said well what about you know we're going to make sure humanity is okay here um and and and um uh and then he called me a speciest

Musk was afraid of what Page would do with AGI because Page called Musk a speciesist (specist?) when they were talking about AGI safety. What did Page mean by this? He probably hadn't worked it all the way through.

These guys stopped being friends, and Musk put a bunch of money and effort into developing an org that could rival DeepMind's progress toward AGI.

That org was captured by Altman. But it was always based on a stupid idea: make AGI open source. That's the dumbest thing you could do with something really dangerous - unless you believed that it would otherwise wind up in hands that just don't care about humanity.

That's probably not what Page meant. On consideration, he would probably have clarified that AI that includes what we value about humanity would be a worthy successor. He probably wasn't even clear on his own philosophy at the time.

A little more careful conversation would've prevented this whole thing, and we'd be in a much better strategic position.

In my mind this also shows how immensely intelligent people can also do really dumb things outside of their area of intellectual expertise.

It seems like Musk in 2018 dramatically underestimated the ability of OpenAI to compete with Google in the medium term.

Thanks for not only doing this but noting the accuracy of the unchecked transcript, it's always hard work to build a mental model of how good LLM tools are at what stuff.

I'm quite happy for laws to be passed and enforced via the normal mechanisms. But I think it's bad for policy and enforcement to be determined by Elon Musk's personal vendettas. If Elon tried to defund the AI safety institute because of a personal vendetta against AI safety researchers, I would have some process concerns, and so I also have process concerns when these vendettas are directed against OAI.