Much appreciated!

3. Naively, actions feel like they require causal reasoning, and causal reasoning of any kind seems to require the ability to reason about two parts of the environment. One of these parts can be you (or the AI).

But I am not sure this is the case. Strong correlation seems to be good enough for a human brain - we do all kinds of actions without any understanding of what we are doing or why. This can go as far as provoking conscious confusion during the action. Based on this lower standard, what correlation would be needed?

Boundaries of some kind, because we need some way to localize what we are doing and looking at. Strong, chiefly as a matter of efficiency. We want a way to describe a correlation such that we can chain it with other correlations, and then eventually bundle them together as an action. Then doing new actions is a matter of chaining the correlations back to something we can currently do.

2. Is thinking about actions just rephrasing the agent-environment question? It feels like the answer is no, because it isn't as though being able to specify the relationship between the agent and the environment changes the need to compute the specific details of any particular action.

But it might be impossible to specify an action exactly without being able to specify the agent-environment relationship exactly. Could it be (or is it) stated implicitly?

1. I am deeply confused by this.

The older conversations about Tool AI seemed to focus on the difference between an Oracle that answers questions and one that does things. I feel like this distinction is bigger and in a different way than it was made out to be, because doing things is really hard. If feels like the paradox of sensing being complicated should go two ways.

Checking the Wikipedia page for Moravec's Paradox, "sensorimotor" is how they describe it, so both sensing and motor skills (inputs and outputs) are covered. My intuition fairly screams that this should generalize to any other environment-affecting action. So:

• The more inputs an AI starts with, the easier it is to recognize other inputs/outputs.
• The more outputs an AI starts with, the easier it is to add other inputs/outputs.
• This still doesn't identify what causes the machine to try to affect the world at all.

4. Information technology has massively increased certain kinds of coordination (e.g., email, eBay, Facebook, Uber), but at the international relations level, IT seems to have made very little impact. Why?

I note the coordination is entirely at a lower-level than those companies: mostly individuals are using these services for coordination, as well as small groups. It seems like coordination innovations aren't bottom up, but rather top-down (even if the IT examples are mostly opt-in). This seems to match other large coordination improvements, like empire, monotheism, or corporations. There is no higher level of abstraction than governments from which to improve international relations, it seems to me.

Quite separately, we could ask: what are the specific challenges in international relations that IT could address? The problems mostly revolve around questions of trust, questions of the basic competence of human agents (diplomats, ambassadors, heads of state, etc), and fundamental conflicts of interest. None of these are really addressable with off-the-shelf IT solutions.

That being said, it's also clear that Facebook and Uber aren't even trying to target problems related to international relations. We know contracting with multiple governments is achievable, because people like Google, Microsoft, and Palantir all manage it selling IT for intelligence purposes. Dominic Cummings has a blog post High performance government, ‘cognitive technologies’, Michael Nielsen, Bret Victor, & ‘Seeing Rooms’ that speculates about how international relations could be improved by making the stupendous complexity of the information at work more readily available to decision makers, both for educational purposes and in real time. Maybe there would be an opportunity for a Situation Room Industries, or similar.

This might be more fundamental than I initially thought. If people have a preference for shared information, this provides an upward pressure on communication in general - there is now a reward for telling people things just because, and also a reward for listening to people tell you things. This is entirely separate from any instrumental value the shared information may have.

I suspect that a lot of traditional things do double-duty, being instrumentally valuable and encouraging mutual information. Material example: the decorations on otherwise mundane items, like weapons or cauldrons.

It occurs to me that any given practice will probably be selected for resilience more than it will be for efficiency - efficiency is a threshold that must be met, but beyond that maximizing the likelihood that the minimum will be met seems more likely to propagate.

If people have an intrinsic desire to share information, then it seems likely that a practice for which sharing information is more common is more likely to persist. Hence the prevalence of so many multi-step processes; every additional step is another opportunity to share information (teach, tell where resources are, etc).

From Introducing: Asabiyah, which itself summarizes one concept from Ibn Khaldun's Muqaddimah:

Thus Khaldun notes:

“The consequences of common descent, though natural, still are something imaginary. The real thing to bring about the feeling of close contact is social intercourse, friendly association, long familiarity, and the companionship that results from growing up together having the same wet nurse, and sharing the other circumstances of life and death. If close contact is established in such a manner, the result will be affection and cooperation.”

And later:

In Ibn Khaldun’s thought, conquest itself seems to be the driving force behind the consolidation of two asabiyah into one. Once a weaker tribal group is defeated, its leaders removed and men of valor killed, pacified, or subsumed under a new organization so utterly that the ‘tit for tat’ vengeance schemes so common to nomadic society (which Ibn Khaldun sees as the root cause of war) are no longer possible, then their asabiyah can be swallowed up in the larger group’s. What is key here is that the other groups – after their initial defeat – are not coerced into having the same feeling of asabiyah as the main group. Asabiyah that must be coerced is not asabiyah at all (this is a theme Ibn Khaldun touches on often and we will return to it in more detail when we talk about why asabiyah declines in civilized states). Instead, those who have been allowed to join the conquering host slowly start to feel its asabiyah be subsumed as the two groups “enter into close contact,” sharing the same trials, foods, circumstances, and becoming acquainted with the others' customs, but just as importantly, sharing the same set of incentives. Once the losers are are forced together with the winners, defeat for the main clan is defeat for all; glory for the main clan is glory for all; booty gained by the main clan’s conquests becomes booty to be shared with all. Once people from a subordinate group begin to feel like the rise and fall of their own fortunes is inextricably linked to the fate of the group that overpowered them then they become willing to sacrifice and die for the sake of this group, for it has become their group.

Shared experiences create a lot of mutual information, and enough of it builds the fearsome bonds which populate our legends and histories.

Seemingly altruistic actions such as creating art and music may qualify. From Sexual Selection Through Mate Choice Does Not Explain the Evolution of Art and Music:

Miller, however, criticizes the idea that “art conveys cultural values and socializes the young,” writing that,

"The view that art conveys cultural values and socializes the young seems plausible at first glance. It could be called the propaganda theory of art. The trouble with propaganda is that it is usually produced only by large institutions that can pay propagandists. In small prehistoric bands, who would have any incentive to spend the time and energy producing group propaganda? It would be an altruistic act in the technical biological sense: a behavior with high costs to the individual and diffuse benefits to the group. Such altruism is not usually favored by evolution."

The answer to Miller’s question—who produces the propaganda?—is quite clear in the ethnographic data: the old men do.

Altruism is not usually favored by evolution, but if the same mechanism by which we prefer to spread our genetic information recognizes other kinds of information, then it would not feel altruistic from the inside. Rather, making songs and having other people sing them would be its own reward in precisely the same way having children does.

Some related posts:

• Optimizing for [LW · GW] Stories [LW · GW]
• Simulacra [LW · GW]