My understanding of Steel Late Wittgenstein's response would be that you could agree with that words and concepts are distinct, and mapping is not always 1-1, but that what concepts get used is also significantly influenced by which features of the world are useful in some contexts of language (/word) use. 

Update: This comment was generated using GPT4 as an experiment.

Thank you for your comment and insights. I appreciate your engagement with the topic. However, I believe there may be a misunderstanding regarding Wittgenstein's ideas and their implications for the Natural Abstraction Hypothesis.

Firstly, Wittgenstein did not deny the existence of an external reality or the importance of human interaction with it. His primary focus was on the limits of language and how it shapes our understanding of the world. As he famously stated, "Whereof one cannot speak, thereof one must be silent," implying that there are aspects of reality that language cannot capture. He emphasized the role of context and usage in determining meaning, which suggests that words and concepts are not the same thing and that their relationship is more complex and variable than a simple one-to-one correspondence.

In this regard, Wittgenstein's ideas about language games and the contextual nature of meaning are indeed relevant to the discussion of the NAH. The original essay critiqued the hypothesis by highlighting the context-dependent, diverse, and evolving nature of language games, which challenges the idea of a fixed, universally natural set of abstractions. These points do not conflate words with concepts but emphasize the complex relationship between them and the importance of context in determining meaning.

Your clarification that the NAH is ultimately about concepts, not words, is duly noted. However, the critique presented in the essay is not solely based on a linguistic analysis. The contextual variability and diversity observed in language games can also be extended to the realm of concepts and abstractions, as they are intricately linked to the cultural and social practices that inform our understanding of the world.

In conclusion, the original essay's critique of the NAH remains relevant, as it emphasizes the context-dependent and diverse nature of abstractions, which challenges the notion of fixed, universally natural abstractions. This critique does not conflate words and concepts but highlights the complex, fluid, and context-dependent relationships between them, as well as the importance of considering cultural and social factors when evaluating the convergence of abstractions in cognitive systems.

Oh, that's interesting. Yeah, I hadn't thought about how instrumental convergence might play into this before. Just thought I'd note that the language game critique is very similar to your telos frame as Wittgenstein has this concept of "language as use" where in most cases language is a tool to achieve a particular result within a particular language game. So it sounds like you're actually suspicious of NAH for mostly the same reasons, but where you depart is that instrumental convergence limits the effects of this divergence.

My expectations are the Natural Abstractions Hypothesis probably works out as long as we don't try to include values/ethics/morality into the mix, so I am more optimistic on the convergence of non moral abstractions.

This is important, because while it wouldn't let us automatically solve the alignment problem, it does make it way easier to change a model's goals.

I tried asking it to include examples which didn't work so well. So I instead asked it to generate four examples for each of the three points.

Overall, it's a lot easier to produce quality content when you can increase the maximum content generation size and the ability to generate examples seems useful, but you still need to do a lot manually yourself.

Results:

Certainly! Here are four different examples for each of the three points:

1. Context-dependent and diverse abstractions
a) The concept of "freedom" can yield different abstractions in various political contexts, such as individual liberty in a liberal democracy or collective autonomy in a socialist system (less compelling example, maybe people in a socialist system adopt that definition of freedom, hard for me to gauge really)
b) The idea of "family" can have diverse abstractions across cultures, with some emphasizing nuclear family structures, others prioritizing extended family networks, and some valuing chosen families or non-blood relations (solid example)
c) The notion of "success" can differ across societies, with some emphasizing material wealth and career achievements, while others prioritize personal happiness, community involvement, or spiritual growth (solid example)
d) The concept of "ownership" can manifest differently in various economic and cultural systems, from the individual property rights in capitalist societies to communal ownership in certain indigenous cultures or socialist systems (solid example)

2. Evolution of language games
a) The term "gentleman" has evolved over time, initially referring to a man of noble birth, then encompassing courteous and chivalrous behavior, and now representing a well-mannered and respectful man (not clear how the second and third points differ)
b) The concept of "privacy" has shifted with technological advancements, expanding from physical solitude or personal correspondence to include digital data and online interactions (meaning has probably shifted, unsure if it's the best example as it hasn't shifted that much)
c) The meaning of "friendship" has evolved as social media platforms have emerged, broadening the term to include not only close personal bonds but also casual acquaintances and online connections (not sure that the meaning of friendship has broadened vs. an additional usage)
d) The understanding of "health" has changed over time, moving from a focus on the absence of disease to a more holistic perspective that considers mental, emotional, and social well-being (solid example)

3. Human-Compatibility
a) The perception of "beauty" varies significantly across cultures and cognitive systems, with different aesthetic preferences and ideals shaping the development of distinct abstractions (maybe?)
b) The concept of "intelligence" may have diverse abstractions in different cognitive systems, with some focusing on analytical and logical abilities, others prioritizing emotional intelligence, and some considering creativity or problem-solving skills
c) The idea of "trust" can be understood differently across cognitive systems, such as trust in institutions, trust in personal relationships, or trust in technology and artificial intelligence (seems like mostly the concept in different contexts)
d) The notion of "fairness" may be represented differently in various cognitive systems, such as in human cultures with different moral values or in AI systems with distinct ethical frameworks and decision-making processes (kind of relevant)

Merging these into the original gives:

Introduction
The Natural Abstraction Hypothesis (NAH), put forth by John Wentworth, posits that certain natural abstractions exist, which cognitive systems are expected to converge upon. In this essay, we will scrutinize the NAH through the lens of Ludwig Wittgenstein's concept of language games. We will argue that the variability and contextual nature of language games challenge the NAH, as they illustrate that abstractions are neither fixed, universal, nor entirely convergent across cognitive systems.

Wittgenstein's Language Games
Ludwig Wittgenstein, a prominent philosopher of language, introduced the concept of language games in his seminal work, "Philosophical Investigations." Language games are social practices wherein language and meaning are deeply intertwined. Wittgenstein contended that the meaning of words is not static or inherent but derives from their usage within various language games. These language games are context-dependent, diverse, and ever-evolving, reflecting the intricacies of human social practices and cultural contexts.

Critique of the NAH from the Perspective of Language Games
1. Context-dependent and diverse abstractions
The concept of language games undermines the idea of a single set of natural abstractions that cognitive systems can converge upon. Language games are context-dependent, signifying that the meaning of words, concepts, and abstractions relies on the specific language game being played. Furthermore, the diversity of language games indicates that a multitude of abstractions related to a single concept may exist, each stemming from distinct linguistic and cultural practices. This amalgamation of context-dependence and diversity casts doubt on the notion of convergence towards a single, natural abstraction. For example, the notion of "success" can differ across societies, with some emphasizing material wealth and career achievements, while others prioritize personal happiness, community involvement, or spiritual growth (solid example)
 

2. Evolution of language games
Language games are not static entities; they evolve over time as social practices and cultural contexts shift. This dynamism suggests that abstractions themselves may be subject to change, further complicating the idea of fixed, natural abstractions. The NAH's assumption that cognitive systems will converge on a consistent set of abstractions may be undermined by the fluid nature of language games and the abstractions they produce. For example, the understanding of "health" has changed over time, moving from a focus on the absence of disease to a more holistic perspective that considers mental, emotional, and social well-being

3. Human-Compatibility
The NAH posits human compatibility, referring to the idea that the lower-dimensional summaries or abstractions employed by humans in day-to-day thought and language are natural and convergent across cognitive systems. However, Wittgenstein's language games demonstrate that human thought and language are diverse, context-dependent, and constantly evolving. This variability complicates the notion that human-compatible abstractions are universally "natural" and brings into question the assumption that disparate cognitive systems will necessarily converge on the same abstractions. For example, the concept of "intelligence" may have diverse abstractions in different cognitive systems, with some focusing on analytical and logical abilities, others prioritizing emotional intelligence, and some considering creativity or problem-solving skills

Conclusion
Wittgenstein's concept of language games offers a potent critique of the Natural Abstraction Hypothesis. The context-dependent, diverse, and evolving nature of language games emphasizes the complexities of human linguistic and conceptual practices, which challenge the idea of a fixed, universally natural set of abstractions. By accentuating the importance of context and diversity, Wittgenstein's language games encourage us to reevaluate the assumptions of the NAH and investigate alternative frameworks for understanding the development and convergence of abstractions in cognitive systems.

There are correct and incorrect ways to play language games.

That's the crux. Wittgenstein himself believed otherwise and spent the most part of the book arguing against it. I think he makes good points.

At one point, he argues that there's no single correct interpretation for "What comes next in the sequence: '2, 4, 6, 8, 10, 12, ...?'" 

Maybe this goes a bit too far. :) I think he's right in some nitpicky sense, but for practical purposes, sane people will say "14" every time and that works well for us.

We can see this as version of realism vs anti-realism debates [EA · GW]: realism vs anti-realism about natural abstractions.  As I argue in the linked post, anti-realism is probably the right way of looking at most or even all of these, but that doesn't mean "anything goes." Sometimes there's ambiguity about our interpretations of things, but reality does have structure, and "ambiguity" isn't the same as "you can just make random stuff up and expect it to be useful."

and some audiences have measurably better calibration.

It's not straightforward in all contexts to establish what counts as good calibration. It's straightforward for empirical forecasting, but if we were to come up with a notion like "good calibration for ethical judgments," we'd have to make some pretty subjective judgment calls. Similarly, something like "good calibration for coming up with helpful abstractions for language games" (which we might call "doing philosophy" or a subskill of it) also seems (at least somewhat) subjective. 

That doesn't mean "anything goes," but I don't yet see how your point about dialogue trees applies to "maybe a society of AIs would build abstractions we don't yet understand, so there'd be a translation problem between their language games and ours." 

Sorry, I can't quite follow why you are saying that dialogue trees are the substrate of language games or how this ties into the arguments. Any chance you could clarify?