As someone that does think about a lot of the things you care about at least some of the time (and does care pretty deeply), I can speak for myself why I don't talk about these things too much:

Epistemic problems:

• Mostly, the concept of "metaphilosophy" is so hopelessly broad that you kinda reach it by definition by thinking about any problem hard enough. This isn't a good thing, when you have a category so large it contains everything (not saying this applies to you, but it applies to many other people I have met who talked about metaphilosophy), it usually means you are confused.
• Relatedly, philosophy is incredibly ungrounded and epistemologically fraught. It is extremely hard to think about these topics in ways that actually eventually cash out into something tangible, rather than nerdsniping young smart people forever (or until they run out of funding).
• Further on that, it is my belief that good philosophy should make you stronger, and this means that fmpov a lot of the work that would be most impactful for making progress on metaphilosophy does not look like (academic) philosophy, and looks more like "build effective institutions and learn interactively why this is hard" and "get better at many scientific/engineering disciplines and build working epistemology to learn faster". Humans are really, really bad at doing long chains of abstract reasoning without regular contact with reality, so in practice imo good philosophy has to have feedback loops with reality, otherwise you will get confused.  I might be totally wrong, but I expect at this moment in time me building a company is going to help me deconfuse a lot of things about philosophy more than me thinking about it really hard in isolation would.
• It is not clear to me that there even is an actual problem to solve here. Similar to e.g. consciousness, it's not clear to me that people who use the word "metaphilosophy" are actually pointing to anything coherent in the territory at all, or even if they are, that it is a unique thing. It seems plausible that there is no such thing as "correct" metaphilosophy, and humans are just making up random stuff based on our priors and environment and that's it and there is no "right way" to do philosophy, similar to how there are no "right preferences". I know the other view ofc and still worth engaging with in case there is something deep and universal to be found (the same way we found that there is actually deep equivalency and "correct" ways to think about e.g. computation).

Practical problems:

• I have short timelines and think we will be dead if we don't make very rapid progress on extremely urgent practical problems like government regulation and AI safety. Metaphilosophy falls into the unfortunate bucket of "important, but not (as) urgent" in my view.
• There are no good institutions, norms, groups, funding etc to do this kind of work.
• It's weird. I happen to have a very deep interest in the topic, but it costs you weirdness points to push an idea like this when you could instead be advocating more efficiently for more pragmatic work.
• It was interesting to read about your successive jumps up the meta hierarchy, because I had a similar path, but then I "jumped back down" when I realized that most of the higher levels is kinda just abstract, confusing nonsense and even really "philosophically concerned" communities like EA routinely fail basic morality such as "don't work at organizations accelerating existential risk" and we are by no means currently bottlenecked by not having reflectively consistent theories of anthropic selection or whatever. I would like to get to a world where we have bottlenecks like that, but we are so, so far away from a world where that kind of stuff is why the world goes bad that it's hard to justify more than some late night/weekend thought on the topic in between a more direct bottleneck focused approach.

All that being said, I still am glad some people like you exist, and if I could make your work go faster, I would love to do so. I wish I could live in a world where I could justify working with you on these problems full time, but I don't think I can convince myself this is actually the most impactful thing I could be doing at this moment.

Philosophy is a social/intellectual process taking place in the world. If you understand the world, you understand how philosophy proceeds.

Sometimes you don't need multiple levels of meta. There's stuff, and there's stuff about stuff, which could be called "mental" or "intensional". Then there's stuff about stuff about stuff (philosophy of mind etc). But stuff about stuff about stuff is a subset of stuff about stuff. Mental content has material correlates (writing, brain states, etc). I don't think you need a special category for stuff about stuff about stuff, it can be thought of as something like self-reading/modifying code. Or like compilers compiling themselves; you don't need a special compiler to compile compilers.

Philosophy doesn't happen in a vacuum, it's done by people with interests in social contexts, e.g. wanting to understand what other people are saying, or be famous by writing interesting things. A sufficiently good theory of society and psychology would explain philosophical discourse (and itself rely on some sort of philosophy for organizing its models). You can think of people as having "a philosophy" that can be studied from outside by analyzing text, mental states, and so on.

Reasoning about mind embeds reasoning about matter, reasoning about people embeds reasoning about mind, reasoning about matter embeds reasoning about people. Mainstream meta-philosophy consists of comparative analysis of philosophical texts, contextualized by the historical context and people and so on.

Your proposed reflection process for designing a utopia is your proposed utopia. If you propose CEV or similar, you propose that the world would be better if it included a CEV-like reflection context, and that this context had causal influence over the world in the future.

I'm not sure how clear I'm being, but I'm proposing something like collapsing levels of meta by finding correspondences between meta content and object content, and thinking of meta-meta content as meta relative to the objects corresponding to the meta content. This leads to a view where philosophy is one of many types of discourse/understanding that each shape each other (a non-foundationalist view). This is perhaps disappointing if you wanted ultimate foundations in some simple framework. Most thought is currently not foundationalist, but perhaps a foundational re-orientation could be found by understanding the current state of non-foundational thought.

I wonder if more people would join you on this journey if you had more concrete progress to show so far?

If you're trying to start something approximately like a new field, I think you need to be responsible for field-building. The best type of field-building is showing that the new field is not only full of interesting problems, but tractable ones as well.

Compare to some adjacent examples:

• Eliezer had some moderate success building the field of "rationality", mostly though explicit "social" field-building activities like writing the sequences or associated fanfiction, or spinning off groups like CFAR. There isn't much to show in terms of actual results, IMO; we haven't developed a race of Jeffreysai supergeniuses who can solve quantum gravity in a month by sufficiently ridding themselves of cognitive biases. But the social field-building was enough to create a great internet social scene of like-minded people.
• MIRI tried to kickstart a field roughly in the cluster of theoretical alignment research, focused around topics like "how to align AIXI", decision theories, etc. In terms of community, there are a number of researchers who followed in these footsteps, mostly at MIRI itself to my knowledge, but also elsewhere. (E.g. I enjoy @Koen.Holtman [LW · GW]'s followup work such as Corrigibility with Utility Preservation.) In terms of actual results, I think we see a steady stream of papers/posts showing slow-but-legible progress on various sub-agendas here: infra-bayesianism, agent foundations, corrigibility, natural abstractions, etc. Most (?) results seem to be self-published to miri.org, the Alignment Forum, or the arXiv, and either don't attempt or don't make it past peer review. So those who are motivated to join a field by legible incentives such as academic recognition and acceptance are often not along for the ride. But it's still something.
• "Mainstream" AI alignment research, as seen by the kind of work published by OpenAI, Anthropic, DeepMind, etc., has taken a much more conventional approach. People in this workstream are employed at large organizations that pay well; they publish in peer-reviewed journals and present at popular conferences. Their work often has real-world applications in aligning or advancing the capabilities of products people use.

In contrast, I don't see any of this sort of field-building work from you for meta-philosophy. Your post history doesn't seem to be trying to do social field-building, nor does it contain published results that could make others sit up and take notice of a tractable research agenda they could join. If you'd spent age 35-45 publishing a steady stream of updates and progress reports on meta-philosophy, I think you'd have gathered at least a small following of interested people, in the same way that the theoretical alignment research folk have. And if you'd used that time to write thousands of words of primers and fanfic, maybe you could get a larger following of interested bystanders. Maybe there's even something you could have done to make this a serious academic field, although that seems pretty hard.

In short, I like reading what you write! Consider writing more of it, more often, as a first step toward getting people to join you on this journey.

@jessicata [LW · GW] @Connor Leahy [LW · GW] @Domenic [LW · GW] @Daniel Kokotajlo [LW · GW] @romeostevensit [LW · GW] @Vanessa Kosoy [LW · GW] @cousin_it [LW · GW] @ShardPhoenix [LW · GW] @Mitchell_Porter [LW · GW] @Lukas_Gloor [LW · GW] (and others, apparently I can only notify 10 people by mentioning them in a comment)

Sorry if I'm late in responding to your comments. This post has gotten more attention and replies than I expected, in many different directions, and it will probably take a while for me to process and reply to them all. (In the meantime, I'd love to see more people discuss each other's ideas here.)

ensuring AI philosophical competence won't be very hard. They have a specific (unpublished) idea that they are pretty sure will work.

Cool, can you please ask them if they can send me the idea, even if it's just a one-paragraph summary or a pile of crappy notes-to-self?

From my current position, it looks like "all roads lead to metaphilosophy" (i.e., one would end up here starting with an interest in any nontrivial problem that incentivizes asking meta questions) and yet there's almost nobody here with me. What gives?

Facile response: I think lots of people (maybe a few hundred a year?) take this path, and end up becoming philosophy grad students like I did. As you said, the obvious next step for many domains of intellectual inquiry is to go meta / seek foundations / etc., and that leads you into increasingly foundational increasingly philosophical questions until you decide you'll never able to answer all the questions but maybe at least you can get some good publications in prestigious journals like Analysis and Phil Studies, and contribute to humanity's understanding of some sub-field.

 

First, I think that the theory of agents [LW · GW] is a more useful starting point than metaphilosophy. Once we have a theory of agents, we can build models, within that theory, of agents reasoning about philosophical questions. Such models would be answers to special cases of metaphilosophy. I'm not sure we're going to have a coherent theory of "metaphilosophy" in general, distinct from the theory of agents, because I'm not sure that "philosophy" is an especially natural category^[1].

Some examples of what that might look like:

• An agent inventing a theory of agents in order to improve its own cognition is a special case of recursive metalearning (see my recent talk [LW(p) · GW(p)] on metacognitive agents).
• There might be theorems about convergence of learning systems to agents of particular type (e.g. IBP [LW · GW] agents), formalized using some brand of ADAM [LW · GW], in the spirit of John's Selection Theorems [LW · GW] programme. This can be another model of agents discovering a theory of agents and becoming more coherent as a result (broader in terms of its notions of "agent" and "discovering" and narrower in terms of what the agent discovers).
• An agent learning how to formalize some of its intuitive knowledge (e.g. about its own values) can be described in terms of metacognition, or more generally, the learning of some formal symbolic language. Indeed, understanding is translation [LW · GW], and formalizing intuitive knowledge means translating it from some internal opaque language to an external observable language.

Second, obviously in order to solve philosophical problems (such as the theory of agents), we need to implement a particular metaphilosophy. But I don't think it needs to has to be extremely rigorous. (After all, if we tried to solve metaphilosophy instead, we would have the same problem.) My informal theory of metaphilosophy is something like: an answer to a philosophical question is good when it seems intuitive, logically consistent and parsimonious^[2] after sufficient reflection (where "reflection" involves, among other things, considering special cases and other consequences of the answer, and also connecting the answer to empirical data).

1. ^{^}

   I think that philosophy just consists of all domains where we don't have consensus about some clear criteria of success. Once such a consensus forms, this domain is no longer considered philosophy. But the reasons some domains have this property at this point of time might be partly coincidental and not especially parsimonious.

2. ^{^}

   Circling back to the first point, what would a formalization of this within a theory of agents look like? "Parsimony" refers to a simplicity prior, "intuition" refers to opaque reasoning in the core of a metacognitive agent, and "logically consistency" is arguably some learned method of testing hypotheses (but maybe we will have a more elaborate theory of the latter). 

I'm pretty much with you on this. But it's hard to find a workable attack on the problem.

One question though, do you think philosophical reasoning is very different from other intelligence tasks? If we keep stumbling into LLM type things which are competent at a surprisingly wide range of tasks, do you expect that they'll be worse at philosophy than at other tasks?

How can I better recruit attention and resources to this topic?

Consider finding an event organizer/ops person and running regular retreats on the topic. This will give you exposure to people in a semi-informal setting, and help you find a few people with clear thinking who you might want to form a research group with, and can help structure future retreats.

I've had great success with a similar approach.

Why is there virtually nobody else interested in metaphilosophy or ensuring AI philosophical competence (or that of future civilization as a whole) 

I interpret your perspective on AI as combining several things: believing that superhuman AI is coming; believing that it can turn out very bad or very good, and that a good outcome is a matter of correct design; believing that the inclinations of the first superhuman AI(s) will set the rules for the remaining future of civilization. 

This is a very distinctive combination of beliefs. At one time, I think Less Wrong was the only intellectual community in which that combination was commonplace. I guess that it then later spread to parts of the Effective Altruism and AI safety communities, once they existed.  

Your specific take is then that correct philosophical cognition may be essential, because decision theory, and normativity in general, is one of the things that AI alignment has to get right, and the best thinking there came from philosophy. 

I suspect that the immediate answer to your question, is that this specific line of thought would only occur to people who share those three presuppositions - those "priors", if you like - and that was always a small group of people, busy with a very multifaceted problem. 

And furthermore, if someone from that group did try to identify the kind of thinking by the AI, that needs to be correct for a good outcome, they wouldn't necessarily identify it as "philosophical thinking" - especially since many such people would disdain what is actually done in philosophy. They might prefer cognitive labels like metacognition, concept formation, or theory formation, or they might even think in terms of the concepts and vocabulary of computer programming. 

One way to get perspective on this, is to see if someone else managed to independently invent this line of thought, but under a different name, or even in a different concept. Here's something ironic: it occurred to me to wonder, if anyone asked this question, during the advent of psychoanalysis. Someone might have thought, psychoanalysis has the power to shape minds, it could determine the future of the human race, we'd better make sure that psychoanalysts have the right philosophy. If you look for discussions of psychoanalysis and metaphilosophy, I don't think you'll find that exact concern, but you will find that the first recorded use of the term "metaphilosophy" was by a psychoanalyst, Morris Lazerowitz. However, he was psychoanalyzing the preoccupations of philosophers, rather than sophoanalyzing the presuppositions of psychoanalysts. 

Another person I checked was Jurgen Schmidhuber, the AI pioneer. I found a 2012 paper by him, telling "philosophers and futurists [to] catch up" with new computer-science definitions of intelligence, problem-solving, and creativity - many of them due to him. This is an example of someone in the AI camp who went seeking cognitive fundamentals too, but who came to regard something computational (in Schmidhuber's case, data compression), rather than "philosophy", as the wellspring of cognitive progress. (Incidentally, Schmidhuber's attitude to the future of morality is relativism tempered by darwinism - there will be multiple AI value systems, and the "survivors" will determine what is regarded as moral.) 

On the other hand, I belong to a camp that arrives at the importance of philosophical cognition, owing to concerns about inadequate philosophy in the community, and its consequences for scientific ontology and AI consciousness. I wrote an essay here a decade ago, "Friendly AI and the limits of computational epistemology" [LW · GW], arguing that physicalism (as well as more esoteric ontologies like mathematical platonism and computational platonism) is incomplete, but that the favored epistemologies, here and in adjacent communities, are formally incapable of noticing this, and that these ontological and epistemological presuppositions might be built into the AIs. 

As it turns out, an even more pragmatist and positivist approach to AI, deep learning, won out, and as a result we now have AI colleagues that can talk to us, who have a superhuman speed and breadth of knowledge, but whose inner workings we don't even understand. It remains to be seen whether the good that their polymathy can do, outweighs the bad that their inscrutability portends, for the future of AI alignment. 

When I look at metaphilosophy, the main places I go looking are places with large confusion deltas. Where, who, and why did someone become dramatically less philosophically confused about something, turning unfalsifiable questions into technical problems. Kuhn was too caught up in the social dynamics to want to do this from the perspective of pure ideas. A few things to point to.

1. Wittgenstein noticed that many philosophical problems attempt to intervene at the wrong level of abstraction and posited that awareness of abstraction as a mental event might help
2. Korzybski noticed that many philosophical problems attempt to intervene at the wrong level of abstraction and posited that awareness of abstraction as a mental event might help
3. David Marr noticed that many philosophical and technical problems attempt to intervene at the wrong level of you get the idea
4. Hassabis cites Marr as of help in deconfusing AI problems
5. Eliezer's Technical Explanation of Technical Explanation doesn't use the term compression and seems the worse for it, using many many words to describe things that compression would render easier to reason about afaict.
6. Hanson in the Elephant in the Brain posits that if we mysteriously don't make progress on something that seems crucial, maybe we have strong motivations for not making progress on it.

Question: what happens to people when they gain consciousness of abstraction? My first pass attempt at an answer is that they become a lot less interested in philosophy.

Question: if someone had quietly made progress on metaphilosophy how would we know? First guess is that we would only know if their solution scaled well, or caused something to scale well.

I'm not sure why your path in life is so rare, but I find that as you go "upwards" in intellectual pursuits, you diverge from most people and things, rather than converge into one "correct" worldview.

I used to think about questions like you are now, until I figured that I was just solving my personal problems by treating them as external branches of knowledge. Afterwards I switched over to psychology, which tackled the problems more directly.

I also keep things simple for myself, so that I don't drown in them in any sense. If my thoughts aren't simple, it's likely that I don't understand what I'm thinking about.

I don't have much faith in logic anymore, anyway. What we're doing is essentially just constructing problems, and then eventually solving them by noticing that our initial construction contained an error. But does this snake eating its own tail even have any connection to reality in the first place? And isn't all logic just a path from axioms to whatever follows from them? But in that case, nothing new is ever derived.
And thus, yet another pursuit of mine ends up destroying itself. Thus is the nature of things, they seek an opposite.

But I haven't lost faith in reality (the tree of states that the world can be in), nor in usefulness (the ability to navigate towards desirable states). But at this point, I think that a concrete goal is needed. You seek to solve problems, but what does a solution look like? You need to define a desirable state in order to find a path to it. I don't think it works to ask for "the correct" answer, as that would rely on a unique, external preference. So I advice to have faith in your own evaluations, in order to have a concrete direction. If you don't dare to voice your own preferences in case they're "wrong", I'd say that's the biggest concern here.

Now, on to AI, which seems to be your main interest.
I think there's a big problem with AI alignment which is rarely mentioned: We're not even aligned with ourselves. If a genie appeared and granted us unlimited wishes, we'd ruin everything for ourselves quite quickly. For in the first place, we simply don't know what we want. This is one of the reasons why we contradict ourselves so often.

A whole lot of people want happiness, but that's actually trivial, they just have to be content with what is. But they say "No, it's not good enough yet. I won't be happy before things are better" - so their unhappiness is a choice. Is our real desire then "victory"? No, if we win any battles, we simply start on the next one. Do we want life to resist us? That's closer to the truth, but we certainly don't want too much resistance. If we consider life to be a game of DnD, we should understand that the game-master should be neither too harsh nor too soft.
You think AI will destroy you? It might. But most people would destroy themselves if they had omnipotence for 24 hours. It's too hasty to ask if an AI wants the same thing as us, as we don't even know what we want.
For an AI to be with you, it would also have to be partly against you.

Until you can get an AI to have human irrationality, it will just be a monkey's paw. Any purely logical perspective is inherently anti-human. The most logical statement which could be made about reality is "Everything is always exactly as it should be". Any other takes on reality are only human, so you probably don't want a hyper-intelligent and rational AI to make decisions for you.
Even if you can get an AI to value something that you value, that thing might be valuable only because it's scarce, and scarce because a terrible price has to be paid for its existence.

And just an extra note: You can't seperate anything from its opposite. You can't teach an AI 'good' without also teaching it 'evil' for instance. This is because "Good and evil" is one concept rather than two. There's nothing mysterious about the so-called "Waluigi effect", just like there's nothing mysterious about us fighting ourselves, nor about me using logic to arrive at the conclusion that logic isn't helpful.

To comment on the object (that is: meta) level discussion: One of the most popular theories of metaphilosophy states that philosophy is "conceptual analysis".

The obvious question is: What is "conceptual analysis"? The theory applies quite well to cases where we have general terms like "knowledge", "probability" or "explanation", and where we try to find definitions for them, definitions that are adequate to our antecedent intuitive understanding of those terms. What counts as a "definition"? That's a case of conceptual analysis itself, but the usual answer is that a good definition lists individually necessary and jointly sufficient conditions. And how do we find those?

That's where intuitions and thought experiments come in. There is a special kind of semantic intuition of the "I know it when I see it" kind. We know whether a term does or doesn't apply to a concrete example when we see it, even if we can't readily produce a definition. You might think being black is a necessary condition for our concept of "raven". Now if you see something that looks like a prototypical raven, but with darkish grey feathers, would you still call it a raven? Presumably yes. You can just imagine the example of a grey raven, i.e. you do a thought experiment, and your semantic intuition tells you whether the term applies. So you discovered that being black is not a necessary condition for being a raven, and hence it isn't part of their definition.

Similarly, you might want to know whether some properties are sufficient for applying a term, and a thought experiment could disprove this, if you imagine a thing which has all those properties but to which the term, nonetheless, doesn't intuitively apply. (A well known example for this happened in the case of knowledge.) Note also that these (semantic) intuitions seem very reliable, in contrast to the usual "intuitive guesses", which are simply unjustified and uncertain beliefs about factual matters, and which are also called "intuitions".

It's interesting that conceptual analysis bears some resemblance to the scientific method: Experiments are replaced by thought experiments, observations are replaced by semantic intuitions. Moreover, if I remember correctly, research in experimental philosophy (a branch of psychology) found that that intuitions in philosophical thought experiments are fairly stable across different people.

Now this semantic theory of metaphilosophy applies well to analyzing the meaning of general terms. It is less clear whether it applies to philosophy as a whole. For example, are philosophical ethicists really analyzing the meaning of terms like "good" and "ought"? (In fact I think so, yes.)

And which concepts do philosophical decision theorists analyze according to the semantic theory? Or logicians? Well, perhaps concepts like "rational decision" or "logical consequence". A subject area like decision theory could include a host of interrelated concepts, which stand in certain logical relations to each other, and an axiomatization provides an implicit definition of the involved terms. Like the axioms of (second-order) Peano arithmetic seem to define, in a sense, the meaning of "natural numbers". Another worry is that the conceptual analysis theory deals easily with "what is X?" questions (where X is "rationality", "causation" etc.) but not so obviously with "why" questions like "why does anything exist?" or "why does consciousness exist?". Though one could argue that the analysis of "explanation" should give some insight here. And also these questions seem not immediately semantic: "Is induction justified? If so, how?" Though again, maybe we simply still lack a general theory of induction which would help analyze the concepts of "inductive inference" and "inductive justification", and which would answer these questions.

So the semantic theory is a bit murky outside the prototypical examples. But it seems to fit well with the "deconfunsion theory", since conceptual analysis would aid in clarifying the relations between concepts. For example: Ask a philosophically uninformed person what the relationship is between knowledge and belief. I've done it several times. They struggle for confused examples, they know the difference between those terms, but only as a disposition to use them correctly when presented with specific cases, not when asked about an abstract relationship. Philosophers won't get confused by this case (they'd say knowledge implies belief, and belief doesn't imply knowledge), so the deconfunsion worked. They will still get confused when asked about other concepts, like when asked about the relationship between knowledge and information.

My guess is that a sufficiently general AI would be quite good at philosophy. Because deconfusing its conceptual framework has instrumental value. The danger comes from somewhat narrow superintelligences, which could have some important parts of their cognition "hard coded", like AIXI always using the conditionalization rule. And natural selection heavily optimized animals for generality first, and increased intelligence came only slowly [LW · GW], while this situation seems to be reversed in AI. Highly specialized narrow systems came first. So there is some risk the first powerful superintelligence could be relatively narrow and hence have dangerous philosophical blind spots.

I feel like there are two different concerns you've been expressing in your post history:

(1) Human "philosophical vulnerabilities" might get worsened (bad incentive setting, addictive technology) or exploited in the AI transition. In theory and ideally,  AI could also be a solution to this and be used to make humans more philosophically robust.

(2) The importance of "solving metaphilosophy," why doing so would help us with (1).

My view is that (1) is very important and you're correct to highlight it as a focus area we should do more in. For some specific vulnerabilities or failure modes, I wrote a non-exhaustive list here in this post [EA · GW] under the headings "Reflection strategies require judgment calls" and "Pitfalls of reflection procedures." Some of it was inspired by your LW comments.

Regarding (2), I think you overestimate how difficult the problem is. My specific guess is you might overestimate its difficulty because you might confuse uncertainty over a problem with objective solutions with indecisiveness about mutually incompatible ways of reasoning. Uncertainty and indecisiveness may feel similar when you're in that mental state, but they imply different solutions to step forward.

I feel like you already know all there is to know about metaphilosophical disagreements or solution attempts. When I read your posts, I don't feel like "oh, I know more than Wei Dai does." But then you seem uncertain between things that I don't feel uncertain about, and I'm not sure what to make of that. I subscribe to the view of philosophy as "answering confused questions." I like the following Wittgenstein's quote:

[...] philosophers do not—or should not—supply a theory, neither do they provide explanations. “Philosophy just puts everything before us, and neither explains nor deduces anything. Since everything lies open to view there is nothing to explain (PI 126).”

As I said elsewhere, per this perspective, I see the aim of [...] philosophy as to accurately and usefully describe our option space – the different questions worth asking and how we can reason about them.

This view also works for metaphilosophical disagreements. 

There's a brand of philosophy (often associated with Oxford) that's incompatible with the Wittgenstein quote because it uses concepts that will always remain obscure, like "objective reasons" or "objective right and wrong," etc. The two ways of doing philosophy seem incompatible because one of them is all about concepts that the other doesn't allow. But if you apply the perspective from the Wittgenstein quote to look at the metaphilosophical disagreement between "Wittgensteinian view" vs. "objective reasons views," well then you're simply choosing between two different games to play. Do you want to go down the path of increased clarity and clear questions, or do you want to go all-in on objective reasons. You gotta pick one or the other. 

For what it's worth, I feel like the prominent alignment researchers in the EA community almost exclusively reason about philosophy in the anti-realist, reductionist style. I'm reminded of Dennett's "AI makes philosophy honest." So, if we let alignment researchers label the training data, I'm optimistic that I'd feel satisfied with the "philosophy" we'd get out of it, conditional on solving alignment in an ambitious and comprehensive way.

Other parts of this post [EA · GW] (the one I already linked to above) might be relevant to our disagreement, specifically with regard to the difference between uncertainty and indecisiveness. 

If you think it would be helpful, you are welcome to suggest a meta philpsophy topic for AI Safety Camp.

More info at aisafety.camp. (I'm typing on a phone, I'll add actuall link later if I remember too)

At a glance meta-philosophy sounds similar to the problem of what is good, which is normally considered to be within the bounds of regular philosophy. (And to the extent that people avoid talking about it I think it's because the problem of good is on a deep enough level inherently subjective and therefore political, and they want to focus on technical problem solving rather than political persuasion)

What's an example of an important practical problem you believe can only be solved by meta-philosophy?

FWIW I think some of the thinking I've been doing about meta-rationality [? · GW] and ontological shifts [LW · GW] feels like metaphilosophy. Would be happy to call and chat about it sometime.

I do feel pretty wary about reifying the label "metaphilosophy" though. My preference is to start with a set of interesting questions which we can maybe later cluster into a natural category, rather than starting with the abstract category and trying to populate it with questions (which feels more like what you're doing, although I could be wrong).

The classification heading "philosophy," never mind the idea of meta-philosophy, wouldn't exist if Aristotle hadn't tutored Alexander the Great. It's an arbitrary concept which implicitly assumes we should follow the aristocratic-Greek method of sitting around talking (or perhaps giving speeches to the Assembly in Athens.) Moreover, people smarter than either of us have tried this dead-end method for a long time with little progress. Decision theory makes for a better framework than Kant's ideas; you've made progress not because you're smarter than Kant, but because he was banging his head against a brick wall. So to answer your question, if you've given us any reason to think the approach of looking for "meta-philosophy" is promising, or that it's anything but a proven dead-end, I don't recall it.

I'm also interested in this topic but it feels very hard to directly make progress. It seems to require solving a lot of philosophy, which has as its subject matter the entire universe and how we know about it, so solving metaphilosophy in a really satisfying way seems to almost require rationally apprehending all of existence and our place within it, which seems really hard, or maybe even fundamentally impossible(or perhaps there are ways of making progress in metaphilosophy without solving most of philosophy first, but finding such ways also seems hard)

That said I do have some more indirect/outside-view theories which make me think we could obtain a good future even if we can't directly solve metaphilosophy before getting AGI. I think we can see philosophy as the process by which "messy"/incoherent agents, which arose from evolution or other non-agentic means, can become more coherent and unified-agent-like. So obtaining an AI which can do philosophy would not consist of hardcoding a 'philosophy algorithm', but creating base agents which are messy in similar ways to us, who will then hopefully resolve that messiness and become more coherent in similar ways also(and thereafter make use of the universe in approximately as good of a way as we would have). A whole brain emulation would obviously qualify, but I think that it's also plausible that we could develop a decent high-level understanding of how human brain algorithms work and create AIs that are similar enough for philosophical purposes without literally scanning peoples' brains(which seems like it will take too long relative to AI) For this reason and others I think creating sufficiently-human-like AI is a promising route for obtaining a good future(but this topic also seems curiously neglected).

As to why few other people are trying to solve metaphilosophy, I think there are just very few people with the temperament to become interested in such things, then the few that do end up deciding that some other topic has a better combination of importance/neglectedness/tractability/personal fit to invest major effort in.

Where does decision theory come from? It seems to come from philosophers trying to do philosophy.

An alternate view is that certain philosophical and mathematical concepts are "spotlighted", in the sense that they seem likely to recur in a wide variety of minds above a certain intelligence / capabilities level. 

A concept which is mathematically simple or elegant to describe and also instrumentally useful across a wide variety of possible universes is likely to be instrumentally convergent. The simpler and more widely useful the concept is, the more likely it is to occur by convergence in a wider variety of minds, at lower overall capabilities thresholds.

This doesn't necessarily mean that a sufficiently spotlighted concept is "objective" or a philosophical truth, and there might be exotic corners of mindspace in which there are extremely powerful minds which can't or don't converge on these concepts, e.g. the one you give in Metaphilosophical Mysteries [LW · GW].

But note that recognizing that there are recurring useful patterns in mindspace and being able to reason correctly about them (i.e. being able to do philosophy) is itself likely to be an instrumentally useful (though perhaps not very mathematically simple) concept, and is thus itself a spotlighted concept.

(BTW, another place where Eliezer appears to at least allude to his own views on this general topic in his fiction is here.)

As for your meta question of why more people aren't interested in this, one possibility is that there's just not that many people who make it all the way up the ladder of intellectual journey the way you did:

• Lots of people in AI and even AI safety probably haven't heard that much, or aren't that interested in, even the basics of mainstream game theory (Prisoners' Dilemma, Nash Equilibria, etc.), though on LW these concepts have very high penetration.
• Lots of people who have heard of the basics find object-level questions and thought experiments about specific decision theories more than interesting enough to keep arguing about forever.
• Among the remaining people who tentatively accept something like UDT (or at least get far enough to have interesting takes on it), most of them probably don't wonder about the internal mental processes by which its inventors came up with it, nor try to re-create that process in themselves.

Personally, I think something like the spotlighting concept is probably correct, and thus that it is unlikely that practical AGIs will struggle with philosophy.

I do enjoy reading about this general topic though, including your posts and some loosely related things @TsviBT [LW · GW] has written, and I'd like to see more of them, from you and others. I'm not sure that simply drawing attention to the issue will have much effect though - the number of people both qualified and interested in writing about this is probably very small. To make real progress, you might need to bootstrap a larger community of peers. 

Finding ways to move people who are stuck at lower rungs of the ladder through the funnel is one way to do that. This might take the form of more distillations and explanations of metaphilosophy, decision theory, and other object-level concepts that target a wider audience.  (Or, if you're inclined, write 1M+ words of collaborative fiction [LW · GW] about these topics.)

I'm super grateful to have stumbled across someone who also cares about meta-philosophy! I have an intuition that we don't understand philosophy. Therefore, I think its advantageous to clarify the nature, purpose, and methodologies of philosophy, or in other words, solve meta-philosophy. 

Let's explore some questions...

1. What would it look like for a civilization to constantly be solving problems but not necessarily solving the right problems?
2. How does reflection and meta-cognition relate to finding the root problems?
3. What if our lack of reflection and meta-cognition is the root problem?
4. What if we defined philosophy as civilization level reflection and meta-cognition as opposed to the traditional interpretation?

What would it look for a civilization to constantly be solving problems but not necessarily solving the right problems?

Summary

Scenario: A civilization focused on constant problem-solving without addressing the right issues.

Characteristics: Quick fixes, frequent shifts in focus, and neglect of underlying, systemic problems.

Implications

This scenario would involve a civilization that frequently engages in reactive measures--tackling symptoms rather than root causes. This could lead to a cycle of temporary solutions and recurring issues, with resources being spent on problems that may not contribute to long-term stability or improvement. The focus on immediate, superficial, problem-solving could prevent effective handling of more significant, complex challenges, leading to unintended consequences.

How does reflection and meta-cognition relate to finding the root problems?

Summary

Role of Reflection: Helps revisit and analyze past actions and outcomes.

Role of Meta-cognition: Enables critical evaluation of decision-making processes. 

Implications

Reflection allows for a retrospective analysis where past decisions and their impacts are reconsidered. This process aids in uncovering patterns and recurring problems, providing insights into the underlying causes of issues rather than merely their manifestations. It's about looking back to understand what happened and why.

Meta-cognition, on the other hand, is a deeper layer of thinking about one's thinking. It involves examining the cognitive processes used in problem solving and decision-making. This includes questioning one's assumptions, biases, and frameworks within which decisions were made. Meta-cognition helps in recognizing the limitations and flaws in these processes, which might contribute to ongoing problems.

What if our lack of reflection and meta-cognition is a root problem?

Summary

Potential Root Problem: Lack of reflection and meta-cognition.

Consequences: Poor decision-making, repeated errors, and superficial solutions.

Implications

If the absence of reflection and meta-cognition is identified as a root problem, this has broad implications for how issues are addressed and prevented across various domains.

1. Poor Decision-Making: Without sufficient reflection, decisions may be based on incomplete information or immediate pressures, rather than well-considered analysis. Meta-cognition is crucial for evaluating the thought processes behind decisions, including the recognition of biases, assumptions, and gaps in knowledge. 
2. Recurring Problems: A lack of deep thinking and questioning means that solutions tend to be reactive and symptom-based. Without addressing deeper causes, the same or similar problems are likely to reoccur, indicating that the interventions are not reaching the underlying issues. 
3. Superficial Solutions: Solutions developed without the aid of reflection and meta-cognitive processes tend to address manifestations of a problem. This can lead to a cycle of short-term fixes that fail to tackle more complex, systemic issues.

What if we defined philosophy as civilization level reflection and meta-cognition as opposed to the traditional interpretation?

Summary

Philosophy as Reflection and Meta-cognition: Envisioning philosophy primarily as a reflective and meta-cognitive practice emphasizes active and ongoing analysis of thought processes and decision-making.

Challenges with Traditional Philosophy: The conventional focus on static fundamental questions may limit philosophy's practical effectiveness and relevance to contemporary issues. 

Implications

If philosophy is framed a dynamic practice of reflection and meta-cognition, rather than merely an academic pursuit of answering age-old questions, it becomes a more vital, active part of addressing modern challenges. This perspective shifts philosophy from being an abstract discipline into a practical tool that continually assesses and adapts our thinking strategies, decision-making processes, and the very frameworks we use to interpret the world. 

Closing Thoughts

To me...solving meta-philosophy looks like devising a framework that explains these questions and their relationships in an accessible way. 

What is thinking about our thoughts?

Why do we think about our thoughts?

How do we think about our thoughts?

After that, we could have even more fun and start to dissect the collective worldview that underpins our institutions, identify imbalances, and imagine an alternative that feels more authentic and ecologically harmonious. 

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

Hello Wei Dai,

Something that might be useful to a certain degree might be to see it through the lens of Collective Intelligence.
Or simply that the sum is greater than the value of the individual parts, and also that we can synch our efforts together, directly or indirectly, with the people around us. A recent BBC reel explores this as well. 

Like you say, as you 'move on', you leave many behind. - But at least you will feel massively more vital and see an increase in growth if you have someone that enhances your learning directly through "being on the same wavelength/page" - who not only 'says' they have the same goals, but that works towards them at the same level of depth and inner coherence - as yourself.
Wanting 'immediate' and 'visible' results is pretty understandable as well, and not only waiting for the indirect changes that might occur through the different forms of collective intelligences you have already outlined, but that might not really dive into things the same way you want/are able to.

If so, an answer to the questions isn't necessarily one of finding the right answers, but finding the right people to ask that want to find an answer. People that are aware of the problem at the complexity-level where you can find a suitable solution, and that are willing to structure their individual mind, will and effort around solving it, as well as synchronizing their efforts with you in a way that has inner coherence. That would probably also feel more stimulating than only indirectly looking at these challenges. 

Was wondering how that lands with you, and if this is a useful direction to you, or if you want something else? If you have questions related to this in particular or something, I'm up for continuing on DM. 

Kindly, 
Caerulea-Lawrence

I'm currently investigating the moral reasoning capabilities of AI systems. Given your previous focus on decision theory and subsequent shift to Metaphilosophy, I'm curious to get your thoughts.

Say an AI system was an excellent moral reasoner prior to having especially dangerous capability. What might be missing to ensure it is safe? What do you think the underlying capabilities to getting to be an excellent moral reasoner would be ?

I am new to considering this as a research agenda. It seems important and neglected, but I don’t have a full picture of the area yet or all of the possible drawbacks of pursuing it.

Thanks for the post! I just published a top-level post responding to it: https://www.lesswrong.com/posts/pmraJqhjD2Ccbs6Jj/is-metaethics-unnecessary-given-intent-aligned-ai [LW · GW]

I'd appreciate your feedback!

Why is there virtually nobody else [EA(p) · GW(p)] interested in metaphilosophy or ensuring AI philosophical competence (or that of future civilization as a whole), even as we get ever closer to AGI, and other areas of AI safety start attracting more money and talent?

I've written a bit on this topic that you might find interesting; I refer to it as the Set of Robust Concepts (SORC) [LW · GW]. I also employed this framework to develop a tuning dataset, which enables a shutdown mechanism [LW · GW] to activate when the AI's intelligence poses a risk to humans. It works 57.33% [LW · GW] of the time.

I managed to improve the success rate to 88%. However, I'm concerned that publishing the method in the conventional way could potentially put the world at greater risk. I'm still contemplating how to responsibly share information on AI safety, especially when it could be reverse-engineered to become dangerous.