What should experienced rationalists know?

This is an excellent question. Here's some of the things I consider personally important.

Regarding probability, I recently asked the question: Why is Bayesianism Important? [LW · GW] I found this Slatestarcodex post to provide an excellent overview of thinking probabilistically, which seems way more important than almost any of the specific theorems.

I would include basic game theory - prisoner's dilemma, tragedy of the commons, multi-polar traps (see Meditations on Moloch for this later idea).

In terms of decision theory, there's the basic concept of expected utility, decreasing marginal utility, then the Inside/Outside views [? · GW].

I think it's also important to understand the limits of rationality. I've written a post on this (pseudo-rationality [LW · GW]), there's Barbarians vs. Bayesians [LW · GW] and there's these two posts by Scott Alexander -  Seeing as a State and The Secret of Our Success. Thinking Fast and slow has already been mentioned.

The Map is Not the Territory [LW · GW] revolutionised my understanding of philosophy and prevented me from ending up in stupid linguistic arguments. I'd suggest supplementing this by understanding how Conceptual Engineering [LW · GW] avoids the plague of counterexample philosophy prevalent with conceptual engineering (Wittgenstein's conception of meanings as Family Resemblances is useful too - Eliezier talks about the cluster structure of thingspace [LW · GW]).

Most normal people are far too ready to dismiss hypothetical situations [? · GW]. While if taken too far Making Beliefs Pay Rent [LW · GW] can lead to a naïve kind of logical positivism, it is in general a good heuristic. Where Recursive Justification Hits Bottom [LW · GW] argues for a kind of circular epistemology.

In terms of morality Torture vs. Dust Specks [LW · GW] is a classic.

Pragmatically, there's the Pareto Principle (or 80/20 rule) and I'll also throw in my posts on Making Exceptions to General Rules [LW · GW] and Emotions are not Beliefs [LW · GW].

In terms of understanding people better there's Inferential Distance, Mistake Theory vs. Conflict Theory, Contextualising vs. Decoupling Norms [LW · GW], The Least Convenient Possible World [LW · GW], Intellectual Turing Tests and Steelmanning [? · GW]/Principal of Charity.

There seems to be an increasingly broad agreement that meditation is really important and compliments rationality beautifully insofar as irrationality is more often a result of lack of control over our emotions, than lack of knowledge. But beyond this, it can provide extra introspective capacities and meditative practises like circling can allow us to relate better with humans.

One of my main philosophical disagreements with people here is that they often lean towards verificationism, while I don't believe that the universe has to play nice [LW · GW] and so that often things will be true that we can't actually verify.

I'll answer with some of the "rationalist" ideas I've personally learned and consider important. (I expect that I still have important knowledge gaps compared to other experienced rationalists though. I'm working on it.)

Intelligence is not the same thing as rationality. [? · GW] So much of "Epistemic Rationality" comes down to not lying to yourself. Clever people tell themselves clever lies. The Litany of Tarski is the correct attitude for a rationalist.

Raising the Sanity Waterline [LW · GW] is a mini-sequence in its own right, if you read its links. The worldviews taught by all major religions are extremely confused at best—including (especially) yours, if you still have one—and if this isn't already blatantly obvious, your epistemology is very broken! People are crazy, the world is mad.

Death is Bad.

There are No Guardrails. [LW · GW] The Universe is crushingly indifferent to human well-being. Everything is allowed to go horribly wrong.

Rationalists Should Win. Or "Instrumental Rationality". This does not mean "Go team!" It means that if you're not winning, then you're doing rationality wrong, and you should correct your errors, regardless of what the "rationalists" or the "great teacher" think. This can make "rationality" hard to pin down, but the principle of correcting one's errors is very important because it catches a lot of failure modes on the path. Then why not just say, "Correct your errors"? Because there are a lot of ways of misidentifying errors, and "not winning" cuts through all that and gets to the heart of what errors are.

You have to be willing to be correct even when that makes you weird. [LW · GW] Doing better than normal is, by definition, abnormal. But the goal is correctness, not weirdness. Reversed stupidity is not intelligence.

Understand Social Status. [? · GW]

Value your Slack. [LW · GW] Guard it. Spend it only when Worth It. The world is out to get your Slack! If you lose it, fight to get it back!

You Need More Money [LW · GW]. I wrote this one, but it's a synthesis of earlier rationalist thought.

The mountains of philosophy are the foothills of AI. Even if you're not planning to become an AI researcher yourself, understanding how to see the world through the lens of AI clarifies many important things.

Read textbooks. [LW · GW] (And you can download almost all of them if you know where to look.)

Google how not to suck at X. [LW · GW]

Econ in general, no. The specific model of rational actors optimizing for outcomes, the intuition for why markets often succeed at delivering on desires (at least for those with money), and the practice of making multi-stage models and following an impact in one area through to others, yes. Nobody needs macro. Lots of people need to reflexively think of how other actors in a system respond to a change, and econ is one of the more effective ways of teaching this. Critical if you want to actually have a good understanding of multi-polar scenarios. What I'm talking about is rarely adequately understood just by studying game theory, which usually analyzes games too simple to stick in student's minds, but it matters.

In addition to understanding what statistics are on a theoretical level, an intuitive understanding that getting the right numbers yields real benefits, and that the right numbers are very precisely defined, seems important.

These are both softer skills, but I think that they're important.

I still think that this problem is intractable so long as people refuse to define 'rationality' beyond 'winning'.

https://www.thelastrationalist.com/rationality-is-not-systematized-winning.html

I, in general, try to avoid using the frame of 'rationality' as much as possible precisely because of this intractability. If you talk about things like existential risk, it's clearer what you should know to work on that.

See my caveats in the comment section, so those being said, I'd say the most useful thing I know/read on my path to being something of an "experienced" rationalist was Godel, Escher, Bach. The older I get the more I realize how much was in that book that set me down my path, and yes there were lots of opportunities to get various things wrong and confused along the way, but in the end I think it might be the single best source I've seen that might turn someone to the rationalist mindset if they really grok what it has to say.

I have a problem with the short term existence of real AI. I am new to this site and likely will discuss topics many are familiar with. I don't know if self consciousness can be created artificially through code influenced by even the smallest amount of human emotion and unconscious irrationality. Humans evolved through conditions that, when it comes down to it, were based in objective reality. Even if an "irrational" event threating existence occurred, it is safe to say that those influenced the most by objective reason survived, even if the example of "rationality" sounds insane to us today. It makes sense naturally that an organism rejecting the innumerable physical laws guiding our universe(most of which are unknown to our "higher level of consciousness") are the least likely to propagate. I almost think there is an unwritten physical law, too hard to prove at the moment through any known scientific means, that states one consciousness can not create another species/ mind with  a consciousness grounded in intelligence vastly superior to its creators. I don't think it is possible to teach a computer 2+2= 4 and then attempt to teach it to learn from an algorithm that isn't written, whether consciously or unconsciously, to answer questions 100% coinciding w/ objective physical/ mathematical reality, and expect it will develop a superior and even threatening consciousness. If a consciousness does develop it will be through drawing from info in a corrupted/unrepresentative data set. It will be extremely flawed and easy to combat with any objective human reasoning. Again, I am new to this site and likely wrong. I'm hoping to hear a perspective that uses either empirical evidence or basic logic to reject my hypothesis.

Computational complexity theory (if only the rudiments), I think.