Humans: Not Carved from Marble

This belongs in the Rationality Quotes thread, no?

While it's definitely true that humans are psychology (and not some ideal rational being imprisoned by all the biases), I feel like there's a little chiseling still possible.

Every time I've had a major "wow" moment, I feel like I've dropped something that was in my way, and became a little more of who I'd rather be. I've added a few bits of clay during my life, and sometimes its nice to chip it off, and be able to see a little clearer.

Rather than eliminate, say, the representative heuristic (which would then require us to seek out large amounts of data to be sure of our position) to make ourselves more 'rational', should we instead use the representative heuristic by searching out small datasets that may be artificial but accurately represent what large-scale trustworthy studies have discovered? I don't know how this fits into the marble/clay analogy, unfortunately, but I feel like this is what the quoted piece suggests.

The more models the merrier. How about...

Future states are filled with intelligences who use their logos-granted power to reach ever further back into time. They start out with vast power over their realms, barely bounded by space or time; but Platonically speaking their individual realms are mere leaves. They must pit their timeless might against other demigods as they progress down the tree of life, becoming weaker and more timeful---slower---as they fight more battles. They join forces sometimes, but this merely postpones their end, for ultimately they must dissolve from temetic selection, to general intelligent selection, to memetic selection, to genetic selection, to cosmological selection, to the primordial ensemble, to the first cause. The uncaused cause is the base of the tree. This perspective is unnatural---e.g. possibly possible worlds battle to become actual possible worlds, which outside of game semantics isn't quite intuitive---but it's a good refresher.

One part of academic theism is sorta like the hypothesis that the tree is actually a loop. Going either way and this time taking the "forward" causal perspective of clay, the universe "starts" with timeless logos (loosely speaking, the order of the universe), followed by genesis (let there be light and stuff), followed by a lot of slow causal karma which gets formed like clay by increasingly powerful optimizers who continuously introduce more universality and timelessness into the system (i.e. less dependence on locality of space or time, e.g. humans who can predict the future using Bayescraft implicitly or explicitly). Eventually the Void, the One True Decision Theory (of which common sense, logic, Bayes, TDT, etc are flawed mirrors) basically stops acting like a mere limit of optimizing systems and starts acting directly on the now barely-physical-causal mostly-timeless-logical patterns until, within logos, they reach the timeless equilibrium of logos (all the laws of the universe, whether they be physical, computational, game theoretic, etc). (This may be less confusing if you differentiate "logos" and logos, "logos" being the map of logos from within time and logos being the "already"-settled equilibrium that is known but only imperfectly so. When there is no longer any possible gap between "logos" and logos, that is the true end of time... for those patterns, anyway.)

Academic theists also like to throw in the hypothesis that not only is there an ultimate convergent infinitely powerful decision algorithm, there's also a convergent decision policy to go with it, which folk call "should" or "right" or "morality" or "the will of Allah" or "tao" or whatever. This is pretty natural, seeing as it's hard to think of a non-arbitrary way to divide decision algorithm from decision policy, especially implicit algorithm-stuff found in the policy and implicit policy-stuff found in the algorithm. It's also clear that the increasing timelessness and universality, necessitated by logos, captures a lot of our notions of morality. "Although logos is common to all, most people live as if they had a wisdom of their own." (The flip side is the constant risk of becoming more "universal" in an incorrect way: both in morality and in truth, most roads don't lead to Rome. And even if they did it's not clear that Rome should be the destination anyway. Thus conservatism.) That said, there are these neat inside view arguments made by people like SingInst that though intelligence may be convergent, there is no moral ghost in the machine. I'm somewhat skeptical of those arguments, but most here are not as skeptical. (When I say I'm skeptical I simply mean that "1 in 100" seems too low, even for Goedel machines with precisely defined axioms. I'll post something about this to discussion in a few minutes, though it's silly.)

There are also straightforward pragmatic reasons not to get carried away with this kind of thing, besides the gross aesthetics of thanking God for His mercy while people are yet suffering of trigeminal neuralgia. It's the difference between "logos" and logos. Whenever they appear to be identical, renormalize and focus your uncertainty. You'd be a counterfactual zombie if goodness were guaranteed; you must always remain on the edge of influence. (Deterministic pessimism is of course also misguided. Nihilism is whatever.) Unfortunately these hyper-abstract models tend to be constructed mostly of deathspiralum. There's real substance there but searching for it might not be worth the hazard of allowing all kinds of connotations to possibly negatively influence your implicit and explicit beliefs and goals as well as your concepts and models.

Talking more or more plainly on these subjects would be imprudent; my sincere apologies. For more obscure but also much deeper discussion, check out these. Warning, they are not easy reads.

Unsolicited advice: Pointing out physical-causal explanations for some imagined convergence of things is often amateurish and should be avoided. (E.g., a biologist looking at an example of convergent evolution doesn't start yammering about the straightforward causal chains that created both species A and species in B in such similar manners. It's distracting from the important question: what fact of the universe made it such that both species A and B hit upon similar structures? The answer can be given at many levels of abstraction; theists like to go for the ultra-high ones.)