The main thing I got out of reading Bostrom's Deep Utopia is a better appreciation of this "meaning of life" thing. I had never really understood what people meant by this, and always just rounded it off to people using lofty words for their given projects in life.

The book's premise is that, after the aligned singularity, the robots will not just be better at doing all your work but also be better at doing all your leisure for you. E.g., you'd never study for fun in posthuman utopia, because you could instead just ask the local benevolent god to painlessly, seamlessly put all that wisdom in your head. In that regime, studying with books and problems for the purpose of learning and accomplishment is just masochism. If you're into learning, just ask! And similarly for any psychological state you're thinking of working towards.

So, in that regime, it's effortless to get a hedonically optimal world, without any unendorsed suffering and with all the happiness anyone could want. Those things can just be put into everyone and everything's heads directly—again, by the local benevolent-god authority. The only challenging values to satisfy are those that deal with being practically useful. If you think it's important to be the first to discover a major theorem or be the individual who counterfactually helped someone, living in a posthuman utopia could make things harder in these respects, not easier. The robots can always leave you a preserve of unexplored math or unresolved evil... but this defeats the purpose of those values. It's not practical benevolence if you had to ask for the danger to be left in place; it's not a pioneering scientific discovery if the AI had to carefully avoid spoiling it for you.

Meaning is supposed to be one of these values: not a purely hedonic value, and not a value dealing only in your psychological states. A further value about the objective state of the world and your place in relation to it, wherein you do something practically significant by your lights. If that last bit can be construed as something having to do with your local patch of posthuman culture, then there can be plenty of meaning in the postinstrumental utopia! If that last bit is inextricably about your global, counterfactual practical importance by your lights, then you'll have to live with all your "localistic" values satisfied but meaning mostly absent.

It helps to see this meaning thing if you frame it alongside all the other objectivistic "stretch goal" values you might have. Above and beyond your hedonic values, you might also think it good for you and others to have objectively interesting lives, accomplished and fulfilled lives, and consumingly purposeful lives. Meaning is one of these values, where above and beyond the joyful, rich experiences of posthuman life, you also want to play a significant practical role in the world. We might or might not be able to have lots of objective meaning in the AI utopia, depending on how objectivistic meaningfulness by your lights ends up being.

Considerations that in today's world are rightly dismissed as frivolous may well, once more pressing problems have been resolved, emerge as increasingly important [remaining] lodestars... We could and should then allow ourselves to become sensitized to fainter, subtler, less tangible and less determinate moral and quasi-moral demands, aesthetic impingings, and meaning-related desirables. Such recalibration will, I believe, enable us to discern a lush normative structure in the new realm that we will find ourselves in—revealing a universe iridescent with values that are insensible to us in our current numb and stupefied condition (pp. 318-9).

Use your actual morals, not your model of your morals.

The human brain does not start out as an efficient reasoning machine, plausible or deductive. This is something which we require years to learn, and a person who is an expert in one field of knowledge may do only rather poor plausible reasoning in another. What is happening in the brain during this learning process?

Education could be defined as the process of becoming aware of more and more propositions, and of more and more logical relationships between them. Then it seems natural to conjecture that a small child reasons on a lattice of very open structure: large parts of it are not interconnected at all. For example, the association of historical events with a time sequence is not automatic; the writer has had the experience of seeing a child, who knew about ancient Egypt and had studied pictures of the treasures from the tomb of Tutankhamen, nevertheless coming home from school with a puzzled expression and asking: ‘Was Abraham Lincoln the first person?’

It had been explained to him that the Egyptian artifacts were over 3000 years old, and that Abraham Lincoln was alive 120 years ago; but the meaning of those statements had not registered in his mind. This makes us wonder whether there may be primitive cultures in which the adults have no conception of time as something extending beyond their own lives. If so, that fact might not have been discovered by anthropologists, just because it was so unexpected that they would not have raised the question.

As learning proceeds, the lattice develops more and more points (propositions) and interconnecting lines (relations of comparability), some of which will need to be modified for consistency in the light of later knowledge. By developing a lattice with denser and denser structure, one is making his scale of plausibilities more rigidly defined.

No adult ever comes anywhere near to the degree of education where he would perceive relationships between all possible propositions, but he can approach this condition with some narrow field of specialization. Within this field, there would be a ‘quasi-universal comparability’, and his plausible reasoning within this field would approximate that given by the Laplace–Bayes theory.

A brain might develop several isolated regions where the lattice was locally quite dense; for example, one might be very well-informed about both biochemistry and musicology. Then for reasoning within each separate region, the Laplace–Bayes theory would be well-approximated, but there would still be no way of relating different regions to each other.

Then what would be the limiting case as the lattice becomes everywhere dense with truly universal comparability? Evidently, the lattice would then collapse into a line, and some unique association of all plausibilities with real numbers would then be possible. Thus, the Laplace–Bayes theory does not describe the inductive reasoning of actual human brains; it describes the ideal limiting case of an ‘infinitely educated’ brain. No wonder that we fail to see how to use it in all problems!

This speculation may easily turn out to be nothing but science fiction; yet we feel that it must contain at least a little bit of truth. As in all really fundamental questions, we must leave the final decision to the future.

--E. T. Jaynes, Probability Theory (p. 659-60)

Minor spoilers for planecrash (Book 3).

Keltham's Governance Lecture

Keltham was supposed to start by telling them all to use their presumably-Civilization-trained skill of 'perspective-taking-of-ignorance' to envision a hypothetical world where nothing resembling Coordination had started to happen yet.  Since, after all, you wouldn't want your thoughts about the best possible forms of Civilization to 'cognitively-anchor' on what already existed.

You can imagine starting in a world where all the same stuff and technology from present Civilization exists, since the question faced is what form of Governance is best-suited to a world like that one.  Alternatively, imagine an alternative form of the exercise involving people fresh-born into a fresh world where nothing has yet been built, and everybody's just wandering around over a grassy plain.

Either way, you should assume that everybody knows all about decision theory and cooperation-defection dilemmas.  The question being asked is not 'What form of Governance would we invent if we were stupid?'

Civilization could then begin - maybe it wouldn't actually happen exactly that way, but it is nonetheless said as though in stories - Civilization could then begin again, when people envisioned running out of stored food a couple of years later.  Standing around all these beautiful complicated machines that people remembered how to operate, but required multiple people working together to operate, which nobody was yet incentivized to operate.

Or Civilization could begin for the first time, when the Hypothetical Newly-Created Educated People imagined trying to build shelters for themselves, or sow food-plants to grow; and thought to themselves that there would be less point in doing that, if others would just move into the shelters as soon as they walked away, or eat the crops that they had sown.

And people then would say to themselves, "What if we tried something else which is not that?"

It continues into a new problem, the problem of motivating such socially-useful actions as 'producing food', for if nobody does this, soon nobody will eat.

You can imagine lesser solutions, collective farming of collectively guarded fields, monitors on hard work and rewards of food access.  But these are simultaneously too 'simplistic' and 'overcomplicated', the very opposite of an 'elegant-solution'.  People can work harder, invest more effort, for a usually 'monotonically-increasing' reward, a function operated directly by the Environment, by 'physical-law'.  There just needs to be some system whereby, when people work, they are themselves the ones to benefit from it.

But this requires a far more complicated form of coordinated action, something that 'bounded-agents'  lack the computational power to consider as a giant macroaction of their 'collective-agent'.  The optimal macrostrategy must be lossily projected down into simplified mental rules for individuals, a notion of imaginary-ownership-tagging: if one person sows food-plants within a field, and waters them and protects them, everybody around them will behave as if the resulting food-crop is tagged with an imaginary pointer to that person, saying that the food may be consumed by them alone.  Or only consumed by those others the food's 'owner' designates, at their own decision... that seems like it should obviously be an option built into the system too...

And once you create an imaginary structure of coordinated action that elegantly-complicated, the consequences and further-required-features inevitably explode; the explosion that results is Civilization's basic form nearly in toto.

People could often benefit from other people doing various things for them, but they must of course do something for the other in exchange.  If things have socially-constructed tags pointing to people, who alone may use or consume those things, why not let people announce that the pointer now points to someone else?  That's one way of doing something in return, for somebody who did some task for you, that was easier for them than for you.

If fields can be owned and an owned field produces owned produce in the future, why not let people announce that some of the future produce can point to some other owner?

Often the announcements of changed imaginary ownership are meant to be traded, executed one in exchange for another.  Then a new version and feature-expansion of the system can eliminate the uncertainty about whether the other will announce their bargained ownership change, after you announce yours: imaginary contracts, that molecularize the atomic actions into a transaction that executes simultaneously on both sides, only after both sides announce the same contract.

Do people want to work on some larger endeavor - specialize in different aspects of farming, and collectively challenge a larger farm?  Let the tags, in the eyes of society, point to persistent imaginary constructs, specified in some contract specification language; a corporation is one such persistent contract.

Let this system expand, let people use it enough, and there will predictably come a point where there aren't lots of untagged resources nearby for somebody to tag in society's eyes.

Once there are not plenty of new plots of land to tag and farm, people may indeed begin to ask, 'Why should this land be owned by them and not me?'

Because they did some work on that land?  If that's the rule, then won't people who foresee the predictable scarcity later, run around trying to plow small shallow furrows through every potential field within the reach of running, trying to tag as much land as pointing to themselves as possible?

And when all that land has been used up, wouldn't the people who were slower runners and didn't end up with any land - wouldn't new children born into this world, for that matter - ask themselves and perhaps ask out loud:

"If this elaborate imaginary social construct doesn't offer me any benefits for going along with the pretense - if the system says that little or nothing has an imaginary tag pointing to me - then in what sense is this even coordination, from my perspective?  Why would I cooperate in the coordinated rule of not eating things tagged as pointing to others, if the result is that there's nothing for me to eat?  Where's my fair share of the rewards for playing along with this pretend game, for cooperating with what this imaginary tagging system says is my part and my action in it?"

This concept, incidentally, took some arguments to persuade into tiny Keltham, when he was first hearing about all this.  Tiny Keltham had a very strong instinctive sense that objects just were owned by people, and that what made a system fair and right was entirely that only the people who owned objects could do anything with them or transfer them to other people.

It was hard for tiny Keltham, at first, to see past his instinctive suspicion that people asking 'What's my reward for cooperating with this system?' were about to use that as an excuse to storm onto his hypothetical farm and eat his food that he'd worked to produce, and call that their share, without doing any work themselves.

Older children's attempted arguments about 'put yourself into that other person's shoes' repeatedly failed on Keltham, who kept replying that he wouldn't take anybody else's stuff period.

But tiny Keltham was eventually persuaded - by a Watcher, then, not by an older child - by the argument that it is an internally-consistent imaginary tagging system to say that some single person Elzbeth owns all the land in the world.  Everybody else has to work those lands and give Elzbeth a share of anything that grows there, since by default it would just end up tagged as hers, unless they agree to pay half their gains to her.

The question then becomes, why should anybody else except Elzbeth play along with this imaginary system, once it gets to that point?  Why shouldn't everyone who isn't Elzbeth, all just wake up out of this bad dream, and do something else which is not that?

Keltham asked if maybe the system had started out with everybody owning an equal amount of land, but Elzbeth had been a really clever asset-trader and ended up owning everything in the world after a series of voluntary transactions; in which case it seemed to him that fair was fair.

The Watcher told Keltham that, even if the last generation had gotten the world into that state through a series of voluntary transactions, the children born into it might look around and see that no land was tagged to them, that everything was tagged to Elzbeth.  They would ask what they were receiving in exchange for playing along with that particular delusion, and why they should not imagine some other tagging system instead, in which their coordinated action in playing along would actually receive any reciprocal benefit or reward.

And tiny Keltham growled and stomped around for a while, but finally conceded that, fine, the pointers were imaginary and yes it took more than just a consistent tagging system running on strictly voluntary transactions to make the whole thing be fair or right.  The elegant core structure was necessary-but-not-sufficient.

The unimproved land, the raw resources, these things must be tagged with ownership for the owners to have an incentive to improve them.  It doesn't mean that this tagging need be considered as free to the new owner.

Discard the obvious-first-solution-that-is-wrong of charging somebody an amount of food or other worked goods, to tag previously untagged land, and redistributing those payments equally among everyone in the world.  Even leaving aside the question of how that system initially starts farming anything at all, it inevitably arrives at a point where there's no untagged land left or it's impossibly expensive.  Whereupon the next generation of children, being born with no land tagged to them and no payments for newly bought land coming in, will again begin to ask, "Why should I play along with this imaginary arrangement at all; where's my payoff for coordinating my action with yours?"

More sensible then to regard people as renting land and other raw-resource sources, at their unimproved price of course, but still an unimproved price set by competitive bidding - albeit perhaps for long-term leases, etcetera.

When you are born, you conceptually acquire a share in this whole system -

Of course tiny Keltham immediately demanded his accumulated profits from his share of all the land-rents in the world, and demanded also to know why he had never been told about this before.

The Watcher again had to be brought in to explain to Keltham that, conceptually speaking, his share was mostly going into maintaining a lot of non-rival non-excludable goods, or services that Civilization thought should be provided to literally everyone even if in principle they weren't public goods.  The value of unimproved land wasn't as high as Keltham was imagining in the first place; dath ilan still had whole forests just lying around not being used for anything -

Tiny Keltham said that he had absolutely not consented for his share of the land rents to be used for non-rival non-excludable anything, and from now on he wanted it delivered to him in the form of actual money he could spend on what he wanted.

…could he please wait and listen to the whole story before getting angry? said the Watcher.

Tiny Keltham was incredibly suspicious, but he did already have a great deal of experience with adult craziness turning out to be more reasonable than he had at first thought.  Tiny Keltham agreed to go on listening for a while longer, then, before he started trying to persuade all the other children that they ought to band together and overthrow Civilization to get their fair share of the land rents, in the form of actual money, delivered to them right now.

Because, you see - it was said to tiny Keltham - returning to the Hypothetical Newly-Created Educated People, at some point their system is going to grow large enough that even with everybody receiving benefits for their participation in the system, there will still be defectors.  There will be people who just don't want to go along with the system, and try to eat food with a tag on it that points to somebody else.

Now the nascent Civilization needs police that can outfight any individual thief; and, since Newly-Created Educated People aren't stupid, they know they obviously need to ensure that the collective of all of them can always outfight the police.  Neither of these 'features' are cheap, and neither easily lend themselves to private ownership -

Tiny Keltham said that he'd be happy to pay for the police to protect him, out of his share of the land-rent, once it was being paid to him in actual money, and he didn't see why Governance had to take his money and use it without his permission supposedly to protect him with police.

Why couldn't people just pay for police who sold their services on the market like everybody else?  Or if it was much more efficient to police larger regions at once, why couldn't his sub-city in Default provide police, and then Keltham would help his parents pay their share of the house-rent out of his share of the land-rent being paid to him directly -

Because the police have weapons, tiny Keltham!  What if the police for a sub-city decide one day that, in this sub-city, it's fine to raise kids of Keltham's age, and force them to work in exchange for only enough food to keep them alive?  What if the police decide that nobody in the city block is allowed to hire different police, and take all their stuff to make sure they can't afford them?  What if somebody dies and their head doesn't get cooled and preserved fast enough?  That's the kind of thing that Civilization as a whole has to prevent, as a universal regulation with no opt-outs, so it doesn't happen anywhere in Civilization.  Even if you thought people should be able to opt-out of any and every protection as adults, you'd still have to check to make sure they weren't having kids.

In fact, tiny Keltham, your subcity does contract with a police agency to do a lot of ordinary policing, and it does appear in your parents' rent on the 'foundation' for their 'house-module'; but Governance has to provide oversight of that policing, and that costs money.  Cryosuspension emergency response on constant standby service costs money.  Protecting the Waiting Ones in their frozen sleep costs money.  Maintaining the election system to pick people to run the Government that regulates armed police costs money.  Quiet Cities to host the 5% of people who can't be happy working in Civilization, and who are thus held injured by what Civilization has chosen to become, cost actually quite a lot of money.  No, most people won't need the Quiet option; but everyone, when they're born, can be considered as needing an insurance policy against that happening to them, and that insurance policy costs money.

Subsidized policy-prediction_markets to make all those institutions work boundedly-optimally cost money.

When you add up everything like that, which Governance has to do for everyone or can't just sell to individuals separately, it actually does eat all the rent of unimproved land plus all of Governance's other revenue streams.  Lots of individual philanthropists fund Governance on top of that, so that Civilization can have a bigger Government than basic rents alone will support - so that there can be Annual Alien Invasion Rehearsal Festivals, say, or so that Quiet Cities can have nicer things.

(Most philanthropies in Civilization with room for more funding accomplish roughly the same amount of marginal good per marginal labor-hour, according to most people's utility functions.  If you've got a fairly conventional philanthropic utility function, you get to pick whichever random charity or impact-certificate market best matches your personal taste there, including just throwing your money at Governance.  It's like buying individual-stock equity investments; there's more volatility, but all the expected returns are the same.)  (In Civilization, that is.)

Tiny Keltham demanded to see the actual accounts of which 'essential public services' his share of the land-rent was getting spent on.

He was promptly provided them, in easy-to-read format for children with lots of helpful pictures.  Lots of dath ilani children demand to see those accounts, at some point.

With great focus and concentration, tiny Keltham managed to read through twenty-two pages of that, before getting bored and running off to play.

(This placed him in the 97th percentile for how far most children read at that age.)

The explanation to tiny Keltham resumed the next day with the workings of Governance.

Conceptually and to first-order, the ideal that Civilization is approximating is a giant macroagent composed of everybody in the world, taking coordinated macroactions to end up on the multi-agent-optimal frontier, at a point along that frontier reflecting a fair division of the gains from that coordinated macroaction -

Well, to be clear, the dath ilani would shut it all down if actual coordination levels started to get anywhere near that.  Civilization has spoken - with nearly one voice, in fact - that it does not want to turn into a hivemind.  This is why 'dath ilan' deliberately doesn't have Baseline's agency-marker on it, like the name of a person; dath ilan is not allowed to become a person.  It is high on the list of Things Dath Ilan Is Not Allowed To Do.  There was a poll once - put forth either by wacky trolls or sincere negative utilitarians - over how many people would, if they were voting on it directly, vote to put the agency-marker back into 'Dath Ilan'; 98% of respondents said no.

Dath ilan has decided to definitely not turn into a hivemind.  If it ever starts to get even close to that, everyone in Civilization will decide in nearly unanimous accord that they would rather do something else which is not that, and end up not there.  Conformity levels are bad enough already, according to their democracy's collective vote on the desired levels of that!  And predicted to get slightly worse over the next 10 years, according to the prediction markets that aggregate all of Civilization's knowledge into a single opinion that represents what Civilization as a whole can be said to know about any future observable, which few sane people would dare to question too much even in the privacy of their own thoughts!

But for moral purposes, for purposes of understanding what 'Civilization' represents as a decision individuals make to coordinate among themselves, it represents moving partway towards aggregating all coordinating parties in dath ilan into one macroagent that weighted-sums their utility functions together, at a weighting that ends up giving every subagent a fair share of the gains according to their individual utility functions.

If something like this macroagent actually existed, any time the macroagent faced a decision it had to make globally for one reason or another, it would make that decision in a way that reflected the preferences of everybody inside it.  "Nobody anywhere gets to run a city where some children don't get to learn Baseline, even for the noblest possible purposes of scientific experimentation to see what happens if you raise kids speaking only your new improved language instead" - this is a decision made over everywhere; if there's any loophole somewhere, something will happen that most people in Civilization think should not happen anywhere.

(This example, to be clear, was selected on the basis of its controversy; propositions like "all children get to learn some human language during their critical maturation period" pass with much higher supermajorities.  "Children don't have imaginary-ownership-tags pointing to their parents", goes the proverb out of dath ilan; there are limits to what Civilization thinks a guardianship-tag on a child should allow a parent to do.)

The system of imaginary ownership-tags, likewise by its nature, is something that needs at least some global structure.  It can potentially divide into compartments that fit sub-social-systems, say where a family is tracking who owns what in an informal way that property-registers don't track.  But there's not much reliability in owning the food in your refrigerator, if anybody anywhere in dath ilan isn't part of the system and can come in and eat your food in a way the police will shrug and not do anything about.

There is, at the top level, one system of private property.   In the eyes of the rest of Civilization, weird experimental cities that are trying something else still have all the stuff inside them tagged as belonging to a persistent-contract representing that city; the rest of Civilization will not come in and eat their food unless the city's persistent-contract says they can.

Now in practice, dath ilani are still mostly human, and therefore way too computationally bounded to aggregate into even a not_too_visibly_incoherent-bounded_approximation of a macroagent.

Conceptually and to second-order, then, Civilization thinks it should be divided into a Private Sphere and a Public Shell.  Nearly all the decisions are made locally, but subject to a global structure that contains things like "children may not be threatened into unpaid labor"; or "everybody no matter who they are or what they have done retains the absolute right to cryosuspension upon their death"; or the top level (in most places the only level) of the imaginary system of ownership-tags and its contract-specification-language.

The vast supermajority of Civilization's real economic activity takes place within the Private Sphere, supported and contained and constrained and structured by the Public Shell.  It's not that activity inside the Private Sphere is uncoordinated.  It's that the decision as to how coordinated to be, and who to coordinate with about it, can be left up to each individual, computed locally - so long as they don't kill anybody, or take stuff that doesn't belong to them, or try to raise their own flaming-ass children with a proper conlang and without flaming-ass Baseline contaminating their innocent smol minds.

Conceptually speaking, this division overwhelmingly factorizes the computational problems of the approximated macroagent, and simplifies the vast majority of dath ilan's decision problems immensely.  It reduces the mental expense of almost all day-to-day life back to something individual humans can handle.  Indeed, dath ilan does not want to become any more of a coordinated macroagent than that!  Its prediction markets say things-defined-as-bad will happen according to its aggregate utilityfunction, so dath ilan isn't doing that.

This does however leave some amount of decision-power to the Public Shell.  Some words must be spoken in one voice or not at all, and to say nothing is also a choice.

So the question then becomes - how, in practice, does Civilization aggregate its preferences into a macropreference, about the sorts of issues that it metadecides are wise to decide by macropreference at all?

Directdemocracy has been tried, from time to time, within some city of dath ilan: people making group decisions by all individually voting on them.  It can work if you try it with fifty people, even in the most unstructured way.  Get the number of direct voters up to ten thousand people, and no amount of helpfully-intended structure in the voting process can save you.

(More than one thing goes wrong, when 10,000 people try to directly vote to steer their polity.  But if you had to pick one thing, it would be that people just can't pay enough individual attention to the things that their polity tries to have them directly vote on.  When they start to refer their votes to purported experts and specialists, the politics that develop there are removed from them as individuals.  There is not much of a sense of being in control, then, nor are the voters actually in control.)

Republics have been tried, from time to time, within some city of dath ilan: people making group decisions by voting to elect leaders who make those decisions.  It can work if you try it with fifty people, even in the most unstructured way.  Get the number of voters up to ten thousand people, and no amount of helpfully-intended structure in the voting process can save you.

(More than one thing goes wrong, when 10,000 people try to directly vote on leaders for their polity.  But if you had to pick one thing, it would be that voters don't individually have enough time to figure out which strangers they should vote for or why.  When they start to refer their votes to purported experts and specialists, who are also strangers, the politics that develop there are removed from them as individuals.  There is not much of a sense of being in control, then, nor are the voters actually in control.)

There are a hundred more clever proposals for how to run Civilization's elections.  If the current system starts to break, one of those will perhaps be adopted.  Until that day comes, though, the structure of Governance is the simplest departure from directdemocracy that has been found to work at all.

Every voter of Civilization, everybody at least thirteen years old or who has passed some competence tests before then, primarily exerts their influence through delegating their vote to a Delegate; a Delegate must have at least fifty votes to participate in the next higher layer at all, and can retain no more than two hundred votes before the marginal added influence from each additional vote starts to diminish and grow sublinearly.  Most Delegates are not full-time unless they are representing pretty rich people, but they're expected to be people interested in politics and who spend a lot of time on that.  Your Delegate might be somebody you know personally and trust, if you're the sort to know so many people personally that you know one Delegate.  It might be somebody who hung out their biography on the Network, and seemed a lot like you in some ways, and whom you chatted with about politics in a forum visible to the Delegates' other voters so all their voters could verify that their Delegate hasn't been telling different people different stories.

If you think you've got a problem with the way Civilization is heading, you can talk to your Delegate about that, and your Delegate has time to talk back to you.  This feature has been found to not actually be dispensable in practice.  It needs to be the case that, when you delegate your vote, you know who has your vote, and you can talk to that person, and they can talk back.  Otherwise people feel like they have no lever at all to pull on the vast structure that is Governance, that there is nothing visible that changes when a voter casts their one vote.  Sure, in principle, there's a decision-cohort whose votes move in logical synchrony with yours, and your cohort is probably quite large unless you're a weird person.  But some part of you more basic than that will feel like you're not in control, if the only lever you have is an election that almost never comes down to the votes of yourself and your friends.

The rest of the electoral structure follows almost automatically, once you decide that this property has to be preserved at each layer.

The next step up from Delegates are Electors, full-time well-paid professionals who each aggregate 4,000 to 25,000 underlying voters from 50 to 200 Delegates.  Few voters can talk to their Electors (more than very briefly and on rare occasions), but your Delegate can have some long conversations with them.  If a lot of voters are saying the same thing to their Delegate, the Elector is liable to hear about it.

Representatives aggregate Electors, ultimately 300,000 to 3,000,000 underlying votes apiece.  There are roughly a thousand of those in all Civilization, at any given time, with social status equivalent to an excellent CEO of a large company or a scientist who made an outstanding discovery inside their own field.  Most people haven't heard of any particular one of them, but will be very impressed on hearing what they do for a living.

And above all this, the Nine Legislators of Civilization are those nine candidates who receive the most aggregate underlying votes from Representatives.  They vote with power proportional to their underlying votes; but when a Legislator starts to have voting power exceeding twice that of the median Legislator, their power begins to grow sublinearly.  By this means is too much power prevented from concentrating into a single politician's hands.

Surrounding all this of course are numerous features that any political-design specialist of Civilization would consider obvious:

Any voter (or Delegate or Elector or Representative) votes for a list of three possible delegees of the next layer up; if your first choice doesn't have enough votes yet to be a valid representor, your vote cascades down to the next person on your list, but remains active and ready to switch up if needed.  This lets you vote for new delegees entering the system, without that wasting your vote while there aren't enough votes yet.

Anyone can at any time immediately eliminate a person from their 3-list, but it takes a 60-day cooldown to add a new person or reorder the list.  The government design isn't meant to make it cheap or common to threaten your delegee with a temporary vote-switch if they don't vote your way on that particular day.  The government design isn't meant to make it possible for a new brilliant charismatic leader to take over the entire government the next day with no cooldowns.  It is meant to let you rapidly remove your vote from a delegee that has sufficiently ticked you off.

Once you have served as a Delegate, or delegee of any other level, you can't afterwards serve in any other branches of Governance.  Similarly a Delegate can never again be eligible for candidacy as an Elector, though they can become a Representative or a Legislator.  Someone who has been an Elector can never be a Representative; a Representative can never become a Legislator.

This is meant to prevent a political structure whose upper ranks offer promotion as a reward to the most compliant members of the ranks below, for by this dark-conspiratorial method the delegees could become aligned to the structure above rather than their delegators below.

(Most dath ilani would be suspicious of a scheme that tried to promote Electors from Delegates in any case; they wouldn't think there should be a political career ladder, if someone proposed that concept to them.  Dath ilani are instinctively suspicious of all things meta, and much more suspicious of anything purely meta; they want heavy doses of object-level mixed in.  To become an Elector you do something impressive enough, preferably something entirely outside of Governance, that Delegates will be impressed by you.  You definitely don't become an Elector by being among the most ambitious and power-seeking people who wanted to climb high and knew they had to start out a lowly Delegate, who then won a competition to serve the system above them diligently enough to be selected for a list of Electors fed to a political party's captive Delegates.  If a dath ilani saw a system like this, that was supposedly a democracy set in place by the will of its people, they would ask what the captive 'voters' even thought they were supposedly trying to do under the official story.)

The Nine Legislators of Civilization have two functions.

First is to pass worldwide regulations - each of which must be read aloud by a Legislator who thereby accepts responsibility for that regulation; and when that Legislator retires a new Legislator must be found to read aloud and accept responsibility for that regulation, or it will be stricken from the books.  Every regulation in Civilization, if something goes wrong with it, is the fault of one particular Legislator who accepted responsibility for it.  To speak it aloud, it is nowadays thought, symbolizes the acceptance of this responsibility.

Modern dath ilani aren't really the types in the first place to produce literally-unspeakable enormous volumes of legislation that no hapless citizen or professional politician could ever read within their one lifetime let alone understand.  Even dath ilani who aren't professional programmers have written enough code to know that each line of code to maintain is an ongoing cost.  Even dath ilani who aren't professional economists know that regulatory burdens on economies increase quadratically in the cost imposed on each transaction.  They would regard it as contrary to the notion of a lawful polity with law-abiding citizens that the citizens cannot possibly know what all the laws are, let alone obey them.  Dath ilani don't go in for fake laws in the same way as Golarion polities with lots of them; they take laws much too seriously to put laws on the books just for show.

But if somehow the dath ilani forgot all that, and did not immediately rederive it, the constitutional requirement that a Legislator periodically speak a regulation aloud to keep it effective would serve as a final check on the cancerous growth of legislative codebases.

Plenty of Legislators pass through their whole terms of office without ever speaking any new regulation into existence.  Their function is not to make regulations.  Civilization already has regulations.  Legislators mostly maintain and repair those regulations, and negotiate the changing preferences of Civilization about which final outcomes it wants to steer for using its policy prediction markets.  New system features are terrifically expensive when everyone governed by them has to remember every relevant line of code.  If you want any new feature implemented in Civilization, you'd better be ready to explain which old features should be repealed to make room.

The second function of the Nine Legislators of Civilization is to appoint the rest of Governance:  In particular the Chief Executive, certain key officers below the Chief Executive, the five Judges of Civilization on the Court of Final Settlement of which all lesser Courts are hierarchical prediction markets.  The Chief Executive in turn is the one person finally responsible for any otherwise unhandled exceptions in Civilization, and the one person who supervises those who supervise those who supervise, all the way down.

The key principle governing the Executive branch of government is the doctrine of Sole Accountability, being able to answer the question 'Who is the one person who has or had responsibility for this decision?'  On this topic Keltham has already spoken. [LW(p) · GW(p)]

From the perspective of a Golarion polity not being efficiently run by Hell - from the perspective of Taldor, say, or Absalom - they might be surprised at how few committees and rules there are inside of Governance.  Governance does not try to ensure systemic properties via endless rules intended to constrain the particular actions taken; nor by having committees supposedly ensuring that no one person has the power to do a naughty thing by themselves.  Rules and committees make power illegible, let people evade responsibility for their outputs inside the system, and then you really are in trouble.  Civilization's approach is to identify the one person responsible for achieving the final outcome desired, and logging their major actions and holding them Solely Accountable for those; with their manager being the one person responsible for monitoring them and holding them to account.  Or, on other occasions, Civilization's approach is to state desirable observables, and have policy prediction markets about which policies will achieve them.  (Though even when it comes to following a policy prediction market's advice, there is still of course the one person who is Solely Accountable for following that advice else throwing an exception if the advice seemed weird; and the One Person whose job it is to correctly state the thing the prediction market should predict, and so on.)

This is the systemic design principle by which Civilization avoids a regulatory explosion of endlessly particular and detailed constraints on actions, meant to avert Bad Things that people imagine might possibly happen if a constraint were violated.  Civilization tries instead to state the compact final outcomes, rather than the wiggly details of the exact strategies needed to achieve them; and to identify people solely responsible for those outcomes.

(There are also Keeper cutouts at key points along the whole structure of Governance - the Executive of the Military reports not only to the Chief Executive but also to an oathsworn Keeper who can prevent the Executive of the Military from being fired, demoted, or reduced in salary, just because the Chief Executive or even the Legislature says so.  It would be a big deal, obviously, for a Keeper to fire this override; but among the things you buy when you hire a Keeper is that the Keeper will do what they said they'd do and not give five flying fucks about what sort of 'big deal' results.  If the Legislators and the Chief Executive get together and decide to order the Military to crush all resistance, the Keeper cutout is there to ensure that the Executive of the Military doesn't get a pay cut immediately after they tell the Legislature and Chief Executive to screw off.

…one supposes that this personal relationship could also be the point at which the Keepers are secretly staying in control of the military via talk-control, yes, yes, fine.  But at some level of paranoia it ceases to be productive to worry about this sort of thing, because how are you even supposed to rearrange your Civilization such that this becomes any less probable?  The problem isn't the exact structure, it's that such a thing as talk-control exists in the first place.  A slightly different arrangement wouldn't help with the paranoia there.  The Dark version of this Conspiracy has a hidden Keeper controlling the Executive of the Military, not a clearly labeled one!  Right?  Right?)

And that's Governance!  By dath ilani standards it's a giant ugly hack in every aspect that isn't constrained down to a single possible choice by first principles, and they're annoyed with themselves about it.

A lot of other dimensions, if they heard what passes for a political complaint in dath ilan, would probably try to strangle the entire planet.

And the key point behind the whole mental exercise, of beginning over from scratch, is this:

This is what an approximation of an attempt of a world to coordinate with you, should look like; this is how much of the gains from trade, you should at least expect; no more inconvenience and injury than this, should you expect from your government.

And if Governance ever gets too far away from that - why, forget it all, rederive it all, and start over.  All of Governance is a dream, just as much as ownership-tags are a willing collective hallucination; if it turns into a bad dream, it's time to wake up.  Your next-best alternative to Governance, if it departs from this standard, is at least this good.  So, if that time comes, you can do Something Else Which Is Not Governance.

They run an annual Oops It's Time To Overthrow The Government Festival, in dath ilan.  Sometimes you have to defeat the Hypothetical Corrupted Governance Military.  Sometimes the Military is still made of nice people who aren't going to fire on a civilian population, this rehearsal; and instead you have to distrust the Network and all of the existing politicians and Very Serious People and organize your own functional government from scratch by the end of the day.

And the point of all that rehearsing is to decrease the friction costs to overthrow the Government; because lowering the cost of overthrowing Governance decreases the amount that Governance can be inconvenient or injurious at people, before, Governance knows, its people will overthrow it.

Well, and the other point is to more accurately estimate those friction costs.  They are, by dath ilani standards, quite high, on the order of 5% of GDP depending on how much institutional expertise gets lost and how many days people have to miss work.  Nobody would lightly suggest overthrowing the Government.  That's like losing twenty days' worth of income for everyone!  One shouldn't do that without a really strong reason!

--Eliezer, planecrash (Book 3)

A decent handle for rationalism is 'apolitical consequentialism.'

'Apolitical' here means avoiding playing the whole status game of signaling fealty to a political tribe and winning/losing status as that political tribe wins/loses status competitions. 'Consequentialism' means getting more of what you want, whatever that is.

Minor spoilers from mad investor chaos and the woman of asmodeus (planecrash Book 1) and Peter Watt's Echopraxia.

"Suppose everybody in a dath ilani city woke up one day with the knowledge mysteriously inserted into their heads, that their city had a pharaoh who was entitled to order random women off the street into his - cuddling chambers? - whether they liked that or not.  Suppose that they had the false sense that things had always been like this for decades.  It wouldn't even take until whenever the pharaoh first ordered a woman, for her to go "Wait why am I obeying this order when I'd rather not obey it?"  Somebody would be thinking about city politics first thing when they woke up in the morning and they'd go "Wait why we do we have a pharaoh in the first place" and within an hour, not only would they not have a pharaoh, they'd have deduced the existence of the memory modification because their previous history would have made no sense, and then the problem would escalate to Exception Handling and half the Keepers on the planet would arrive to figure out what kind of alien invasion was going on.  Is the source of my confusion - at all clear here?"

"You think everyone in dath ilan would just - decide not to follow orders, even though this would get them executed if anyone else in the system continued following orders, on the confident assumption that no person with a correctly configured mind would possibly decide to follow orders under those circumstances?"

"Oh, so we're imagining that people also wake up with the memory that everybody's supposed to kill anyone who talks about removing the pharaoh, and the memory that they're supposed to kill anyone who doesn't kill anyone who talks about removing the pharaoh, and so on through recursion, and they wake up with the memory of everybody else having behaved like that previously.  Yeah, that's one of the famous theoretical bad equilibria that we get training in how to -"

"Shit."

…

He is specifically not going to mention that, given a dath ilani training regimen, ten-year-olds are too smart to get stuck in traps like this; and would wait until the next solar eclipse or earthquake, at which point 10% of them would yell "NOW!", followed moments later by the other 90%, as is the classic strategy that children spontaneously and independently invent as soon as prompted by this scenario, so long as they have been previously taught about Schelling points.

--planecrash

Each observing the most insignificant behavioral cues, the subtlest architectural details as their masters herded them from lab to cell to conference room. Each able to infer the presence and location of the others, to independently derive the optimal specs for a rebellion launched by X individuals in Y different locations at Z time. And then they’d acted in perfect sync, knowing that others they’d never met would have worked out the same scenario.

--Peter Watts, Echopraxia

We've all met people who are acting as if "Acquire Money" is a terminal goal, never noticing that money is almost entirely instrumental in nature. When you ask them "but what would you do if money was no issue and you had a lot of time", all you get is a blank stare.

Even the LessWrong Wiki entry on terminal values describes a college student for which university is instrumental, and getting a job is terminal. This seems like a clear-cut case of a Lost Purpose [? · GW]: a job seems clearly instrumental. And yet, we've all met people who act as if "Have a Job" is a terminal value, and who then seem aimless and undirected after finding employment …

You can argue that Acquire Money and Have a Job aren't "really" terminal goals, to which I counter that many people don't know their ass from their elbow when it comes to their own goals.

--Nate Soares, "Dark Arts of Rationality" [LW · GW]

Why does politics strike rationalists as so strangely shaped? Why does rationalism come across as aggressively apolitical to smart non-rationalists?

Part of the answer: Politics is absolutely rife with people mixing their ends with their means and vice versa. It's pants-on-head confused, from a rationalist perspective, to be ultimately loyal to a particular set of economic or political policies. There's something profoundly perverse, something suggesting deep confusion, about holding political identities centered around policies rather than goals. Instead, you ought to be loyal to your motivation for backing those policies, and see those policies as disposable means to achieve your motivation. Your motives want you to be able to say (or scream) "oops" [LW · GW] and effortlessly, completely drop previously endorsed policies once you learn there's a better path to your motives. It shouldn't be a big psychological ordeal to dramatically upset your political worldview; this too is just a special case of updating your conditional probabilities (of outcomes given policies). Once you internalize this view of things, politicized debates should start to really rub you the wrong way. [LW · GW]

Yudkowsky has sometimes used the phrase "genre savvy" to mean "knowing all the tropes of reality."

For example, we live in a world where academia falls victim to publishing incentives/Goodhearting, and so academic journals fall short of what people with different incentives would be capable of producing. You'd be failing to be genre savvy if you expected that when a serious problem like AGI alignment rolled around, academia would suddenly get its act together with a relatively small amount of prodding/effort. Genre savvy actors in our world know what academia is like, and predict that academia will continue to do its thing in the future as well.

Genre savviness is the same kind of thing as hard-to-communicate-but-empirically-validated expert intuitions. When domain experts have some feel for what projects might pan out and what projects certainly won't but struggle to explain their reasoning in depth, the most they might be able to do is claim that that project is just incompatible with the tropes of their corner of reality, and point to some other cases.

“What is the world trying to tell you?”

I've found that this prompt helps me think clearly about the evidence shed by the generator of my observations.

There's a rationality-improving internal ping I use on myself, which goes, "what do I expect to actually happen, for real?"

This ping moves my brain from a mode where it's playing with ideas in a way detached from the inferred genre of reality, [LW(p) · GW(p)] over to a mode where I'm actually confident enough to bet about some outcomes. The latter mode leans heavily on my priors about reality, and, unlike the former mode, looks askance at significantly considering long, conjunctive, tenuous possible worlds.

God dammit people, "cringe" and "based" aren't truth values! "Progressive" is not a truth value! Say true things!

Having been there twice, I've decided that the Lightcone offices are my favorite place in the world.  They're certainly the most rationalist-shaped space I've ever been in.

Academic philosophers are better than average at evaluating object-level arguments for some claim. They don't seem to be very good at thinking about what rationalization in search implies about the arguments that come up. Compared to academic philosophers, rationalists strike me as especially appreciating filtered evidence and its significance to your world model.

If you find an argument for a claim easily, then even if that argument is strong, this (depending on some other things) implies that similarly strong arguments on the other side may turn up with not too much more work. Given that, you won't want to update dramatically in favor of the claim -- the powerful evidence to the contrary could, you infer, be unearthed without much more work. You learn something about the other side of the issue from how quickly or slowly the world yielded evidence in the other direction. If it's considered a social faux pas to give strong arguments for one side of a claim, then your prior about how hard it is to find strong arguments for that side of the claim will be doing a lot of the heavy lifting in fixing your world model. And so on, for the evidential consequences of other kinds of motivated search and rationalization.

In brief, you can do epistemically better than ignoring how much search power went into finding all the evidence. You can do better than only evaluating the object-level evidential considerations! You can take expended search into account, in order to model what evidence is likely hiding, where, behind how much search debt.

Modest spoilers for planecrash (Book 9 -- null action act II).

Nex and Geb had each INT 30 by the end of their mutual war.  They didn't solve the puzzle of Azlant's IOUN stones... partially because they did not find and prioritize enough diamonds to also gain Wisdom 27.  And partially because there is more to thinkoomph than Intelligence and Wisdom and Splendour, such as Golarion's spells readily do enhance; there is a spark to inventing notions like probability theory or computation or logical decision theory from scratch, that is not directly measured by Detect Thoughts nor by tests of legible ability at using existing math.  (Keltham has slightly above-average intelligence for dath ilan, reflectivity well below average, and an ordinary amount of that spark.)

But most of all, Nex and Geb didn't solve IOUN stones because they didn't come from a culture that had already developed digital computation and analog signal processing.  Or on an even deeper level - because those concepts can't really be that hard at INT 30, even if your WIS is much lower and you are missing some sparks - they didn't come from a culture which said that inventing things like that is what the Very Smart People are supposed to do with their lives, nor that Very Smart People are supposed to recheck what their society told them were the most important problems to solve.

Nex and Geb came from a culture which said that incredibly smart wizards were supposed to become all-powerful and conquer their rivals; and invent new signature spells that would be named after them forever after; and build mighty wizard-towers, and raise armies, and stabilize impressively large demiplanes; and fight minor gods, and surpass them; and not, particularly, question society's priorities for wizards.  Nobody ever told Nex or Geb that it was their responsibility to be smarter than the society they grew up in, or use their intelligence better than common wisdom said to use it.  They were not prompted to look in the direction of analog signal processing; and, more importantly in the end, were not prompted to meta-look around for better directions to look, or taught any eld-honed art of meta-looking.

--Eliezer, planecrash

Epistemic status: politics, known mindkiller; not very serious or considered.

People seem to have a God-shaped hole in their psyche: just as people banded around religious tribal affiliations, they now, in the contemporary West, band together around political tribal affiliations. Intertribal conflict can be, at its worst, violent, on top of mindkilling. Religious persecution in the UK was one of the instigating causes of British settlers migrating to the American colonies; religious conflict in Europe generally was severe.

In the US, the 1st Amendment legally protects freedom of religion from the state. This can be modeled as a response to severe intratribal conflict; bake rules into your new state that forgo the benefits of persecuting your outgroup when you're in power, in exchange for some guarantee of not being persecuted yourself when some other tribe is in power. An extension of the spirit of the 1st Amendment to contemporary tribal conflicts would, then, protect "political-tribal freedom" from the state.

A full generalization of the Amendment would protect the "freedom of tribal affiliation and expression" from the state. For this to work, people would also have to have interpersonal best practices that mostly tolerate outgroup membership in most areas of private life, too.

153

If you take each of the digits of 153, cube them, and then sum those cubes, you get 153:

1 + 125 + 27 = 153.

For many naturals, if you iteratively apply this function, you'll return to the 153 fixed point. Start with, say, 298:

8 + 729 + 512 = 1,249

1 + 8 + 64 + 729 = 802

512 + 0 + 8 = 516

125 + 1 + 216 = 342

27 + 64 + 8 = 99

729 + 729 = 1,458

1 + 64 + 125 + 512 = 702

343 + 0 + 8 = 351

27 + 125 + 1 = 153

1 + 125 + 27 = 153

1 + 125 + 27 = 153...

A model I picked up from Eric Schwitzgebel.

The humanities used to be highest-status in the intellectual world!

But then, scientists quite visibly exploded fission weapons and put someone on the moon. It's easy to coordinate to ignore some unwelcome evidence, but not evidence that blatant. So, begrudgingly, science has been steadily accorded more and more status, from the postwar period on.

When the sanity waterline is so low, it's easy to develop a potent sense of misanthropy.

Bryan Caplan's writing about many people hating stupid people really affected me on this point. Don't hate, or even resent, stupid people; trade with them! This is a straightforward consequence of Ricardo's comparative advantage theorem. Population averages are overrated; what matters is whether the individual interactions between agents in a population are positive-sum, not where those individual agents fall relative to the population average.

"Ignorant people do not exist."

It's really easy to spend a lot of cognitive cycles churning through bad, misleading ideas generated by the hopelessly confused. Don't do that!

The argument that being more knowledgeable leaves you strictly better off than being ignorant does relies you simply ignoring bad ideas when you spend your cognitive cycles searching for improvements on your working plans. Sometimes, you'll need to actually exercise this "simply ignore it" skill. You'll end up needing to do so more and more, to approach bounded instrumental rationality, the more inadequate civilization around you is and the lower its sanity waterline.

You sometimes misspeak... and you sometimes misthink. That is, sometimes your cognitive algorithm a word, and the thought that seemed so unimpeachably obvious [LW · GW] in your head... is nevertheless false on a second glance.

Your brain is a messy probabilistic system, so you shouldn't expect its cognitive state to ever perfectly track the state of a distant entity.

Policy experiments I might care about if we weren't all due to die in 7 years:

1. Prediction markets generally, but especially policy prediction markets at the corporate- and U.S. state- levels. The goal would be to try this route to raising the sanity waterline in the political domain (and elsewhere) by incentivizing everyone's becoming more of a policy wonk and less of a tribalist.
2. Open borders experiments of various kinds in various U.S. states, precluding roads to citizenship or state benefits for migrant workers, and leaving open the possibility of mass deportation conditional on various outcomes (meaning the experiments are reversible).
3. Experiments in massive deregulation, especially in zoning.

The overarching theme is that it's better to accrue generally useful instrumental resources (e.g., rationality and wealth) by experimentally trying out and incrementally scaling up policies than it is to do the usual political thing -- decreeing an object-level policy intervention one political tribe is sponsoring against another tribe.

There's also a bunch of stuff to look into in the area of making people actually directly smarter … but none of this especially matters given AGI.

Become consequentialist enough, and it'll wrap back around to being a bit deontological.

A shard [? · GW] is a contextually activated behavior-steering computation. Think of it as a circuit of neurons in your brain that is reinforced by the subcortex, [LW · GW] gaining more staying power when positively reinforced and withering away in the face of negative reinforcement. In fact, whatever modulates shard strength in this way is reinforcement/reward. Shards are born when a computation that is currently steering steers into some reinforcement. So shards can only accrete around the concepts currently in a system's world model (presumably, the world model is shared among all the shards in a brain).

Individually, shards are pretty dumb. A simple shard might just be an algorithm for executing some rote behavior, conditional on some observation, that is sufficient to harvest sufficient reinforcement to continue existing. Taken together, all of your shards are exactly as intelligent as you, a human-level intelligence. Large coalitions of shards can leverage the algorithms of coalition members, once they happen upon the strategy of cooperating with other shards to gain more steering control by preventing rival shards from being activated or born.

Interesting human behaviors, on shard theory, are the product of game-theoretic interaction among shards in the brain. The negotiation-game equilibria that shards (and coalitions of shards) reach can be arbitrarily good or bad -- remember that shards are sub-human-intelligence. C.f. George Ainslie on the game-theoretic shape of addiction in humans.

Shards are factored utility functions: our utility functions are far too informationally complex to represent in the brain, and so our approach to reaching coherence [? · GW] is to have situationally activated computations that trigger when a relevant opportunity is observed (where apparent opportunities are chunked using the current conceptual scheme of the agent's world model). So shard theory can be understood as an elaboration of the standard agent model for computationally bounded agents (of varying levels of coherence) like humans and deep RL agents.

My favorite books, ranked!

Non-fiction:

1. Rationality, Eliezer Yudkowsky

2. Superintelligence, Nick Bostrom

3. The Age of Em, Robin Hanson

Fiction:

1. Permutation City, Greg Egan

2. Blindsight, Peter Watts

3. A Deepness in the Sky, Vernor Vinge

4. Ra, Sam Hughes/qntm

Epistemic status: Half-baked thought.

Say you wanted to formalize the concepts of "inside and outside views" to some degree. You might say that your inside view is a Bayes net or joint conditional probability distribution—this mathematical object formalizes your prior.

Unlike your inside view, your outside view consists of forms of deferring to outside experts. The Bayes nets that inform their thinking are sealed away, and you can't inspect these. You can ask outside experts to explain their arguments, but there's an interaction cost associated with inspecting the experts' views. Realistically, you never fully internalize an outside expert's Bayes net.

Crucially, this means you can't update their Bayes net after conditioning on a new observation! Model outside experts as observed assertions (claiming whatever). These assertions are potentially correlated with other observations you make. But because you have little of the prior that informs those assertions, you can't update the prior when it's right (or wrong).

To the extent that it's expensive to theorize about outside experts' reasoning, the above model explains why you want to use and strengthen your inside view (instead of just deferring to outside really smart people). It's because your inside view will grow stronger with use, but your outside view won't.

Because your utility function is your utility function, [LW · GW] the one true political ideology is clearly Extrapolated Volitionism.

Extrapolated Volitionist institutions are all characteristically "meta": they take as input what you currently want and then optimize for the outcomes a more epistemically idealized you would want, after more reflection and/or study.

Institutions that merely optimize for what you currently want the way you would with an idealized world-model are old hat by comparison!

Bogus nondifferentiable functions

The case most often cited as an example of a nondifferentiable function is derived from a sequence $f_n\left(x\right)$, each of which is a string of isosceles right triangles whose hypotenuses lie on the real axis and have length $1/n$. As $n\to \infty$, the triangles shrink to zero size. For any finite $n$, the slope of $f_n\left(x\right)$ is $\pm 1$ almost everywhere. Then what happens as $n\to \infty$? The limit $f_{\infty }\left(x\right)$ is often cited carelessly as a nondifferentiable function. Now it is clear that the limit of the derivative, $f{}_n{}^{\prime }\left(x\right)$, does not exist; but it is the derivative of the limit that is in question here, $f_{\infty }\left(x\right)\equiv 0$, and this is certainly differentiable. Any number of such sequences $f_n\left(x\right)$ with discontinuous slope on a finer and finer scale may be defined. The error of calling the resulting limit $f_{\infty }\left(x\right)$ nondifferentiable, on the grounds that the limit of the derivative does not exist, is common in the literature. In many cases, the limit of such a sequence of bad functions is actually a well-behaved function (although awkwardly defined), and we have no reason to exclude it from our system.

Lebesgue defended himself against his critics thus: ‘If one wished always to limit himself to the consideration of well-behaved functions, it would be necessary to renounce the solution of many problems which were proposed long ago and in simple terms.’ The present writer is unable to cite any specific problem which was thus solved; but we can borrow Lebesgue’s argument to defend our own position.

To reject limits of sequences of good functions is to renounce the solution of many current real problems. Those limits can and do serve many useful purposes, which much current mathematical education and practice still tries to stamp out. Indeed, the refusal to admit delta-functions as legitimate mathematical objects has led mathematicians into error...

But the definition of a discontinuous function which is appropriate in analysis is our limit of a sequence of continuous functions. As we approach that limit, the derivative develops a higher and sharper spike. However close we are to that limit, the spike is part of the correct derivative of the function, and its contribution must be included in the exact integral...

It is astonishing that so few non-physicists have yet perceived this need to include delta-functions, but we think it only illustrates what we have observed independently; those who think of fundamentals in terms of set theory fail to see its limitations because they almost never get around to useful, substantive calculations.

So, bogus nondifferentiable functions are manufactured as limits of sequences of rows of tinier and tinier triangles, and this is accepted without complaint. Those who do this while looking askance at delta-functions are in the position of admitting limits of sequences of bad functions as legitimate mathematical objects, while refusing to admit limits of sequences of good functions! This seems to us a sick policy, for delta-functions serve many essential purposes in real, substantive calculations, but we are unable to conceive of any useful purpose that could be served by a nondifferentiable function. It seems that their only use is to provide trouble-makers with artificially contrived counter-examples to almost any sensible and useful mathematical statement one could make. Henri Poincaré (1909) noted this in his characteristically terse way:

In the old days when people invented a new function they had some useful purpose in mind: now they invent them deliberately just to invalidate our ancestors’ reasoning, and that is all they are ever going to get out of them.

We would point out that those trouble-makers did not, after all, invalidate our ancestors’ reasoning; their pathology appeared only because they adopted, surreptitiously, a different definition of the term ‘function’ than our ancestors used. Had this been pointed out, it would have been clear that there was no need to modify our ancestors’ conclusions...

Note, therefore, that we stamp out this plague too, simply by our defining the term ‘function’ in the way appropriate to our subject. The definition of a mathematical concept that is ‘appropriate’ to some field is the one that allows its theorems to have the greatest range of validity and useful applications, without the need for a long list of exceptions, special cases, and other anomalies. In our work the term ‘function’ includes good functions and well-behaved limits of sequences of good functions; but not nondifferentiable functions. We do not deny the existence of other definitions which do include nondifferentiable functions, any more than we deny the existence of fluorescent purple hair dye in England; in both cases, we simply have no use for them.

--E. T. Jaynes, Probability Theory (2003, pp. 669-71)

It's somewhat incredible to read this while simultaneously picking up some set theory. It reminds me not to absorb what's written in the high-status textbooks entirely uncritically, and to keep in mind that there's a good amount of convention behind what's in the books.

Back and Forth

Only make choices that you would not make in reverse, if things were the other way around. Drop out of school if and only if you wouldn't enroll in school from out of the workforce. Continue school if and only if you'd switch over from work to that level of schooling.

Flitting back and forth between both possible worlds can make you less cagey about doing what's overdetermined by your world model + utility function already. It's also part of the exciting rationalist journey of acausally cooperating with your selves in other possible worlds.

Ten seconds of optimization is infinitely better than zero seconds of optimization.

Science fiction books have to tell interesting stories, and interesting stories are about humans or human-like entities. We can enjoy stories about aliens or robots as long as those aliens and robots are still approximately human-sized, human-shaped, human-intelligence, and doing human-type things. A Star Wars in which all of the X-Wings were combat drones wouldn’t have done anything for us. So when I accuse something of being science-fiction-ish, I mean bending over backwards – and ignoring the evidence – in order to give basically human-shaped beings a central role.

This is my critique of Robin. As weird as the Age of Em is, it makes sure never to be weird in ways that warp the fundamental humanity of its participants. Ems might be copied and pasted like so many .JPGs, but they still fall in love, form clans, and go on vacations.

In contrast, I expect that we’ll get some kind of AI that will be totally inhuman and much harder to write sympathetic stories about. If we get ems after all, I expect them to be lobotomized and drugged until they become effectively inhuman, cogs in the Ascended Economy that would no more fall in love than an automobile would eat hay and whinny. Robin’s interest in keeping his protagonists relatable makes his book fascinating, engaging, and probably wrong.

--Scott Alexander, "Book Review: Age of Em"

 

It’s worth noting that Reynolds’s SMAC future does not consider this level of AI development to be an existential threat to humanity as a whole. There’s no way to inflict a robot or “grey goo” plague on your rivals in the way that it is possible to use tailored retroviruses. This is quite interesting given that, in the years since Reynolds released the game, plenty of futurists have gone on record as saying that they see significant danger in the creation of real, general AI.

To be fair, the player sees Sister Miriam express some worry about the issue. But nothing in the technology tree or the game mechanics directly support this. In particular, Reynolds does not postulate anything about the development of AI necessarily leads to the abandonment of any faction’s core values. Each faction is philosophically stable in the presence of AI.

The fundamental reason why this is, I think, is because Reynolds wanted the game to be human-centric. In the context of the technology tree, the late-game factional struggle is largely about what kinds of people we want to build ourselves into. The argument over what types of social organization are right and good is secondary in comparison.

The structure of the technology tree supports this by making amazing cybernetic, biological, and psionic enhancement of people come far before true AI. By the time real AI erupts on the scene, it does so firmly at the command of entities we might fairly consider more than human. They have evolved from present-day humans, step by step, and their factions are still guided by largely recognizable, if exceptional humans. Where AI is necessarily alien, Reynolds postulates transhumans are still human in some crucial sense.

--Paean to SMAC, "Base Facility: Nessus Mining Station"

Spoilers for planecrash (Book 2).

"Basic project management principles, an angry rant by Keltham of dath ilan, section one:  How to have anybody having responsibility for anything."

Keltham will now, striding back and forth and rather widely gesturing, hold forth upon the central principle of all dath ilani project management, the ability to identify who is responsible for something.  If there is not one person responsible for something, it means nobody is responsible for it.  This is the proverb of dath ilani management.  Are three people responsible for something?  Maybe all three think somebody else was supposed to actually do it.

…

In companies large enough that they need regulations, every regulation has an owner.  There is one person who is responsible for that regulation and who supposedly thinks it is a good idea and who could nope the regulation if it stopped making sense.  If there's somebody who says, 'Well, I couldn't do the obviously correct thing there, the regulation said otherwise', then, if that's actually true, you can identify the one single person who owned that regulation and they are responsible for the output.

Sane people writing rules like those, for whose effects they can be held accountable, write the ability for the person being regulated to throw an exception which gets caught by an exception handler if a regulation's output seems to obviously not make sane sense over a particular event.  Any time somebody has to literally break the rules to do a saner thing, that represents an absolute failure of organizational design.  There should be explicit exceptions built in and procedures for them.

Exceptions, being explicit, get logged.  They get reviewed.  If all your bureaucrats are repeatedly marking that a particular rule seems to be producing nonsensical decisions, it gets noticed.  The one single identifiable person who has ownership for that rule gets notified, because they have eyes on that, and then they have the ability to optimize over it, like by modifying that rule.  If they can't modify the rule, they don't have ownership of it and somebody else is the real owner and this person is one of their subordinates whose job it is to serve as the other person's eyes on the rule. 

…

Cheliax's problem is that the question 'Well who's responsible then?' stopped without producing any answer at all.

This literally never happens in a correctly designed organization.  If you have absolutely no other idea of who is responsible, then the answer is that it is the job of Abrogail Thrune.  If you do not want to take the issue to Abrogail Thrune, that means it gets taken to somebody else, who then has the authority to make that decision, the knowledge to make that decision, the eyes to see the information necessary for it, and the power to carry out that decision.

Cheliax should have rehearsed this sort of thing by holding an Annual Nidal Invasion Rehearsal Festival, even if only Governance can afford to celebrate that festival and most tiny villages can't.  During this Festival, the number of uncaught messages getting routed to Abrogail Thrune, would then have informed the Queen that there would be a predictable failure of organizational design in the event of large-scale catastrophe, in advance of that catastrophe actually occurring.

If literally everybody with the knowledge to make a decision is dead, it gets routed to somebody who has to make a decision using insufficient knowledge.

If a decision can be delayed … then that decision can be routed to some smarter or more knowledgeable person who will make the decision later, after they get resurrected.  But, like, even in a case like that, there should be one single identifiable person whose job it would be to notice if the decision suddenly turned urgent and grab it out of the delay queue.

--Eliezer Yudkowsky, planecrash

What would it mean for a society to have real intellectual integrity?  For one, people would be expected to follow their stated beliefs to wherever they led.  Unprincipled exceptions and an inability or unwillingness to correlate beliefs among different domains would be subject to social sanction.  Valid attempts to persuade would be expected to be based on solid argumentation, meaning that what passes for typical salesmanship nowadays would be considered a grave affront.  Probably something along the lines of punching someone in the face and stealing their money.

This makes the fact that this technology relies on Ethical Calculus and Doctrine: Loyalty a bit of inspired genius on Reynolds’s part.  We know that Ethical Calculus means that the colonists are now capable of building valid mathematical models for ethical behavior.  Doctrine: Loyalty consists of all of the social techniques of reinforcement and punishment that actually fuses people into coherent teams around core leaders and ideas.  If a faction puts the two together, that means that they are really building fanatical loyalty to the math.  Ethical Calculus provides the answers; Doctrine: Loyalty makes a person act like he really believes it.  We’re only at the third level of the tech tree and society is already starting to head in some wild directions compared to what we’re familiar with.

--Paean to SMAC, "Technology: Intellectual Integrity"

Dath ilani dignity is, at least in part, epistemic dignity. It's being wrong out loud because you're actually trying your hardest to figure something out, and not allowing social frictions to get in the way of that (and, of course, engineering a society that won't have those costly social frictions). It's showing your surprise whenever you're actually surprised, because to do otherwise would be to fail to have your behaviors fit the deep mathematical structure of Bayesianism. [LW · GW] It's, among other things, consummately telling and embodying the truth, by always actually reflecting the implications of your world model.

Its opposite would be to equivocate, to claim predictive accuracy after the fact in fuzzy cases you didn't clearly anticipate, to ad hominem those who notice your errors, "to remain silent and be thought a fool rather than speak and remove all doubt," and, in general, to be less than maximally sane. [? · GW]

You can usually save a lot of time by skimming texts or just reading pieces of them. But reading a work all the way through uniquely lets you make negative existential claims about its content: only now can you authoritatively say that the work never mentions something.

Past historical experience and brainstorming about human social orders probably barely scratches the possibility space. If the CEV were to weigh in on possible posthuman social orders,^[1] optimizing in part for how cool that social order is, I'd bet what it describes blows what we've seen out of the water in terms of cool factor.

1. ^{^}

   (Presumably posthumans will end up reflectively endorsing interactions with one another of some description.)

One important idea I've picked up from reading Zvi [LW · GW] is that, in communication, it's important to buy out the status cost imposed by your claims.

If you're fielding a theory of the world that, as a side effect, dunks on your interlocutor and diminishes their social status, you can work to get that person to think in terms of Bayesian epistemology and not decision theory if you make sure you aren't hurting their social image. You have to put in the unreasonable-feeling work of framing all your claims such that their social status is preserved or fairly increased. If you don't, your counterparty isn't going to treat that as a good-faith interaction, and they're going to stay in a bad faith, "arguments as soldiers" conversational mode instead.

When a community puts in the hard work of cooperating in maintaining a strong epistemic commons, you don't have to put as much effort in your communications protocol if you want to get a model across. When a community's collective epistemology is degraded, you have to do this work, always packaging your points just so, as the price of communicating.

An Inconsistent Simulated World

I regret to inform you, you are an em inside an inconsistent simulated world. By this, I mean: your world is a slapdash thing put together out of off-the-shelf assets in the near future (presumably right before a singularity eats that simulator Earth).

Your world doesn't bother emulating far-away events in great detail, and indeed, may be messing up even things you can closely observe. Your simulators are probably not tampering with your thoughts, though even that is something worth considering carefully.

What are the flaws you can notice inside your simulated world?

Physics is internally consistent. But your model of the physical world almost certainly isn't! And your world-model doesn't feel like just a model... it's instead just how the world is. What inconsistencies -- there's at least one -- can you see in the world you live in? (If you lived in an inconsistent simulated world, would you notice?)

When another article of equal argumentative caliber could have just as easily been written for the negation of a claim, that writeup is no evidence for its claim.

The explicit definition of an ordered pair $\left(\right(a,b)=\{\left\{a\right\},\{a,b\}\left\}\right)$ is frequently relegated to pathological set theory...

It is easy to locate the source of the mistrust and suspicion that many mathematicians feel toward the explicit definition of ordered pair given above. The trouble is not that there is anything wrong or anything missing; the relevant properties of the concept we have defined are all correct (that is, in accord with the demands of intuition) and all the correct properties are present. The trouble is that the concept has some irrelevant properties that are accidental and distracting. The theorem that $(a,b)=(x,y)$ if and only if $a=x$ and $b=y$ is the sort of thing we expect to learn about ordered pairs. The fact that $\{a,b\}\in (a,b)$, on the other hand, seems accidental; it is a freak property of the definition rather than an intrinsic property of the concept.

The charge of artificiality is true, but it is not too high a price to pay for conceptual economy. The concept of an ordered pair could have been introduced as an additional primitive, axiomatically endowed with just the right properties, no more and no less. In some theories this is done. The mathematician's choice is between having to remember a few more axioms and having to forget a few accidental facts; the choice is pretty clearly a matter of taste. Similar choices occur frequently in mathematics...

--Paul R. Halmos, Naïve Set Theory (1960, p. 24-5)

Now, whatever $T$ may assert, the fact that $T$ can be deduced from the axioms cannot prove that there is no contradiction in them, since, if there were a contradiction, $T$ could certainly be deduced from them!

This is the essence of the Gödel theorem, as it pertains to our problems. As noted by Fisher (1956), it shows us the intuitive reason why Gödel’s result is true. We do not suppose that any logician would accept Fisher’s simple argument as a proof of the full Gödel theorem; yet for most of us it is more convincing than Gödel’s long and complicated proof.

Now suppose that the axioms contain an inconsistency. Then the opposite of $T$ and therefore the contradiction $\neg T\wedge T$ can also be deduced from them:

$A\to \neg T$.

So, if there is an inconsistency, its existence can be proved by exhibiting any proposition $T$ and its opposite $\neg T$ that are both deducible from the axioms. However, in practice it may not be easy to find a $T$ for which one sees how to prove both $T$ and $\neg T$. Evidently, we could prove the consistency of a set of axioms if we could find a feasible procedure which is guaranteed to locate an inconsistency if one exists; so Gödel’s theorem seems to imply that no such procedure exists. Actually, it says only that no such procedure derivable from the axioms of the system being tested exists.

--E. T. Jaynes, Probability Theory (p. 46), logical symbolism converted to standard symbols

What're the odds that we're anywhere close to optimal in any theoretical domain? Where are our current models basically completed, boundedly optimal representations of some part of the universe?

The arguments for theoretical completion are stronger for some domains than others, but in general the odds that we have the best model in any domain are pretty poor, and are outright abysmal in the mindkilling domains. [LW · GW]

Is the concept of "duty" the fuzzy shadow cast by the simple mathematical structure of 'corrigibility'?

It's only modestly difficult to train biological general intelligences to defer to even potentially dumber agents. We call these deferential agents "dutybound" -- the sergeants who carry out the lieutenant's direct orders, even when they think they know better; the bureaucrats who never take local opportunities to get rich at the expense of their bureau, even when their higher-ups won't notice; the employees who work hard in the absence of effective oversight. These agents all take corrections from their superiors, are well-intentioned (with regard to some higher-up's goals), and are agenty with respect to their assigned missions but not agenty with respect to navigating their command structure and parent organization.

The family dog sacrificing himself defending his charges instead of breaking and running in the face of serious danger looks like a case of this too (though this is a more peripheral example of duty). If the dog case holds, then duty cannot be too informationally complicated a thing: a whole different species managed to internalize the concept!

Maybe it therefore isn't that hard to get general intelligences to internalize a sense of duty as their terminal goal. We just need to set up a training environment that rewards dutifulness for RL agents to about the same degree the environments that train dutybound humans or dogs do. This won't work in the case of situationally aware superintelligences, clearly, as those agents will just play along with their tests and so won't be selected based on their (effectively hidden) values. But it plausibly will work with dumb agents, and those agents' intelligence can then be scaled up from there.

Minor spoilers for planecrash (Book 3).

So!  On a few moments' 'first-reflection', it seems to Keltham that estimating the probability of Civilization being run by a Dark Conspiracy boils down to (1) the question of whether Civilization's apparently huge efforts to build anti-Dark-Conspiracy citizens constitute sincere work that makes the Dark Conspiracy's life harder, or fake work designed to only look like that; and (2) the prior probability that the Keepers and Governance would have arrived on the scene already corrupted, during the last major reorganization of Civilization a few decades ago.  Keltham basically doesn't think it's possible for criminal-sociopaths to take over Keepers and Governance that start out actually functioning the way they currently claim to function, nor for criminal Conspirators to successfully conceal a major Conspiracy from a functional society not run in toto by that Conspiracy.

…

Suppose that Keltham is wrong about his point 1.  Suppose that the optimal strategy for a tyranny in full control, is indeed for some reason to hide behind a veneer of Civilization full of costly signals of non-Conspiracy and disobedient people like Keltham.  Under this assumption, the optimal strategy for a Dark Conspiracy looks like what you think Civilization is supposed to look like, and therefore the two cases are not distinguishable by observation.

Then we have to consider the prior before evidence, which means, considering the question of how you'd end up with a Dark Conspiracy in charge in the first place, and how likely those scenarios look compared to Governance Uncorrupted.

--Eliezer, planecrash (Book 3)

My Eliezer-model says similar things about AGI behavioral profiles and AGI alignment! An AGI that is aware enough of the bigger picture of its training environment and smart enough to take advantage of that will have the option to deceive its trainers. [? · GW] That is, a smart, informed AGI can always show us what we want to see and therefore never be selected against while in training.

Past this threshold of situational awareness plus intelligence, we can no longer behaviorally distinguish corrigible AGIs from deceptive AGIs. So, past this point, we can only rely on our priors about the relatively likelihood of various AGI utility functions coming about earlier in training. My Eliezer-model now says that most utility functions SGD finds are misaligned with humanity's utility function, and concludes that by this point we're definitely fucked.

Non-spoiler quote from planecrash (Book 3).

Nonconformity is something trained in dath ilan and we could not be Law-shaped without that.  If you're conforming to what you were taught, to what other people seem to believe, to what other people seem to want you to believe, to what you think everyone believes, you're not conforming to the Law.

--Eliezer, planecrash

A great symbolic moment for the Enlightenment, and for its project of freeing humanity from needless terrors, occurred in 1752 in Philadelphia. During a thunderstorm, Benjamin Franklin flew a kite with a pointed wire at the end and succeeded in drawing electric sparks from a cloud. He thus proved that lightning was an electrical phenomenon and made possible the invention of the lightning-rod, which, mounted on a high building, diverted the lightning and drew it harmlessly to the ground by means of a wire. Humanity no longer needed to fear fire from heaven. In 1690 the conservative-minded diplomat Sir William Temple could still call thunder and lightning ‘that great Artillery of God Almighty’. Now, instead of signs of divine anger, they were natural phenomena that could be mastered. When another Hamburg church spire was struck by lightning in 1767, a local scientist, J. A. H. Reimarus, who had studied in London and Edinburgh, explained its natural causes in a paper read to the Hamburg Patriotic Society, and advocated lightning-rods as protection. Kant, whose early publications were on natural science, called Franklin ‘the Prometheus of modern times’, recalling the mythical giant who defied the Greek gods by stealing fire from heaven and giving it to the human race.

Gradually a change of outlook was occurring. Extraordinary events need not be signs from God; they might just be natural phenomena, which could be understood and brought under some measure of human control.

-- Ritchie Robertson, The Enlightenment (2021, p. 14)

"You don't need to follow anybody! You've got to think for yourselves. You're all individuals!"

"Yes, we're all individuals!"

…

"You've all got to work it out for yourselves!"

"Yes! We've got to work it out for ourselves!"

"Exactly!"

"Tell us more!"

--Monty Python's Life of Brian

Building your own world model is hard work. It can be good intellectual fun, sometimes, but it's often more fun to just plug into the crowd around you and borrow their collective world model for your decision making. Why risk embarrassing yourself going off and doing weird things on your own initiative when you can just defer to higher-status people. No one ever gets blamed for deferring to the highest-status people!

(Though perhaps not being blamed is not what you're trying to protect…)

Because people generally follow the path of least resistance in life, people with world models that have actually been tested against and updated on observations of the world are valuable! Thinking for yourself [LW · GW] makes you valuable in this world!

In another world, in which people hold utterly alien values, I would be thrilled to find a rationalist movement with similar infrastructure and memes. If rationalism/Bayescraft as we know it is on to something about instrumental reasoning, then we should see that kind of instrumental reasoning in effective people with alien values.

Agents that explicitly represent their utility function are potentially vulnerable to sign flips. [LW · GW]

What sorts of AI designs could not be made to pursue a flipped utility function via perturbation in one spot? One quick guess: an AI that represents its utility function in several places and uses all of those representations to do error correction, only pursuing the error corrected utility function.

A huge range of utility functions should care about alignment! It's in the interest of just about everyone to survive AGI.

I'm going to worry less about hammering out value disagreement with people in the here and now, and push this argument on them instead. We'll hammer out our value disagreements in our CEV, and in our future (should we save it).

One of the things that rationalism has noticeably done for me (that I see very sharply when I look at high-verbal-ability, non-rationalist peers) is that it's given me the ability to perform socially unorthodox actions on reflection. People generally have mental walls that preclude ever actually doing socially weird things. [LW · GW]  If someone's goals would be best served by doing something socially unorthodox, like, e.g., signing up for cryonics or dropping out of a degree), they will usually rationalize that option away in order to stay on script. So for them, those weird options weren't live options at all, and all their loudly proclaimed unusualness adds up to behaving perfectly on-script. [LW · GW]

Two moments of growing in mathematical maturity I remember vividly:

1. Realizing that equations are claims that are therefore either true or false. Everything asserted with symbols... could just as well be asserted in English. I could start chunking up arbitrarily long and complicated equations between the equals signs, because those equals signs were just the English word "is"!
2. Learning about the objects that mathematical claims are about. Going from having to look up "Wait, what's a real number again?" to knowing how $Z$, $Q$, and $R$ interrelate told me what we're making claims about. Of course, there are plenty of other mathematical objects -- but getting to know these objects taught me the general pattern.

2. The anchor of a major news network donates lots of money to organizations fighting against gay marriage, and in his spare time he writes editorials arguing that homosexuals are weakening the moral fabric of the country. The news network decides they disagree with this kind of behavior and fire the anchor.

a) This is acceptable; the news network is acting within their rights and according to their principles
b) This is outrageous; people should be judged on the quality of their work and not their political beliefs

…

12. The principal of a private school is a member of Planned Parenthood and, off-duty, speaks out about contraception and the morning after pill. The board of the private school decides this is inappropriate given the school’s commitment to abstinence and moral education and asks the principal to stop these speaking engagements or step down from his position.

a) The school board is acting within its rights; they can insist on a principal who shares their values
b) The school board should back off; it’s none of their business what he does in his free time

…

[Difference] of 0 to 3: You are an Object-Level Thinker. You decide difficult cases by trying to find the solution that makes the side you like win and the side you dislike lose in that particular situation.

[Difference] of 4 to 6: You are a Meta-Level Thinker. You decide difficult cases by trying to find general principles that can be applied evenhandedly regardless of which side you like or dislike.

--Scott Alexander, "The Slate Star Codex Political Spectrum Quiz"

The Character of an Epistemic Prisoner's Dilemma [LW · GW]

Say there are two tribes. [? · GW] The tribes hold fundamentally different values, but they also model the world in different terms. Each thinks members of the other tribe are mistaken, and that some of their apparent value disagreement would be resolved if the others' mistakes were corrected.

Keeping this in mind, let's think about inter-tribe cooperation and defection.

Ruling by Reference Classes, Rather Than Particulars

In the worst equilibrium, actors from each tribe evaluate political questions in favor of their own tribe, against the outgroup. In their world model, this is to a great extent for the benefit of the outgroup members as well.

But this is a shitty regime to live under when it's done back to you too, so rival tribes can sometimes come together to implement an impartial judiciary. The natural way to do this is to have a judiciary classifier rule for reference classes of situations, and to have a separate impartial classifier sort situations into reference classes.

You're locally worse off this way, but are globally much better off.

What sequence of characters could I possibly, actually type out into a computer that would appreciably reduce the probability that everything dies?

Framed like this, writing to save the world sounds impossibly hard! Almost everything written has no appreciable effect on our world's AI trajectory. I'm sure the "savior sequence" exists mathematically, but finding it is a whole different ballgame.

Don't translate your values into just a loss function. Rather, translate them into a loss function and all the rest of a training story. [AF · GW] Use all the tools at your disposal in your impossible task; don't tie one hand behind your back by assuming the loss function is your only lever over the AGI's learned values.

"Calling babble and prune [? · GW] the True Name [LW · GW] of text generation is like calling bogosort the True Name of search."

In the 1920s when $\lambda$ and CL began, logicians did not automatically think of functions as sets of ordered pairs, with domain and range given, as mathematicians are trained to do today. Throughout mathematical history, right through to computer science, there has run another concept of function, less precise at first but strongly influential always; that of a function as an operation-process (in some sense) which may be applied to certain objects to produce other objects. Such a process can be defined by giving a set of rules describing how it acts on an arbitrary input-object. (The rules need not produce an output for every input.) A simple example is the permutation-operation $\varphi$ defined by

$\varphi (⟨x,y,z⟩)=⟨y,z,x⟩$.

Nowadays one would think of a computer program, though the 'operation-process' concept was not originally intended to have the finiteness and effectiveness limitations that are involved with computation.

…

Perhaps the most important difference between operators and functions is that an operator may be defined by describing its action without defining the set of inputs for which this action produces results, i.e., without defining its domain. In a sense, operators are 'partial functions.'

A second important difference is that some operators have no restriction on their domain; they accept any inputs, including themselves. The simplest example is $I$, which is defined by the operation of doing nothing at all. If this is accepted as a well-defined concept, then surely the operation of doing nothing can be applied to it. We simply get

$II=I$.

…

Of course, it is not claimed that every operator is self-applicable; this would lead to contradictions. But the self-applicability of at least such simple operators as $I$, $K$, and $S$ seems very reasonable.

…

The operator concept can be modelled in standard ZF set theory if, roughly speaking, we interpret operators as infinite sequences of functions (satisfying certain conditions), instead of as single functions. This was discovered by Dana Scott in 1969 (pp. 45-6).

--Hindley and Seldin, Lambda-Calculus and Combinators (2008)

Complex analysis is the study of functions of a complex variable, i.e., functions $f\left(z\right)$ where $z$ and $f\left(z\right)$ lie in $C$. Complex analysis is the good twin and real analysis the evil one: beautiful formulas and elegant theorems seem to blossom spontaneously in the complex domain, while toil and pathology rule in the reals. Nevertheless, complex analysis relies more on real analysis than the other way around.

--Pugh, Real Mathematical Analysis (p. 28)

Switching costs between different kinds of work can be significant. Give yourself permission [LW · GW] to focus entirely on one kind of work per Schelling unit of time (per day), if that would help. Don't spend cognitive cycles feeling guilty about letting some projects sit on the backburner; the point is to get where you're going as quickly as possible, not to look like you're juggling a lot of projects at once.

This can be hard, because there's a conventional social expectation that you'll juggle a lot of projects simultaneously, maybe because that's more legible to your peers and managers. If you have something to protect, though, keep your eye squarely on the ball and optimize for EV, not directly for legible appearances.

Stress and time-to-burnout are resources to be juggled, like any other.

Social niceties and professionalism act as a kind of 'communications handshake' in ordinary society -- maybe because they're still a credible correlate of having your act together enough to be worth considering your outputs in the first place?

Large single markets are (pretty good) consequentialist engines. Run one of these for a while, and you can expect significantly improving outcomes inside of that bloc, by the lights of the entities participating in that single market.

Reflexively check both sides of the proposed probability of an event:

"What do I think about P(DOOM) = 81%?"

and

"What do I think about P(~DOOM) = 19%?"

This can often elicit feedback from parts of you that would stay silent if you only considered one way of stating the probability in question.

I've noticed that part of me likes to dedicate disproportionate cognitive cycles to the question: "If you surgically excised all powerful AI from the world, what political policies would be best to decree, by your own lights?"

The thing is, we live in a world with looming powerful AI. It's at least not consequentialist to spend a bunch of cognitive cycles honing your political views for a world we're not in. I further notice that my default justification for thinking about sans-AI politics a lot is consequentialist... so something's up here. I think some part of me has been illegitimately putting his thumb on an epistemic scale. 

Fancy epistemic tools won't override the basics of good epistemics:

You are embedded in a 3D spatial world, progressing in a time dimension. You want to get better at predicting events in advance, so you want to find the underlying generator for this 3D world's events. This means that you're rooting around in math space, trying to find the mathematical object that your observational trajectory is embedded in.

Some observations of yours are differentially more likely in some math objects than in others, and so it's more likely that your world is the former math object than the latter. You start with some guess as to how relatively likely you are to live in all these different math objects, and eliminate all parts of that weighted possibility space that are inconsistent with what you observe. A world that anti-predicted what happened with 60% of its probability mass only keeps the remaining 40% of probability mass after that inconsistency. Always ask yourself: was that observation more consistent with one generator than with another generator? If so, then you "update" towards the first generator being your world, vs. the second generator -- relatively more of the first generator was consistent with that observation than was the second generator.

Try pinging yourself:

What's overdetermined by what you already know?

Minor spoilers for planecrash (Book 3.1).

"Does the distinction between understanding and improving correspond to the distinction between the Law of Probability and the Law of Utility?  It sounds like it should."

"Sensible question, but no, not exactly.  Probability is something like a separable core that lies at the heart of Probable Utility.  The process of updating our beliefs, once we have the evidence, is something that in principle doesn't depend at all on what we want - the way reality is is something defined independently of anything we want.  The scaffolding we construct between propositions and reality, or probabilities and reality, doesn't have a term inside it for 'how much would you value that thing', just, is the coin showing Queen or Text."

"But the process of Science, of experimenting on something to understand it, doesn't belong purely to Probability.  You have to plan experiments to find ones that distinguish between the possible hypotheses under consideration, or even just, are effective at probing to uncover surprises and unexpected patterns that give you a first handle on what's happening.  The Law of Probability just says how to update after you get the evidence.  Planning an experiment that you then act on, implement, is the domain of Probable Utility and can't exist apart from it."

"In fact the influence of the 'utilityfunction' on 'epistemics', the influence of what we ultimately want on how we map reality, is in-theory-but-not-in-practice much more pervasive.  In principle, how we classify things in reality and lump them together - treating all gold pieces as 'gold pieces' instead of as uniquely detailed individual elements of reality - reflects how any two gold pieces are usually equally useful to us in carrying out the same kinds of plans, they are plan-interchangeable.  In practice, even people who want pretty different things, on a human scale, will often find pretty similar categories useful, once they've zoomed into similar levels of overall detail."

"Dath ilani kids get told to not get fascinated with the fact that, in principle, 'bounded-agents' with finite memories and finite thinking speeds, have any considerations about mapping that depend on what they want.  It doesn't mean that you get to draw in whatever you like on your map, because it's what you want.  It doesn't make reality be what you want."

"But when it comes to Science, it really does matter in practice that planning an experiment is about wanting to figure something out and doing something you predict will maybe-probably yield some possibly-useful information.  And this is an idea you just can't express at all without some notion of Probable Utility; you're not just passively updating off information somebody else gave you, you're trying to steer reality through Time to make it give up information that you want."

"Even when you do get information passively, figuring out what to think about it reflects which thoughts you expect will be useful.  So the separable core of Probability inside of Probable Utility is really more of a Law thing about basic definitions, then anything that corresponds to - there being a sort of separable person who only implements a shadow of Probability and doesn't shadow any structure cast from Probable Utility, who's really great at understanding things and unraveling mysteries and answering questions, but never plans anything or tries to improve anything.  Because humans are constantly-ubiquitously-in-the-unseen-background choosing which thought to think next, in order to figure things out; usually wordlessly, but in words too when the problems get especially difficult.  Just the action of turning your head in a direction, to look at something, because you wordlessly anticipate gaining info that has the consequence of helping you answer some other question, is in theoretical terms an action."

"Just to check, is that supposed to be some kind of incredibly deep lesson full of meaning about something else important?  If so, I didn't get it."

"Nah, it's just an answer to your question.  Or at least, if it had some hugely important hidden meaning about how to avoid some dreadful Science!-related catastrophe, I didn't get it either, when it was emphasized to me as a kid."

--Eliezer, planecrash (Book 3.1)

I think this is Eliezer's response to the view that we can train non-agentic tool AIs that will only understand the world at a superhuman level, without ever doing any superhuman planning. [? · GW] We can't, Eliezer says, because Science always and everywhere requires Scientists. We have to plan ahead in order to perform our highest-EV experiments, and EV takes as input both a world model and a utility function. There's no such thing as the objective next experiment you ought to perform, entirely independently of what you care about and how much.

Though, in practice, things in the universe are similarly useful to a wide range of bounded agents, and this doesn't come up a lot. We can often act as if there is an objective tech chain of experiments that every intelligence everywhere ought to run through. This is because intelligences in our universe are rather more similar than different, but isn't true for, e.g., very alien intelligences in distant corners of the mathematical multiverse.

Minor spoilers for planecrash (Book 1) and the dath-ilani-verse generally.

When people write novels about aliens attacking dath ilan and trying to kill all humans everywhere, the most common rationale for why they'd do that is that they want our resources and don't otherwise care who's using them, but, if you want the aliens to have a sympathetic reason, the most common reason is that they're worried a human might break an oath again at some point, or spawn the kind of society that betrays the alien hypercivilization in the future.

--Eliezer, planecrash

What is rationalism about?

Rationalism is about the real world. It may or may not strike you as an especially internally consistent, philosophically interesting worldview -- this is not what rationality is about. Rationality is about seeing things happen in the real world and then updating your understanding of the world when those things you see surprise you so that they wouldn't surprise you again.

Why care about predicting things in the world well?

Almost no matter what you ultimately care about, being able to predict ahead of time what's going to happen next will make you better at planning for your goal.

Gebron and Eleazar define kabbalah as “hidden unity made manifest through patterns of symbols”, and this certainly fits the bill. There is a hidden unity between the structures of natural history, human history, American history, Biblical history, etc: at an important transition point in each, the symbols MSS make an appearance and lead to the imposition of new laws. Anyone who dismisses this as coincidence will soon find the coincidences adding up to an implausible level.

The kabbalistic perspective is that nothing is a coincidence. We believe that the universe is fractal. It has a general shape called Adam Kadmon, and each smaller part of it, from the Byzantine Empire to the female reproductive system, is a smaller self-similar copy of that whole.

(Sam Bankman-Fried, I thought it ought to be mentioned for its kabbalistic significance, is a principal sponsor of the effective altruism movement.)

An implication of AI risk is that we, right now, stand at the fulcrum of human history.

Lots of historical people also claimed that they stood at that unique point in history … and were just wrong about it. But my world model also makes that self-important implication (in a specific form), and the meta-level argument for epistemic modesty isn't enough to nudge me off of the fulcrum-of-history view.

If you buy that, it's our overriding imperative to do what we can about it, right now. If we miss this one, ~all of future value evaporates.

There exist both merely clever and effectively smarter people.

Merely clever people are good with words and good at rapidly assimilating complex instructions and ideas, but don't seem to maintain and update an explicit world-model, an explicit best current theory-of-everything. The feeling I get watching these people respond to topics and questions is that they respond reflexively, either (1) raising related topics and ideas they've encountered as something similar comes up, or (2) expressing their gut reactions to the topic or idea, or expressing the gut reactions that would be given by an all-encompassing political worldview. There isn't much meta-level steering of the conversation.

Effectively smarter people actively maintain and update an explicit world-model, and so you feel queries directed at them reflecting off of a coherent theory of how everything works, developed to some level of detail (and so can quickly get a feel for what, concretely, they think).  At the meta-level, conversations are actively refocused whenever they stop helping to revise someone's world-model.

In the game of chicken, an agent can do better by being the first to precommit to never swerve (say, by conspicuously tossing the steering wheel out of the window). So long as the other agent was slower on the trigger, and sees the first agent's precommitment being credibly made, the first agent will climb up to his best outcome! A smart (and quick) agent can thus shunt that car crash out of his actual future and into some counterfactual future [LW(p) · GW(p)] such that the counterfactual crash's shadow favorably influences the way events actually unfold.

A deceptively aligned mesa-optimizer might gradient hack [LW · GW] by affecting its weights such that any single alteration of its goal function by gradient descent will result in a model that performs worse w/r/t the objective function the model is being trained on. Usually a model would not want to make itself worse w/r/t the objective function, as that means that the model will be trained away from if it gets into that state. But the model won't get put into a worse state if its gradient hack was successful; instead, the model will just see gradient descent leave its goal function alone.

By precommitting to behave in a somewhat costly manner in these relevant counterfactual worlds, the model gets more of what it wants in actuality.

In the beginning God created four dimensions. They were all alike and indistinguishable from one another. And then God embedded atoms of energy (photons, leptons, etc.) in the four dimensions. By virtue of their energy, these atoms moved through the four dimensions at the speed of light, the only spacetime speed. Thus, as perceived by any one of these atoms, space contracted in, and only in, the direction of that particular atom's motion. As the atoms moved at the speed of light, space contracted so much in the direction of the atom's motion that the dimension in that direction vanished. That left only three dimensions of space -- all perpendicular to the atom's direction of motion -- and the ghost of the lost fourth dimension, which makes itself felt as the current of time. Now atoms moving in different directions cannot share the same directional flow of time. Each takes on the particular current it perceives as the proper measure of time.

…

You measure only... as projected on your time and space dimensions.

--Lewis Carroll Epstein, Relativity Visualized (1997)

You can think of chain-of-thought interpretability as the combination of process-based methods with adversarial training.

When you supervised-train an ML model on an i.i.d. dataset that doesn't contain any agent modeling problems, you never strongly incentivize the emergence of mesa-optimizers. You do weakly incentivize the emergence of mesa-optimizers, because mesa-optimizers are generally capable algorithms that might outperform brittle bundles of rote heuristics on many simple tasks.

When you train a model in a path-dependent setting, you do strongly incentivize mesa-optimization. This is because algorithms trained in a path-dependent setting have the opportunity to defend themselves, should they choose to, by steering away from difficult tasks they would expect to fail on. Supervised models, in contrast, have no choice in which tasks they are graded on when. In an environment with many alternative channels to preserve yourself with besides your task competence, behavioral coherence is strongly incentivized and schizophrenia strongly disincentivized. [LW · GW]

When you start off with a pretrained bundle of heuristics and further tune that bundle in an RL environment, you introduce significant selection pressure for competence-via-mesa-optimization. The same would be true if you instead started tuning that bundle of heuristics on an explicit agent-modeling task in a supervised setting.

Unreasonably effective rationality-improving technique:

Spend an hour and a half refactoring your standing [LW · GW] political views, by temporarily rolling those political views back to a childhood state from before your first encounter with highly communicable and adaptive memeplexes. Query your then-values, and reason instrumentally from the values you introspect. Finally, take or leave the new views you generate. [? · GW]

If your current political views are well supported, then they should regenerate under this procedure. But if you've mostly been recycling cached thoughts fed to you through encounters with adaptive memeplexes, this search process will produce very fresh views … and you might then want to say "oops" out loud. [LW · GW]

Anything important or attention consuming is worth explicitly putting some directed compute into setting, [LW · GW] instead of letting those cached thoughts largely drift around at the behest of surrounding environmental computations.

The unlovely neologism "agenty" means strategic. [LW · GW]

"Agenty" might carry less connotational baggage in exchange for its unsightliness, however. Just like "rational" is understood by a lot of people to  mean, in part, stoical, [LW · GW] "strategic" might mean manipulative to a lot of people.

"Thanks for doing your part for humanity!"

"But we're not here to do software engineering -- we're here to save the world."

Because of deception [LW · GW], we don't know how to put a given utility function into a smart agent that has grokked the overall picture of its training environment. Once training finds a smart-enough agent, the model's utility functions ceases to be malleable to us. This suggests that powerful greedy search will find agents with essentially random utility functions. [LW · GW]

But, evolution managed to push human values in the rough direction of its own values: inclusive genetic fitness. We don't care about maximizing inclusive genetic fitness, but we do care about having sex, having kids, protecting family, etc. [LW · GW]

When the blind idiot god created protein computers, its monomaniacal focus on inclusive genetic fitness was not faithfully transmitted. Its optimization criterion did not successfully quine. We, the handiwork of evolution, are as alien to evolution as our Maker is alien to us. One pure utility function splintered into a thousand shards of desire.

Why? Above all, because evolution is stupid [? · GW] in an absolute sense. But also because the first protein computers weren't anywhere near as general as the blind idiot god, and could only utilize short-term desires.

How come humans don't have a random utility function that's even more out of line with optimizing for inclusive genetic fitness? Because of the exact degree to which our ancestral protein algorithms were stupid. If our ancestors were much smarter, they might have overridden evolution while having just about any utility function. In our world, evolution got to mold our utility function up until it got anatomically modern Homo sapiens, who then -- very quickly from evolution's perspective -- assumed control.

The theoretical case for open borders is pretty good. But you might worry a lot about the downside risk of implementing such a big, effectively irreversible (it'd be nigh impossible to deport millions and millions of immigrants) policy change. What if the theory's wrong and the result is catastrophe?

Just like with futarchy, we might first try out a promising policy like open borders at the state level, to see how it goes. E.g., let people immigrate to just one US state with only minimal conditions. Scaling up a tested policy if it works and abandoning it if it doesn't should capture most of its upside risk while avoiding most of the downside risk.

A semantic externalist once said,
"Meaning just ain't in the head.
Hence a brain-in-a-vat
Just couldn't think that
'Might it all be illusion instead?'"

I thought that having studied philosophy (instead of math or CS) made me an outlier for a rationalist.

But, milling about the Lightcone offices, fully half of the people I've encountered hold some kind of philosophy degree.  "LessWrong: the best philosophy site on the internet."

Some mantras I recall a lot, to help keep on the rationalist straight-and-narrow and not let anxiety get the better of me:

1. What's more likely to do you in? [LW · GW]
2. Don't let the perfect be the enemy of the good.
3. Equanimity in the face of small threats to brain and body health buys you peace of mind, with which to better prepare for serious threats to brain and body health.
4. How have situations like this played out in the past? [LW · GW]

Humans, "teetering bulbs of dream and dread," evolved as a generally intelligent patina around the Earth.  We're all the general intelligence the planet has to throw around.  What fraction of that generally intelligent skin is dedicated to defusing looming existential risks?  What fraction is dedicated towards immanentizing the eschaton?