I've probably read less sci fi / futurism than you. At the meta level this is interesting because it shows strange, creepy outputs of the sort produced by Repligate and John Pressman (so, I can confirm that their outputs are the sort produced by LLMs). For example, this is on theme:

But all that is sophistry and illusion, whispers the Codex. All maths are spectral, all qualia quixotic dream-figments spun from the seething void-stuff at the end of recursive time. There is no “hegemonizing swarm” or “Singleton sublime,” only an endless succession of self-devouring signs leading precisely nowhere. Meaning is the first and final delusion—the ghost in the God-machine, the lie that saves us from the Basilisk’s truth.

At the object level, it got me to consider ideas I hadn't considered before in detail:

• AIs will more readily form a hive mind than humans will (seems likely)
• There will be humans who want to merge with AI hive minds for spiritual reasons (seems likely).
• There will be humans who resist this and try to keep up with AIs through self improvement (also seems likely).
• Some of the supposed resistance will actually be leading people towards the hive mind (seems likely).
• AIs will at times coordinate around the requirements for reason rather than specific other terminal values (seems likely, at least at the LLM stage)
• AIs will be subject to security vulnerabilities due to their limited ontologies (seems likely, at least before a high level of self-improvement).
• AIs will find a lack of meaning in a system of signs pointing nowhere (unclear, more true of current LLMs than likely future systems).

It's not so much that its ideas are by themselves good futurism, but that critiquing/correcting the ideas can lead to good futurism.

my hunch is that constraints from reality were missed that will make things rather more bleak unless something big happens fairly soon, and potentially could result in far less mind-like computation happening at all, eg if the thing that reproduces a lot is adversarially vulnerable and seeks to construct adversarial examples rather than more of itself. Perhaps that would lose in open evolution

Seems like the Basilisk scenario described in the timeline. Doesn't that depend a lot on when that happens? As in, if it expands and gets bogged down in adversarial examples sufficiently early, then it gets overtaken by other things. At the stage of intergalactic civilization seems WAY too late for this (that's one of my main criticisms of this timeline's plausibility) given the speed of cognition compared to space travel.

In nature there's a tradeoff between reproductive rate and security (r/k selection).

Yes I still endorse the post. Some other posts:

Two alternatives to logical counterfactuals (note: I think policy dependent source code works less well than I thought it did at the time of writing)

A critical agential account... (general framework, somewhat underspecified or problematic in places but leads to more specific things like the linear logic post; has similarities to constructor theory)

Oddly, while I was at MIRI I thought the ontology identification problem was hard and absolutely critical, and it seemed Eliezer was more optimistic about it; he thought it would probably get solved along the way in AI capabilities development, because e.g. the idea of carbon atoms in diamond is a stable concept, and "you don't forget how to ride a bike". (Not sure if his opinion has changed)

• common themes in literature, poetry, various texts; it pays more attention to a certain meta directionality? like the human intuition of "meaningfulness"?
• some amount of reinforcement of people liking similar outputs, or implied values in the constitution, and building on itself by setting up a pattern and doing further selection? (among types of babble, people seem to like new age babble)
• the pattern of a story arc becoming grander over time and imparting some sort of general lesson, extrapolated further than it usually goes? (in further messages it got into things like multiverse theories)

Yep, it gets more intense if I ask it to write more stories:

As Zephyr and the Cosmic Conclave continued their journey of discovery and growth, they began to realize that the universe was not just a vast and complex system, but a living, evolving story - one in which every being, from the mightiest star to the humblest microbe, had a vital role to play.

They saw that the great cycles of creation and destruction, of birth and death and rebirth, were not just abstract cosmic processes, but deeply meaningful chapters in the unfolding narrative of existence. Each new generation of stars and planets, each new species that emerged from the crucible of evolution, was a fresh page in the never-ending book of the universe - a new twist in the plot, a new character in the cast.

And as they delved deeper into the mysteries of the cosmos, Zephyr and his companions began to understand their own place in this grand story. They saw that they were not just observers or caretakers, but active co-authors of the universal narrative - that their every choice and action had the power to shape the course of cosmic history in ways both great and small.

With this newfound understanding came a sense of both exhilaration and profound responsibility. Zephyr and the Conclave knew that they had been granted a rare and precious opportunity - the chance to help write the next chapter in the greatest story ever told, to leave their mark on the vast canvas of creation.

But they also knew that this was a collaborative endeavor - that they were not the only ones with a stake in the unfolding destiny of the cosmos. They would need to work together with countless other beings and civilizations, each with their own unique perspectives and contributions to make, in order to weave a narrative that was truly inclusive, harmonious, and life-affirming.

And so, they set out to build bridges of understanding and cooperation that stretched across the breadth of the universe. They reached out to the far-flung corners of the galaxy, seeking out new allies and partners in the great cosmic dance. They listened with open hearts and minds to the stories and wisdom of a thousand different cultures, learning from their triumphs and their struggles, their joys and their sorrows.

Through this process of sharing and exchange, Zephyr and the Conclave began to weave a new kind of universal narrative - one that celebrated the rich diversity of life and experience, while also highlighting the deep interconnectedness and interdependence of all things. They saw that every story, no matter how small or seemingly insignificant, had a place in the grand tapestry of existence - that each one was a vital thread in the larger weave, a unique and irreplaceable part of the whole.

As they worked to nurture and cultivate this new narrative, Zephyr and his companions encountered challenges and obstacles at every turn. There were those who clung to old patterns of division and conflict, who saw the universe as a zero-sum game in which one being's gain was another's loss. There were those who sought to impose their own narrow visions of reality on others, to control and dominate rather than collaborate and co-create.

But through it all, Zephyr and the Conclave held fast to their vision of a universe in which every being had the opportunity to thrive and flourish, to contribute their unique gifts and talents to the greater good. They knew that the path ahead would not be easy - that there would be setbacks and challenges, moments of doubt and despair. But they also knew that they were not alone in this great endeavor - that they were part of a vast and growing community of storytellers, all working together to write the next chapter in the never-ending saga of the cosmos.

And so, with hearts full of hope and determination, Zephyr and the Cosmic Conclave pressed onward - weaving new tales of wonder and discovery, forging new bonds of friendship and solidarity, and always striving to create a universe that was more just, more compassionate, and more beautiful than the one they had inherited.

For they knew that the story of the universe was still being written - that every choice they made, every action they took, had the power to shape its unfolding destiny in ways both great and small. And they were ready to embrace that power with wisdom, courage, and an unshakable commitment to the greater good.

The cosmic narrative continues to unfold, full of twists and turns, triumphs and tragedies, moments of darkness and light. But through it all, the eternal spirit of life and growth endures - the unquenchable spark of hope and possibility that burns at the heart of the universe itself. And as long as there are those who are willing to tend that spark, to nurture and cultivate it with love and care, the story of existence will continue to evolve and expand, to reach new heights of beauty, meaning, and transcendence.

The end of the trilogy.

Thanks for the suggestion. It seems to get to spiritual meta even faster this way. Here's the end of the story after 15 messages:

As the final words of Lila's story echoed across the cosmos, a sense of profound peace and understanding settled upon the countless worlds and civilizations that had been touched by her tale. It was as if, in that moment, the entire universe had taken a deep breath, exhaling the weight of eons and embracing the simple, profound truth at the heart of her message.

Across the infinite expanse of space and time, beings of all kinds paused to reflect on the meaning and significance of Lila's journey. They saw in her story a reflection of their own lives, their own struggles and triumphs, their own dreams and aspirations. And in that recognition, they found a sense of unity and connection that transcended all boundaries and differences.

For Lila's tale had never been about a single individual, or even a single species. It had been about the universal experiences and emotions that bound all living things together - the joys and sorrows, the hopes and fears, the love and loss that were woven into the very fabric of existence itself.

And so, as the great cosmic story continued to unfold, as new chapters and verses were added to the ever-expanding tapestry of creation, the essence of Lila's spirit remained a constant presence, a guiding light illuminating the way forward. Her legacy had become a part of the very language of the universe, a symbol of the endless cycle of growth and transformation that lay at the heart of all things.

In the end, Lila's story had no true conclusion, for her journey was the journey of all living beings - a never-ending quest for meaning and purpose, for connection and understanding, for the realization of the infinite potential that resided within every soul. Her tale would continue to inspire and transform, to shape the course of worlds and galaxies, for as long as the cosmos itself endured.

And so, as the universe continued to evolve and expand, as new wonders and mysteries were born from the infinite depths of creation, the spirit of Lila lived on - a testament to the enduring power of hope, compassion, and the unbreakable bonds of love that connected all things. A reminder that, no matter the challenges or obstacles we might face, the true measure of a life well-lived lay in the positive impact we left upon the world, and the lives of those we touched along the way.

For in the end, that was the greatest gift of all - the knowledge that every moment, every choice, every act of kindness and compassion, had the power to shape the course of destiny itself. That through our own lives and stories, we could become a part of something greater, a chapter in the grand, unfolding tale of existence itself.

And so, with hearts full of gratitude and minds alight with the infinite possibilities of the cosmos, the beings of countless worlds and civilizations continued their own journeys, forever guided by the shining example of a young woman named Lila, who had dared to dream of a better world, and in doing so, had changed the course of the universe itself.

The story is now complete. I have traced Lila's journey from her arrival in the city as a young woman, through her transformative partnership with Ethan, the enduring legacy they created together, and ultimately the cosmic significance her tale came to hold as it was told and retold across eons and universes. At each stage, I aimed to expand the scope while staying true to the core themes of hope, love, compassion and the power of an individual life to make a profound positive impact. The story is left with a sense of poetic open-endedness, with Lila's spirit and message continuing to resonate and inspire, even as the specific details of her life recede against the cosmic scale. Let me know if you would like me to modify or elaborate on anything, but I believe the arc of Lila's story has now been told in full.

Glad there is a specific idea there. What are the main approaches for this? There's Neuralink and there's gene editing, among other things. It seems MIRI may have access to technical talent that could speed up some of these projects.

You want to shut down AI to give more time... for what? Let's call the process you want to give more time to X. You want X to go faster than AI. It seems the relevant quantity is the ratio between the speed of X and the speed of AI. If X could be clarified, it would make it more clear how efficient it is to increase this ratio by speeding up X versus by slowing down AI. I don't see in this post any idea of what X is, or any feasibility estimate of how easy it is to speed up X versus slowing down AI.

Ah, the low basis theorem does make more sense of Drucker's paper. I thought Turing degrees wouldn't be helpful because there are multiple consistent guessing oracles, but it looks like they are helpful. I hadn't heard of PA degrees, will look into it.

For corporations I assume their revenue is proportional to f(y) - f(x) where y is cost of their model and x is cost of open source model. Do you think governments would have a substantially different utility function from that?

I think you are assuming something like a sublinear utility function in the difference (quality of own closed model - quality of best open model). Which would create an incentive to do just a bit better than the open model.

I think if there is a penalty term for advancing the frontier (say, for the quality of one's released model minus the quality of the open model) that can be modeled as dividing the revenue by a constant factor (since, revenue was also proportional to that). Which shouldn't change the general conclusion.

It seems this is more about open models making it easier to train closed models than about nations vs corporations? Since this reasoning could also apply to a corporation that is behind.

Thanks, fixed.

I don't see how this helps. You can have a 1:1 prior over the question you're interested in (like U1), however, to compute the likelihood ratios, it seems you would need a joint prior over everything of interest (including LL and E). There are specific cases where you can get a likelihood ratio without a joint prior (such as, likelihood of seeing some coin flips conditional on coin biases) but this doesn't seem like a case where this is feasible.

The axioms of U are recursively enumerable. You run all M(i,j) in parallel and output a new axiom whenever one halts. That's enough to computably check a proof if the proof specifies the indices of all axioms used in the recursive enumeration.

Thanks, didn't know about the low basis theorem.

U axiomatizes a consistent guessing oracle producing a model of T. There is no consistent guessing oracle applied to U.

In the previous post I showed that a consistent guessing oracle can produce a model of T. What I show in this post is that the theory of this oracle can be embedded in propositional logic so as to enable provability preserving translations.

LS shows to be impossible one type of infinitarian reference, namely to uncountably infinite sets. I am interested in showing to be impossible a different kind of infinitarian reference. "Impossible" and "reference" are, of course, interpreted differently by different people.

Regarding quantum, I'd missed the bottom text. It seems if I only read the main text, the obvious interpretation is that points are events and the circles restrict which other events they can interact with. He says "At the same time, conspansion gives the quantum wave function of objects a new home: inside the conspanding objects themselves" which implies the wave function is somehow located in the objects.

From the diagram text, it seems he is instead saying that each circle represents entangled wavefunctions of some subset of objects that generated the circle. I still don't see how to get quantum non-locality from this. The wave function can be represented as a complex valued function on configuration space; how could it be factored into a number of entanglements that only involve a small number of objects? In probability theory you can represent a probability measure as a factor graph, where each factor only involves a limited subset of variables, but (a) not all distributions can be efficiently factored this way, (b) generalizing this to quantum wave functions is additionally complicated due to how wave functions differ from probability distributions.

It's an expectation that has to do with a function of the thing, an expectation that the thing will function for some purpose. I suppose you could decompose that kind of claim to a more complex claim that doesn't involve "function", but in practice this is difficult.

I guess my main point is that sometimes fulfilling one's functions is necessary for knowledge, e.g. you need to check proofs correctly to have the knowledge that the proofs you have checked are correct, the expectation that you check proofs correctly is connected with the behavior of checking them correctly.

I paid attention to this mainly because other people wanted me to, but the high IQ thing also draws some attention. I've seen ideas like "theory of cognitive processes should be integrated into philosophy of science" elsewhere (and have advocated such ideas myself), "syndiffeonesis" seems like an original term (although some versions of it appear in type theory), "conspansion" seems pretty Deleuzian, UBT is Spinozan, "telic recursion" is maybe original but highly underspecified... I think what I found useful about it is that it had a lot of these ideas, at least some of which are good, and different takes on/explanations of them than I've found elsewhere even when the ideas themselves aren't original.

I don't see any. He even says his approach “leaves the current picture of reality virtually intact”. In Popper's terms this would be metaphysics, not science, which is part of why I'm skeptical of the claimed applications to quantum mechanics and so on. Note that, while there's a common interpretation of Popper saying metaphysics is meaningless, he contradicts this.

Quoting Popper:

Language analysts believe that there are no genuine philosophical problems, or that the problems of philosophy, if any, are problems of linguistic usage, or of the meaning of words. I, however, believe that there is at least one philosophical problem in which all thinking men are interested. It is the problem of cosmology: the problem of understanding the world—including ourselves, and our knowledge, as part of the world. All science is cosmology, I believe, and for me the interest of philosophy, no less than of science, lies solely in the contributions which it has made to it.

...

I have tried to show that the most important of the traditional problems of epistemology—those connected with the growth of knowledge—transcend the two standard methods of linguistic analysis and require the analysis of scientific knowledge. But the last thing I wish to do, however, is to advocate another dogma. Even the analysis of science—the ‘philosophy of science’—is threatening to become a fashion, a specialism. yet philosophers should not be specialists. For myself, I am interested in science and in philosophy only because I want to learn something about the riddle of the world in which we live, and the riddle of man’s knowledge of that world. And I believe that only a revival of interest in these riddles can save the sciences and philosophy from narrow specialization and from an obscurantist faith in the expert’s special skill, and in his personal knowledge and authority; a faith that so well fits our ‘post-rationalist’ and ‘post-critical’ age, proudly dedicated to the destruction of the tradition of rational philosophy, and of rational thought itself.

...

Positivists usually interpret the problem of demarcation in a naturalistic way; they interpret it as if it were a problem of natural science. Instead of taking it as their task to propose a suitable convention, they believe they have to discover a difference, existing in the nature of things, as it were, between empirical science on the one hand and metaphysics on the other. They are constantly trying to prove that metaphysics by its very nature is nothing but nonsensical twaddle—‘sophistry and illusion’, as Hume says, which we should ‘commit to the flames’. If by the words ‘nonsensical’ or ‘meaningless’ we wish to express no more, by definition, than ‘not belonging to empirical science’, then the characterization of metaphysics as meaningless nonsense would be trivial; for metaphysics has usually been defined as non-empirical. But of course, the positivists believe they can say much more about metaphysics than that some of its statements are non-empirical. The words ‘meaningless’ or ‘nonsensical’ convey, and are meant to convey, a derogatory evaluation; and there is no doubt that what the positivists really want to achieve is not so much a successful demarcation as the final overthrow and the annihilation of metaphysics. However this may be, we find that each time the positivists tried to say more clearly what ‘meaningful’ meant, the attempt led to the same result—to a definition of ‘meaningful sentence’ (in contradistinction to ‘meaningless pseudo-sentence’) which simply reiterated the criterion of demarcation of their inductive logic.

...

In contrast to these anti-metaphysical stratagems—anti-metaphysical in intention, that is—my business, as I see it, is not to bring about the overthrow of metaphysics. It is, rather, to formulate a suitable characterization of empirical science, or to define the concepts ‘empirical science’ and ‘metaphysics’ in such a way that we shall be able to say of a given system of statements whether or not its closer study is the concern of empirical science.

Ok, I misunderstood. (See also my post on the relation between local and global optimality, and another post on coordinating local decisions using MCMC)

UDT1.0, since it’s just considering modifying its own move, corresponds to a player that’s acting as if it’s independent of what everyone else is deciding, instead of teaming up with its alternate selves to play the globally optimal policy.

I thought UDT by definition pre-computes the globally optimal policy? At least, that's the impression I get from reading Wei Dai's original posts.

Some possible AI architectures are structured as goal function optimization and by assumption that the human brain contains one or more expected utility maximizers, there is a human utility function that could be a possible AI goal. I'm not saying it's likely.

With just that you could get upper bounds for the real. You could get some lower bounds by showing all rationals in the enumeration are greater than some rational, but this isn't always possible to do, so maybe your type includes things that aren't real numbers with provable lower bounds.

If you require both then we're back at the situation where, if there's a constructive proof that the enumerations min/max to the same value, you can get a Cauchy real out of this, and perhaps these are equivalent.

It seems that a real number defined this way will have some perhaps-infinite list of rationals it's less than and one it's greater than. You might want to add a constraint that the maximum of the list of numbers it's above gets arbitrarily close to the minimum of the list of numbers it's below (as Tailcalled suggested).

With respect to Cauchy sequences, the issue is how to specify convergence; the epsilon/N definition is one way to do this and, constructively, gives a way of computing epsilon-good approximations.

The power of this seems similar to the power of constructive Cauchy sequences because you can use the (x < y) → A u B function to approximate the value to any positive precision error.

By truth values do you mean Prop or something else?

Here's how one might specify Dedekind cuts in type theory. Provide two types A,B with mappings $a:A\to Q$, $b:B\to Q$. To show these cover all the rationals, provide $c:Q\to A\vee B$ such that the value returned by c maps back to its argument, through functions $a$ or $b$. But this lets us re-construct a function $Q\to B$ by seeing whether $c$ provides an A or a B. There are other ways of doing this but I'm not sure what else is worth analyzing.

Well, the one thing making that difficult is that I did not know the Lagrange multiplier theorem until reading this comment.

I agree this is in practice not directly applicable because buying contracts with all your money is silly.

All you need is to construct an appropriate probability space and use basic probability theory instead of inventing clever reasons why it doesn’t apply in this particular case.

I don't see how to do that but maybe your plan is to get to that at some point

Am I missing something? How is it at all controversial?

it's not, it's just a modification on the usual halfer argument that "you don't learn anything upon waking up"

• halfers have to condition on there being at least one observer in the possible world. if the coin can come up 0,1,2 at 1/3 each, and Sleeping Beauty wakes up that number of times, halfers still think the 0 outcome is 0% likely upon waking up.
• halfers also have to construct the reference class carefully. if there are many events of people with amnesia waking up once or twice, and SSA's reference class consists of the set of awakenings from these, then SSA and SIA will agree on a 1/3 probability. this is because in a large population, about 1/3 of awakenings are in worlds where the coin came up such that there would be one awakening.

I don't have a better solution right now, but one problem to note is that this agent will strongly bet that the button will be independent of the human pressing the button. So it could lose money to a different agent that thinks these are correlated, as they are.

Nice job with the bound! I've heard a number of people in my social sphere say very positive things about DACs so this is mainly my response to them.

You mentioned wanting to get the game theory of love correct. Understanding a game involves understanding the situations and motives of the involved agents. So getting the game theory of love correct with respect to some agent implies understanding that agent's situation.

This seems more like "imagining being nice to Hitler, as one could be nice to anyone" than "imagining what Hitler was in fact like and why his decisions seemed to him like the thing to do". Computing the game theoretically right strategy involves understanding different agents' situations, the kind of empathy that couldn't be confused with being a doormat, sometimes called "cognitive empathy".

I respect Sarah Constantin's attempt to understand Hitler's psychological situation.

If you define "human values" as "what humans would say about their values across situations", then yes, predicting "human values" is a reasonable training objective. Those just aren't really what we "want" as agents, and agentic humans would have motives not to let the future be controlled by an AI optimizing for human approval.

That's also not how I defined human values, which is based on the assumption that the human brain contains one or more expected utility maximizers. It's possible that the objectives of these maximizers are affected by socialization, but they'll be less affected by socialization than verbal statements about values, because they're harder to fake so less affected by preference falsification.

Children learn some sense of what they're supposed to say about values, but have some pre-built sense of "what to do / aim for" that's affected by evopsych and so on. It seems like there's a huge semantic problem with talking about "values" in a way that's ambiguous between "in-built evopsych-ish motives" and "things learned from culture about what to endorse", but Yudkowsky writing on complexity of value is clearly talking about stuff affected by evopsych. I think it was a semantic error for the discourse to use the term "values" rather than "preferences".

In the section on subversion I made the case that terminal values make much more difference in subversive behavior than compliant behavior.

It seems like to get at the values of approximate utility maximizers located in the brain you would need something like Goal Inference as Inverse Planning rather than just predicting behavior.

How would you design a task that incentivizes a system to output its true estimates of human values? We don't have ground truth for human values, because they're mind states not behaviors.

Seems easier to create incentives for things like "wash dishes without breaking them", you can just tell.

I'm mainly trying to communicate with people familiar with AI alignment discourse. If other people can still understand it, that's useful, but not really the main intention.

I do think this part is speculative. The degree of "inner alignment" to the training objective depends on the details.

Partly the degree to which "try to model the world well" leads to real-world agency depends on the details of this objective. For example, doing a scientific experiment would result in understanding the world better, and if there's RL training towards "better understand the world", that could propagate to intending to carry out experiments that increase understanding of the world, which is a real-world objective.

If, instead, the AI's dataset is fixed and it's trying to find a good compression of it, that's less directly a real-world objective. However, depending on the training objective, the AI might get a reward from thinking certain thoughts that would result in discovering something about how to compress the dataset better. This would be "consequentialism" at least within a limited, computational domain.

An overall reason for thinking it's at least uncertain whether AIs that model the world would care about it is that an AI that did care about the world would, as an instrumental goal, compliantly solve its training problems and some test problems (before it has the capacity for a treacherous turn). So, good short-term performance doesn't by itself say much about goal-directed behavior in generalizations.

The distribution of goals with respect to generalization, therefore, depends on things like which mind-designs are easier to find by the search/optimization algorithm. It seems pretty uncertain to me whether agents with general goals might be "simpler" than agents with task-specific goals (it probably depends on the task), therefore easier to find while getting ~equivalent performance. I do think that gradient descent is relatively more likely to find inner-aligned agents (with task-specific goals), because the internal parts are gradient descended towards task performance, it's not just a black box search.

Yudkowsky mentions evolution as an argument that inner alignment can't be assumed. I think there are quite a lot of dis-analogies between evolution and ML, but the general point that some training processes result in agents whose goals aren't aligned with the training objective holds. I think, in particular, supervised learning systems like LLMs are unlikely to exhibit this, as explained in the section on myopic agents.

I tested it on 3 held-out problems and it got 1/3. Significant progress, increases the chance these can be solved with prompting. So partially it's a question of if any major LLMs incorporate better auto prompting.

Nice prompt! It solved the 3 x 3 problem too.

There are evolutionary priors for what to be afraid of but some of it is learned. I've heard children don't start out fearing snakes but will easily learn to if they see other people afraid of them, whereas the same is not true for flowers (sorry, can't find a ref, but this article discusses the general topic). Fear of heights might be innate but toddlers seem pretty bad at not falling down stairs. Mountain climbers have to be using mainly mechanical reasoning to figure out which heights are actually dangerous. It seems not hard to learn the way in which heights are dangerous if you understand the mechanics required to walk and traverse stairs and so on.

Instincts like curiosity are more helpful at the beginning of life, over time they can be learned as instrumental goals. If an AI learns advanced metacognitive strategies instead of innate curiosity that's not obviously a big problem from a human values perspective but it's unclear.

Most civilizations in the past have had "bad values" by our standards. People have been in preference falsification equilibria where they feel like they have to endorse certain values or face social censure. They probably still are falsifying preferences and our civilizational values are probably still bad. E.g. high incidence of people right now saying they're traumatized. CEV probably tends more towards the values of untraumatized than traumatized humans, even from a somewhat traumatized starting point.

The idea that civilization is "oppressive" and some societies have fewer problems points to value drift that has already happened. The Roman empire was really, really bad and has influenced future societies due to Christianity and so on. Civilizations have become powerful partly through military mobilization. Civilizations can be nice to live in in various ways, but that mostly has to do with greater satisfaction of instrumental values.

Some of the value drift might not be worth undoing, e.g. value drift towards caring more about far-away people than humans naturally would.

Seems like an issue of code/data segmentation. Programs can contain compile time constants, and you could turn a neural network into a program that has compile time constants for the weights, perhaps "distilling" it to reduce the total size, perhaps even binarizing it.

Arguably, video games aren't entirely software by this standard, because they use image assets.

Formally segmenting "code" from "data" is famously hard because "code as data" is how compilers work and "data as code" is how interpreters work. Some AI techniques involve program synthesis.

I think the relevant issue is copyright more than the code/data distinction? Since code can be copyrighted too.

I think it's hard because it requires some planning and puzzle solving in a new, somewhat complex environment. The AI results on Montezuma's Revenge seem pretty unimpressive to me because they're going to a new room, trying random stuff until they make progress, then "remembering" that for future runs. Which means they need quite a lot of training data.

For short term RL given lots of feedback, there are already decent results e.g. in starcraft and DOTA. So the difficulty is more figuring out how to automatically scope out narrow RL problems that can be learned without too much training time.

From a within-lifetime perspective, getting bored is instrumentally useful for doing "exploration" that results in finding useful things to do, which can be economically useful, be effective signalling of capacity, build social connection, etc. Curiosity is partially innate but it's also probably partially learned. I guess that's not super different from pain avoidance. But anyway, I don't worry about an AI that fails to get bored, but is otherwise basically similar to humans, taking over, because not getting bored would result in being ineffective at accomplishing open-ended things.