What would flourishing look like in Conway's Game of Life?

Thanks for the detailed response. Meta: It feels good to receive a signal that this was a 'neat question', or in general, a positive-seeming contribution to LW. I have several unexpressed thoughts from fear of not actually creating value for the community.

it sounds like what you want is a reward function that is simple, but somehow analogous to the complexity of human value? And it sounds like maybe the underspecified bit is "you, as a human, have some vague notion that some sorts of value-generation are 'cheating'", and your true goal is "the most interesting outcome that doesn't feel like Somehow Cheating to me?"

This is about correct. A secondary reason for simplicity is to attempt to be computationally efficient (for the environment that generates the reward).

"one cell == an atom"

I can see that as being a case, but, again, computational tractability. Actual interesting structures in GoL can be incredibly massive, for example, this Tetris Proccessor (2,940,928 x 10,295,296 cells). Maybe there's some middle ground between truly fascinating GoL patterns made from atoms and my cell-as-a-planet level abstraction, as suggested by Daniel Kokotajlo in another comment.

How 'good' is it to have a repeating loop of, say, a billion flourishing human lives? Is it better than a billion human lives that happens exactly once and ends?

Wouldn't most argue that, in general, more life is better than less life? (but I see some of my hidden assumptions here, such as "the 'life's we're talking about here are qualitatively similar e.g. the repeating life doesn't feel trapped/irrelevant/futile because it is aware that it is repeating")

I think "moral value" (or, "value") in real life is about the process of solving "what is valuable and how to do I get it?"

I don't disagree, but I also think this is sort of outside the scope of finite-space cellular automata.

In this case it might mean that the system optimizes either for true continuous novelty, or the longest possible loop?

Given the constraints of CA, I'm mostly in agreement with this suggestion. Thanks.

I do suspect that figuring out which of your assumptions are "valid" is an important part of the question here.

Yes, I agree. Concretely, to me it looks like 'if I saw X happening in GoL, and I imagine being a sentient being (at some scale, TBD) in that world (well, with my human values), then would I want to live in it?', and translating that into some rules that promote or disincentivise X.

I do think taking this approach is broadly difficult, though. Perhaps its worth getting a v0.1 out with reward being tied to instantiations of novel states to begin with, and then seeing whether to build on that or try a new approach.

If you want to be able to track "can we create a variety of good structures, plus avoid bad structures?" you could weight the structures differently, and label some "bad".

(I'm not sure this actually accomplishes the underlying philosophical goal, but seemed like the obvious extension from your current line of thinking)

Interesting thoughts, thanks. My concerns: 1) Diversity would be restricted to what I specify as interesting shapes, while perhaps what I really want is for the AI to be able to discover new ways to accomplish some target value. 2) From a technological perspective, may be too expensive to implement? (in that, at every pass, must search over all subsets of space and check against all (suitably-sized) patterns in the database in order to determine what reward to provide).

I notice something like "glider guns feel more valuable to me than gliders, and glider-gun-guns (is that a thing?) feel more valuable than glider-guns, but I wouldn't want the universe tiled with either of them."

Thinking about the question from another angle: What would I want the RL agent to end up doing? What would make me most pleased? Answer: If it constructed some sort of infinitely growing pattern, that spirals out from the middle filling the void with interesting oscillating structures of infinite diversity, such that eventually all stable structures of all sizes will be created. Even better would be if not all, but most, stable structures would be created -- and some category of "bad" ones would never be.

This is an elegant solution, but I worry that it will result in something that looks very boring and high-entropy. EDIT: After playing around a bit with GoL, I no longer am worried about this.

After reading through the suggestions, including yours and Raemon's, I'm also sort of circling around this idea. Thanks.

Thanks for the note. I'll let you know if my explorations take me that way.

Fascinating. Thanks. My sense is GoL already has this property; any intuitions on how to formalise it?