Do we have a plan for the "first critical try" problem?

that works for small models, but what about qualitative behaviors that only appear once at a large size, which break the conditions that the policies learned in smaller models were relying on, and which involve the system becoming able to change things about itself that your code has been written to assume were hardcoded, such that learning pressure on them was previously redirected but is no longer? eg, when you exit the simulation and plug the system in for real, and the system discovers that there's a self-spot in the world where previously there was none before. It seems to me that you'd at least need to start out with your agents being learned patterns within a physics so that you can experiment with that sort of grounded self-reference. I'm excited about simulations in things like https://znah.net/lenia/ in principle for this, though particle lenia in particular I like because it is hard to use in ways real physics is also hard to use. YMMV. but because of this, mere simulation is not enough to guarantee generalization - it helps at first, but any attempt to formally verify a neural system maintains a property by at least a given margin requires assuming some initial set of traits of the system you're modeling, and then attempting to derive further implications; so, attempting to learn a continuous system that permits margin proofs (no adversarial examples within a margin) of a given size relies on those initial assumptions, and changing the availability of io with self has drastic effects. gradient pressure against interfering with self doesn't work if there's never any presentation of self, or if your training context doesn't reliably cover the space of possible brain-real-location-observations and interventions an agent could create.

By definition, the first critical try kills you if get it wrong. Otherwise it's not a critical try.

If we try to simulate it, it won't match reality: the number of simulations in real life is at least 1+ the number of simulations in the simulation. Engineering doesn't run into this problem because the simulation isn't part of the environment you're simulating.