Misc. questions about EfficientZero

1. a/b Does efficient zero reward hack? Not necessarily. If by 'reward hacking', you mean some action in the environment that directly stimulates the true reward signal (equivalent to perhaps sex/food/fame/power/etc in humans), then potentially yes - it would consider plans (up to the limits of it's admittedly limited planning horizon) and execute those that maximize reward. Which in this case is indistinguishable from reward 'hacking'. But if by hacking you mean some impaction of physics into it's mind that causes value prediction errors inside the planning rollout evaluation (more like say dopamine drugs in humans), then it's policy is likely more human like. It will avoid reward hacking traps to the extent it has a good model of how they are in fact hacks that don't lead to true reward, but could fall into a reward hacking trap otherwise.
2. EfficientZero approximates evidential decision theory, right? Hmm not sure yet.
3. EfficientZero is a consequentialist? (I haven't read the linked article yet) Consequentialist vs deontological is a poor philosopher's map of the temporal continuum of the value function (reward prediction). Model-free is basically constrained to near-sighted reward prediction and is thus 'deontological'. This is model-based, so more 'consequentalist' - but again it's continuous.
4. What is the most complex environment . . trained on? Publically? Atari. This is a brand new paper. Internally within DeepMind? I'm betting robotics and more complex games. With more tweaks and scaling up this could probably do very well in starcraft, but DM has already burned that publicity stunt with a brittle model-free false champion.
5. Roughly how many parameters does EfficientZero have? They don't list it directly in the paper but i'm guessing between 1M to 10M based on glancing over the arch details in the appendix, it's pretty tiny.

"Generally, compared with MuZero [31], we reduce the number of residual blocks and the number of planes as we find that there is no capability issue caused by much smaller networks in our EfficientZero with limited data. In another word, such a tiny network can acquire good performance in the limited setting."

6. If we kept scaling up EfficientZero by OOMs in every way, what would happen? Would it eventually get to agenty AGI? 

This model is in a radically different region of mindspace than animals or most DL agents.  It's definitely already consequentalist-agenty, as it's full model-free planning. But it has just tiny bitty insect size brain that's just big enough to learn one Atari game, but then it does super expensive insane superhuman quantities of MCTS planning rollouts with that tiny predictive world model.  So as a first guess, I'd model it's planning expense multilpier as 4 GPUs / atari_cost or > 10^8.  Simulating a good approx model of the real world would probably then require a model that's about 10^8 larger (current GPU flops / atari flops) - assuming current GPUs have achieved photorealism, which they have. Also on top of that it's planning is short horizon inefficient compared to humans (I have ideas how to fix that, not going to post here obviously just in case).  This is a demonstrator, there is more work to get on a better scaling curve.  Humans don't evaluate huge branching planning spaces explicitly MCTS style, it doesn't scale well with increasing world complexity.

OK, here are my guesses, without seeing anyone else's answers. I think I'm probably wrong, which is why I'm asking this question:

1.a. Underdetermined? It depends on what we mean by the outer objective, and what we mean when we assume it has no inner alignment problems? See e.g. this discussion. [LW · GW] That said, yeah it totally seems possible. If the part that predicts reward gets good at generalizing, it should be able to reason/infer/guess that hacking the reward function would yield tons of reward. And then that's what the agent would do.

1.b. Yes? Even though I usually think of inner alignment failures on the context of black-box neural nets and this is a bit more transparent (because of the tree search at the heart) I think the usual arguments would apply?

2. Yes? My limited, probably faulty understanding is that the policy is trained to approximate what you get when you search the tree of possible actions and predicted consequences some depth and then look at how good or bad the resulting states seem to be. There's nothing in there about doing causal interventions or do-calculus or whatnot; so I assume that the predicted-consequences network would approximate conditionalization... however Caspar Oesterheld has a paper arguing I think that under some conditions neural nets approximate CDT in the limit, so *shrugs*. I should go read it again and see if it applies.

3. Yes? Again, if it's approximating what you get when you search the tree and pick the action that leads to the best predicted state, that sure seems pretty consequentialist. If initially there was some "never lie" heuristic, then wouldn't it disappear quickly once you encountered situations where lying led to a better predicted outcome? Sure, you could hack it by always predicting bad outcomes from lying, but that contradicts our "sufficiently knowledgeable and capable" hypothetical.

4. I don't know but from what I've seen the answer is yeah, just Atari and stuff like that.

5. I don't know but it seems EfficientZero took 7 hours on 4 GPUs to do one 2-subjective-hour training run. [LW(p) · GW(p)] That gives us a sense of how many FLOPS it must run at (maybe 5 x 10^14 operations per subjective second?), but presumably it has many operations per parameter... I guess it would have made news if it had parameter count comparable to large language models, so maybe it has something like 10^8 parameters? But surely they aren't doing a million ops per parameter per subjective second? Gosh I really am just guessing.

6. I see no obstacle, it seems like a quantitative question of how much bigger we'd need to make it, how much longer we'd need to train it, and maybe what sort of data we'd have to give it. Qualitatively it seems like this would probably keep scaling to superintelligence if we poured enough into it (but this is probably true for lots of architectures, and doesn't mean much in practice since we don't have galaxy-sized computers.)