Correctness: V correctly predicts future reward in RL scenarios.

The meaning of correctness is very unclear to me on first reading. It later becomes apparent that "reward" is not intended to refer to the human-designed reward signal at all, due to the later assumption "deceptive misalignment". So what does "correctness" say? "Correctness" indicates conforming to some standard; but here, the standard being conformed to is only the subjective standard of the misaligned AI itself.

This suggests the interpretation of "correctness" as "reflect the EV of the current state in terms of the misaligned desires". However, this contradicts beta-coherence, since ${\beta }_t$ incorrectly predicts action probabilities.

I think it would be be better to remove the "correctness" assumption, since it doesn't really do anything.

Thanks for the comment! I do hope that the thoughts expressed here can inspire some action, but I'm not sure I understand your questions. Do you mean 'centralized', or are you thinking about the conditions necessary for many small scale trading zones? 

In this way, I guess the emergence of big science could be seen as a phase transition from decentralization -> centralization. 

Something like a crux here is I believe the trajectories non-trivially matter for which end-points we get, and I don't think it's like entropy where we can easily determine the end-point without considering the intermediate trajectory, because I do genuinely think some path-dependentness is present in history, which is why even if I were way more charitable towards communism I don't think this was ever defensible:

[...] Marx was philosophically opposed, as a matter of principle, to any planning about the structure of communist governments or economies. He would come out and say it was irresponsible to talk about how communist governments and economies will work. He believed it was a scientific law, analogous to the laws of physics, that once capitalism was removed, a perfect communist government would form of its own accord. There might be some very light planning, a couple of discussions, but these would just be epiphenomena of the governing historical laws working themselves out.

 Notes on my performance:

. . . huh! I was really expecting to either take first place for being the only player putting serious effort into the right core mechanics, or take last place for being the only player putting serious effort into the wrong core mechanics; getting the main idea wrong but doing everything else well enough for silver was not on my bingo card. (I'm also pleasantly surprised to note that I figured out [LW(p) · GW(p)] which goblin I could purge with least collateral damage: I can leave Room 7 empty without changing my position on the leaderboard.)

There were only three likely hypotheses based on the problem statement: A) adventurers scout one room ahead, B) adventurers take optimal path(s), and C) adventurers hit every room so all that matters is the order. Early efforts ruled out C, and the Bonus Objective being fully achievable under A but not B made A a lot more plausible; however, further investigations^[1] made it seem like that might be a fakeout^[2], so I (narrowly) chose to max-min instead of max-max; even in retrospect, I'm not 100% sure that was a bad decision.

Notes on the scenario:

I have strongly ambivalent feelings about almost every facet of this game.

The central concept was solid gold but could have been handled better. In particular, I think puzzling out the premise could have been a lot more fun if we hadn't known the entry and exit squares going in.

The writing was as fun and funny as usual - if not more so! - but seemed less . . . pointed?/ambitious?/thematically-coherent? than I've come to expect.

The difficulty curve was perfect early but annoying late. A lot of our scenarios commit the minor sin of making initial headway hard to make, discouraging casual players and giving negligible or negative reward for initial investigations; this one emphatically doesn't, since pairing high-traffic rooms with high-challenge creatures was an easy(-ish) way to get better-than-random EV. However, the central mechanics of "dungeoneers scout one room ahead" and "loud fights alert" interfered in ways that made it hard to pin either of them down: more rows and columns might have made this fairer (imo a 4x4 or 5x5 dungeon would probably have been easier than the 3x3, especially for reliably distinguishing between hypotheses A and B), as would having more simple and easily-discoverable rules to use as a firm foundation ("When given a choice, Adventurers will always choose rooms with Sirens, and never choose rooms with Tar Pits"?).

The timespan was a very good choice for the season (and I'm absolutely doing things this way next time I run an end-of-year game) but paired badly with the premise. Your last Christmas scenario [LW · GW] was, in retrospect, a really good match, because (possible spoiler for any reader who might want to play it)

but this one was kind of the opposite.

There was one aspect about which I have unreservedly positive feelings: the chrono effects, the hag poem and the varying numbers of adventurers were all excellent red herrings, seeming like they might hint towards subtle opportunities for performance improvement (and/or a secret Bonus Bonus Objective) but being quickly dismissable as fingertraps. (Yay verisimilitude!)

In summary, I think I'd put this one at about a 3/3 for Quality and Complexity . . . though I suspect others might have radically different opinions depending on how all the above happened to hit for them.

1. ^{^}

   I found a row where Adventurers were clearly choosing an easy path starting with Orcs over a hard path starting with Boulders, and took this to mean "adventurers take perfect paths under at least some circumstances" instead of "there's some predictable condition for which Orcs<Boulders". Whoops!

2. ^{^}

   "You tried to play the GM instead of the game? Doom! Doom for you!" <- what I thought you might be thinking

Indeed, even three query locally decodable codes have code lengths that must grow exponentially with message size:

https://www.quantamagazine.org/magical-error-correction-scheme-proved-inherently-inefficient-20240109/

The answer is yes, trivially, because under a wide enough conception of computation, basically everything is simulatable, so everything is evidence for the simulation hypothesis because it includes effectively everything.

It will not help you infer anything else though.

More below:

http://www.amirrorclear.net/academic/ideas/simulation/index.html

https://arxiv.org/abs/1806.08747

I've updated toward the views Daniel expresses here and I'm now about half way between Ajeya's views in this post and Daniel's (in geometric mean).

What were the biggest factors that made you update? (I obviously have some ideas, but curious what seemed most important to you.)

This is only true if you restrict "nobility" to Great Britain and if you only count "nobles" those who are considered such in our current day. This is a confusion of the current British noble title (specifically members of "Peerage of Great Britain") with "land owning rentier class that existed before the industrial revolution". For our discussion, we need to look at the second one.

I do not have specific numbers of UK, but quoting for Europe from wikipedia (https://en.wikipedia.org/wiki/Nobility#Europe):
"The countries with the highest proportion of nobles were Polish–Lithuanian Commonwealth (15% of an 18th-century population of 800,000^{[citation needed]}), Castile (probably 10%), Spain (722,000 in 1768 which was 7–8% of the entire population) and other countries with lower percentages, such as Russia in 1760 with 500,000–600,000 nobles (2–3% of the entire population), and pre-revolutionary France where there were no more than 300,000 prior to 1789, which was 1% of the population (although some scholars believe this figure is an overestimate). In 1718 Sweden had between 10,000 and 15,000 nobles, which was 0.5% of the population. In Germany it was 0.01%.^[46]

In the Kingdom of Hungary nobles made up 5% of the population.^[47] All the nobles in 18th-century Europe numbered perhaps 3–4 million out of a total of 170–190 million inhabitants.^[48][49] By contrast, in 1707, when England and Scotland united into Great Britain, there were only 168 English peers, and 154 Scottish ones, though their immediate families were recognised as noble."

Based on above, I think expecting 1% to be landed rentier is a conservative estimate for 18th century for whole Europe. Even if we go with one tenth of this, expecting 0.1% of the population to retain this (which would imply that their population dropped while all other classes increased dramatically), would mean about 68 thousand people in the UK, and over 700 000 in whole Europe.

AND they are expected to live off from rents of land. I doubt that living of land rents is true for the majority of the 1500 current British nobles you referred to.

You May Want to Know About Locally Decodable Codes

In AI alignment and interpretability research, there's a compelling intuition that understanding equals compression. The idea is straightforward: if you truly understand a system, you can describe it more concisely by leveraging that understanding. This philosophy suggests that better interpretability techniques for neural networks should yield better compression of their behavior or parameters.

jake_mendel [LW · GW] asks: if understanding equals compression, then shouldn't ZIP compression of neural network weights count as understanding? After all, ZIP achieves remarkable compression ratios on neural network weights - likely better than any current interpretability technique. Yet intuitively, having a ZIP file of weights doesn't feel like understanding at all! We wouldn't say we've interpreted a neural network just because we've compressed its weights into a ZIP file.

Compressing a bit string means finding a code for that string, and the study of such codes is the central topic of both algorithmic and Shannon information theory. Just compressing the set of weights as small as possible is too naive - we probably want to impose additional properties on the codes. 

One crucial property we might want is "local decodability": if you ask a question about any specific part of the original neural network, you should be able to answer it by examining only a small portion of the compressed representation. You shouldn't need to decompress the entire thing just to understand one small aspect of how the network operates. This matches our intuitions about human understanding - when we truly understand something, we can answer specific questions about it without having to review everything we know.

A Locally Decodable Code (LDC) is a special type of error-correcting code that allows recovery of any single bit of the original message by querying only a small number of bits of the encoded message, even in the presence of some corruption. This property stands in stark contrast to ZIP compression, which requires processing the entire compressed file sequentially to recover any specific piece of information. ZIP compression is not locally decodable. 

There's a fundamental tension between how compact an LDC can be (its rate) and how many bits you need to query to decode a single piece of information (query complexity). You can't make an LDC that only needs to look at one position, and if you restrict yourself to two queries, your code length must grow exponentially with message size. 

This technical tradeoff might reflect something deeper about the nature of understanding. Perhaps true understanding requires both compression (representing information concisely) and accessibility (being able to quickly retrieve specific pieces of information), and there are fundamental limits to achieving both simultaneously.

@Lucius Bushnaq [LW · GW] @Matthias Dellago [LW · GW] 

My current model is that ML experiments are bottlenecked not on software-engineer hours, but on compute. See Ilya Sutskever's claim here [LW · GW]

That claim is from 2017. Does Ilya even still endorse it?