LessWrong 2.0 Reader

They used to consider speedrunning games a guilty pleasure, but after goal-factoring their supposed guilty pleasure concludes that the guilt doesn't align with their actual goals and values and feeling bad about enjoying speedrunning doesn't really serve any productive purpose, so now they enjoy speedrunning unabashedly. 

Predict your 2025: a website for recording probabilistic forecasts about your life and the world in the next year.

That TLDR is great! I've read the NVC book through twice and taken half of an online audio course. I've also never directly benefitted from NVC communication suggestions because they're framed awkwardly.

Your distillation rings true, but I have not made it or heard it. Thank you.

I'd just add to your TLDR something like:

Make requests but don't pressure people to do things. Try to be clear about why you're asking them to do things.

You do cover this but your TLDR is missing it. There's probably a better formulation, that's just a first random stab. That part is the nonviolent part.

It's interesting to note that LW communication usually does seem to follow those NVC principles.

This was a nice read but I don't have the slightest clue what "Goal-factored the idea of "guilty pleasures". Now loves speedrunning unabashedly" means

Latecomer, but as this relates to some of my prior work on self- and other-modeling, I thought I'd comment... The consistently high task accuracy displayed on Figure D suggests that even your smallest neural network is significantly over-capacity/over-parameterized for the test dataset. Excess capacity seems to be the only way the model can take on the expensive self-modeling task (*) without losing accuracy on the main task. Indeed, this would suggest that the explanation for the regularization benefit of self-modeling here is precisely that it soaks up the excess capacity, avoiding overfitting. But obviously, you can have too much of a good thing -- as the experiments with fewer hidden layers show, the attention weight can take over the model's focus and destroy accuracy. So it seems that, if you up the problem complexity/network size knob, the "maximum allowable attention weight" that doesn't compromise accuracy will tend to zero. On the other hand, one can think of simpler tasks than fully predicting all of a layer's activations -- for example, predicting the activation signs, the maximum-minimum range, the mean activation, etc. I want to say these seem more meaningful anyway, and a way to avoid Borges's "Map of the Empire whose size was that of the Empire", no? 

* BTW: Unless I missed it, the paper did not report the accuracy of the self-modeling task, only of the primary task, right? I must imagine it was far from perfect, as perfect self-modeling is only possible in trivial edge cases, right?

Note that “thinking through implications” for alignment is exactly the idea in deliberative alignment https://openai.com/index/deliberative-alignment/

Some notes

• Authors claim that just prompting o1 with full safety spec already allows it to figure out aligned answers
• The RL part is simply “distilling” this imto the model (see also, note from a while ago on RLHF as variational inference)
• Generates prompt, CoT, outcome traces from the base reasoning model. Ie data is collected “on-policy”
• Uses a safety judge instead of human labelling

What is the probability that there are not 3^^^3 anti-muggers out there who will kill 3^^^^^^3 people if I submit to the mugger? Not 0.
The original argument against Pascal's Wager does not require you to actually believe in any of the other god's, just that the probability of them existing and having the reverse utility is enough to cancel out the probability of Pascal being right.
 

And that is why the rewiring of the pubescent brain involves changes that enable that. As the brain can't hardwire changes to values, which are high-level learned, it has to go some other way.

• Change the ground truth feedback (brain stem rewiring).
• Weaken all previous connections (synaptic pruning?).
• Reduce learning rate (myelination).
• Something else? Changes to other hyperparameters (hormonal changes?).

John, you know much coding theory much better than I do so I am inclinced to defer to your superior knowledge...

Now behold the awesome power of gpt-Pro

Let’s unpack the question in pieces:

1. Is ZIP (a.k.a. DEFLATE) “locally decodable” or not?

• Standard ZIP files are typically not “locally decodable” in the strictest sense—i.e., you cannot start decoding exactly at the byte corresponding to your region of interest and reconstruct just that portion without doing some earlier decoding.

• The underlying method, DEFLATE, is indeed based on LZ77 plus Huffman coding. LZ77 uses back‐references to prior segments of the uncompressed data, which forces you to at least partially decode from an earlier point to resolve those references.

  However, many ZIP files are stored in blocks or chunks—sometimes called “deflate blocks.” So if you only need data near the start of a block (rather than in the middle of one), you can skip previous blocks and just decode that chunk. Still, you cannot trivially jump to an arbitrary byte offset inside the block without re-initializing the LZ77 state from somewhere earlier.

2. If ZIP is “based on Lempel–Ziv,” can we easily modify it for local decodability?

• Because Lempel–Ziv (LZ) methods revolve around referencing earlier uncompressed substrings, one must store enough information (or replicate the references with separate “partial dictionaries”) if one wants truly local decodability.
• One can certainly modify an LZ-based compression scheme so that each block is smaller, or that references never cross block boundaries, etc. That would allow a more local decoding property: you only decode from the start of the local block rather than from the entire file.
• This usually comes with tradeoffs. If your blocks are small, you lose compression efficiency because you can’t take advantage of references bridging across large swaths of data. So making a compression format “locally decodable” in a fine‐grained way often lowers the overall compression ratio.

3. Is Lempel–Ziv (and thus ZIP) lossless?

• Yes. Standard LZ77 (or LZ78, LZMA, etc.) is a lossless compression scheme: you can reconstruct the exact original data bit for bit. The DEFLATE algorithm used in ZIP is also lossless.

4. Does the picture change when doing lossy compression?

• Yes, quite a bit. In lossy compression (like JPEG for images or MP3/AAC for audio), parts of the original data are thrown away. Often you can still decode small “chunks” of the file—e.g., many lossy audio or video formats do decode in frames or segments. But the format internals (e.g., DCT coefficients in JPEG, wavelet transforms, etc.) differ a lot from LZ-based bit-by-bit exactness.
• In lossy formats that use transform blocks (like 8×8 pixel blocks in JPEG), you can decode each block locally, but you still have boundary effects, optional overlap between blocks, etc. So “local decodability” there can be simpler in some ways (each block is more independent), but it’s also less “exact” because we accept errors from the original data.

---

Short Summary

• ZIP/DEFLATE is not strictly locally decodable because LZ77 references require that you decode from (at least) the beginning of the current block.
• You can modify an LZ-based scheme (for instance, by using smaller blocks or restricting cross‐block references) to make it more locally decodable, but that typically sacrifices some compression ratio.
• LZ methods and ZIP are lossless.
• With lossy compression, “local decodability” can be easier in the sense that each block may be decoded without referencing data too far outside itself—but at the cost that you’re not reconstructing the original data exactly.

 

However, I think there is a group of people who over-optimize for Direction and neglect the Magnitude. Increasing Magnitude often comes with the risk of corrupting the Direction. For example, scaling fast often makes it difficult to hire only mission-aligned people, and it requires you to give voting power to investors that prioritizes profit. To increase Magnitude can therefore feel risky, what if I end up working at something that is net-negative for the world? Therefore it might be easier for one's personal sanity to optimize for Direction, to do something that is unquestionably net-positive. But this is the easy way out, and if you want to have the highest expected value of your Impact, you cannot disregard Magnitude.

You talk here about an impact/direction v ambition/profit tradeoff. I've heard many other people talking about this tradeoff too. I think it's overrated; in particular, if you're constantly having to think about it, that's a bad sign.

1. It's rare that you have a continuous space of options between lots of impact and low profit, and low/negative impact and high profit.
2. If you do have such a continuous space of options then I think you are often just screwed and profit incentives will win.
3. The really important decision you make is probably a discrete choice: do you start an org trying to do X, or an org trying to do Y? Usually you can't (and even if you can, shouldn't) try to interpolate between these things, and making this high-level strategy call will probably shape your impact more than any later finetuning of parameters within that strategy.
4. Often, the profit incentives point towards the more-obvious, gradient-descent-like path, which is usually very crowded and leads to many "mediocre" outcomes (e.g. starting a $10M company), but the biggest things come from doing "Something Else Which Is Not That" (as is said in dath ilan [LW(p) · GW(p)]). For example, SpaceX (ridiculously hard and untested business proposition) and Facebook (started out seeing very small and niche and with no clue of where the profit was).

Instead, I think the real value of doing things that are startup-like comes from:

1. The zero-to-one part of Peter Thiel's zero-to-one v one-to-n framework [LW · GW]: the hardest, progress-bottlenecking things usually look like creating new things, rather than scaling existing things. For example, there is very little you can do today in American politics that is as impactful or reaches as deep into the future as founding America in the first place.
2. In the case of AI safety: neglectedness. Everyone wants to work at a lab instead, humans are too risk averse in general, etc. (I've heard many people in AI safety say that neglectedness is overrated. There are arguments like this one that replaceability/neglectedness considerations aren't that major: job performance is heavy-tailed, hiring is hard for orgs, etc. But such arguments seem like weirdly myopic parameter-fiddling, at least when the alternative is zero-to-one things like discussed above. Starting big things is in fact big. Paradigm shifts matter because they're the frame that everything else takes place in. You either see this or you don't.)
3. To the extent you think the problem is about economic incentives or differential progress, have you considered getting your hands dirty and trying to change the actual economy or the direction of the tech tree? There are many ways to do this, including some types of policy and research. But I think the AI safety scene has a cultural bias towards things that look like research or information-gathering, and away from being "builders" in the Silicon Valley sense. One of the things that Silicon Valey does get right is that being a builder is very powerful. If the AI debate comes down to a culture/influence struggle between anti-steering, e/acc-influenced builder types and pro-steering EA-influenced academic types, it doesn't look good for the world.