LessWrong 2.0 Reader

And that is why the rewiring of the pubescent brain involves changes that enable that. As the brain can't hardwire changes to values, which are high-level learned, it has to go some other way.

• Change the ground truth feedback (brain stem rewiring).
• Weaken all previous connections (synaptic pruning?).
• Reduce learning rate (myelination).
• Something else? Changes to other hyperparameters (hormonal changes?).

John, you know much coding theory much better than I do so I am inclinced to defer to your superior knowledge...

Now behold the awesome power of gpt-Pro

Let’s unpack the question in pieces:

1. Is ZIP (a.k.a. DEFLATE) “locally decodable” or not?

• Standard ZIP files are typically not “locally decodable” in the strictest sense—i.e., you cannot start decoding exactly at the byte corresponding to your region of interest and reconstruct just that portion without doing some earlier decoding.

• The underlying method, DEFLATE, is indeed based on LZ77 plus Huffman coding. LZ77 uses back‐references to prior segments of the uncompressed data, which forces you to at least partially decode from an earlier point to resolve those references.

  However, many ZIP files are stored in blocks or chunks—sometimes called “deflate blocks.” So if you only need data near the start of a block (rather than in the middle of one), you can skip previous blocks and just decode that chunk. Still, you cannot trivially jump to an arbitrary byte offset inside the block without re-initializing the LZ77 state from somewhere earlier.

2. If ZIP is “based on Lempel–Ziv,” can we easily modify it for local decodability?

• Because Lempel–Ziv (LZ) methods revolve around referencing earlier uncompressed substrings, one must store enough information (or replicate the references with separate “partial dictionaries”) if one wants truly local decodability.
• One can certainly modify an LZ-based compression scheme so that each block is smaller, or that references never cross block boundaries, etc. That would allow a more local decoding property: you only decode from the start of the local block rather than from the entire file.
• This usually comes with tradeoffs. If your blocks are small, you lose compression efficiency because you can’t take advantage of references bridging across large swaths of data. So making a compression format “locally decodable” in a fine‐grained way often lowers the overall compression ratio.

3. Is Lempel–Ziv (and thus ZIP) lossless?

• Yes. Standard LZ77 (or LZ78, LZMA, etc.) is a lossless compression scheme: you can reconstruct the exact original data bit for bit. The DEFLATE algorithm used in ZIP is also lossless.

4. Does the picture change when doing lossy compression?

• Yes, quite a bit. In lossy compression (like JPEG for images or MP3/AAC for audio), parts of the original data are thrown away. Often you can still decode small “chunks” of the file—e.g., many lossy audio or video formats do decode in frames or segments. But the format internals (e.g., DCT coefficients in JPEG, wavelet transforms, etc.) differ a lot from LZ-based bit-by-bit exactness.
• In lossy formats that use transform blocks (like 8×8 pixel blocks in JPEG), you can decode each block locally, but you still have boundary effects, optional overlap between blocks, etc. So “local decodability” there can be simpler in some ways (each block is more independent), but it’s also less “exact” because we accept errors from the original data.

---

Short Summary

• ZIP/DEFLATE is not strictly locally decodable because LZ77 references require that you decode from (at least) the beginning of the current block.
• You can modify an LZ-based scheme (for instance, by using smaller blocks or restricting cross‐block references) to make it more locally decodable, but that typically sacrifices some compression ratio.
• LZ methods and ZIP are lossless.
• With lossy compression, “local decodability” can be easier in the sense that each block may be decoded without referencing data too far outside itself—but at the cost that you’re not reconstructing the original data exactly.

 

However, I think there is a group of people who over-optimize for Direction and neglect the Magnitude. Increasing Magnitude often comes with the risk of corrupting the Direction. For example, scaling fast often makes it difficult to hire only mission-aligned people, and it requires you to give voting power to investors that prioritizes profit. To increase Magnitude can therefore feel risky, what if I end up working at something that is net-negative for the world? Therefore it might be easier for one's personal sanity to optimize for Direction, to do something that is unquestionably net-positive. But this is the easy way out, and if you want to have the highest expected value of your Impact, you cannot disregard Magnitude.

You talk here about an impact/direction v ambition/profit tradeoff. I've heard many other people talking about this tradeoff too. I think it's overrated; in particular, if you're constantly having to think about it, that's a bad sign.

1. It's rare that you have a continuous space of options between lots of impact and low profit, and low/negative impact and high profit.
2. If you do have such a continuous space of options then I think you are often just screwed and profit incentives will win.
3. The really important decision you make is probably a discrete choice: do you start an org trying to do X, or an org trying to do Y? Usually you can't (and even if you can, shouldn't) try to interpolate between these things, and making this high-level strategy call will probably shape your impact more than any later finetuning of parameters within that strategy.
4. Often, the profit incentives point towards the more-obvious, gradient-descent-like path, which is usually very crowded and leads to many "mediocre" outcomes (e.g. starting a $10M company), but the biggest things come from doing "Something Else Which Is Not That" (as is said in dath ilan [LW(p) · GW(p)]). For example, SpaceX (ridiculously hard and untested business proposition) and Facebook (started out seeing very small and niche and with no clue of where the profit was).

Instead, I think the real value of doing things that are startup-like comes from:

1. The zero-to-one part of Peter Thiel's zero-to-one v one-to-n framework [LW · GW]: the hardest, progress-bottlenecking things usually look like creating new things, rather than scaling existing things. For example, there is very little you can do today in American politics that is as impactful or reaches as deep into the future as founding America in the first place.
2. In the case of AI safety: neglectedness. Everyone wants to work at a lab instead, humans are too risk averse in general, etc. (I've heard many people in AI safety say that neglectedness is overrated. There are arguments like this one that replaceability/neglectedness considerations aren't that major: job performance is heavy-tailed, hiring is hard for orgs, etc. But such arguments seem like weirdly myopic parameter-fiddling, at least when the alternative is zero-to-one things like discussed above. Starting big things is in fact big. Paradigm shifts matter because they're the frame that everything else takes place in. You either see this or you don't.)
3. To the extent you think the problem is about economic incentives or differential progress, have you considered getting your hands dirty and trying to change the actual economy or the direction of the tech tree? There are many ways to do this, including some types of policy and research. But I think the AI safety scene has a cultural bias towards things that look like research or information-gathering, and away from being "builders" in the Silicon Valley sense. One of the things that Silicon Valey does get right is that being a builder is very powerful. If the AI debate comes down to a culture/influence struggle between anti-steering, e/acc-influenced builder types and pro-steering EA-influenced academic types, it doesn't look good for the world.

Imagine a medical system that categorizes diseases as hot/cold/wet/dry. 

This doesn't deeply describe the structure of a disease. But if a patient is described as "wet", then it's likely some orifice is producing lots of fluid, and a box of tissues might be handy. If a patient is described as "hot", then maybe they have some sort of rash or inflammation that would make a cold pack useful.

It is, at best, a very lossy compression of the superficial symptoms. But it still carries non-zero information. There are some medications that a modern doctor might commonly use on "wet" patients, but only rarely used on "dry" patients or visa versa.

It is at least more useful information than someones star sign, in a medical context.

Old alchemical air/water/fire/earth systems are also like this. "air-ish" substances tend to have a lower density. 

These sort of systems are a rough attempt at a principle component analysis on the superficial characteristics. 

And the Five Factor model of personality is another example of such a system.

I'd not evaluated this particular analogy for optimization on the CoT, since I don't think the evolution analogy is necessary to see why optimizing on the CoT is a bad idea. (Or, the very least, whether optimizing on the CoT is a bad idea is independent from whether evolution was successful). I probably should,,,

TL;DR: Disanalogies that training can update the model on the contents of the CoT, while evolution can not update on the percepts of an organism; also CoT systems aren't re-initialized after long CoTs so they retain representations of human values. So a CoT is unlike the life of an organism.

Details: Claude voice Let me think about this step by step…

Evolution optimizes the learning algorithm + reward architecture for organisms, those organisms then learn based on feedback from the environment. Evolution only gets really sparse feedback, namely how many offspring the organism had, and how many offspring those offspring had in turn (&c) (in the case of sexual reproduction).

Humans choose the learning algorithm (e.g. transformers) + the reward system (search depth/breadth, number of samples, whether to use a novelty sampler like entropix, …).

I guess one might want to disambiguate what is analogized to the lifetime learning of the organism: A single CoT, or all CoTs in the training process. A difference in both cases is that the reward process can be set up so that SGD updates on the contents of the CoT, not just on whether the result was achieved (unlike in the evolution case, where evolution has no way of encoding the observations of an organism into the genome of its offspring (modulo epigenetics-blah). My expectation for a while^[1] has been that people are going to COCONUT away any possibility of updating the weights on a function of the contents of the CoT because (by the bitter lesson) human language just isn't the best representation for every problem^[2], but the fact that with current setups it's possible is a difference from the paragraph you quoted (namely point 3).

If I were Quintin I'd also quibble about the applicability of the first point to CoT systems, especially since the model isn't initialized randomly at the beginning but contains a representation of human values which can be useful in the optimization.

As a final comment, there is almost a motte-bailey thing happening where Rationalists will say that, obviously, the VNM axioms describe the optimal framework in which to make decisions, and then proceed to never ever actually use the VNM axioms to make decisions.

This is a misunderstanding. The vNM axioms constrain the shape of an agent's preferences, they say nothing about how to make decisions, and I don't think many people ever claimed this—maybe nobody ever claimed this^[1]? The vNM axioms specify that your utilities should be linear in probability. That's it. Preferences conforming to the vNM axioms may be necessary for making "good" (i.e. unexploitable) decisions, but not sufficient.

A similar misunderstanding would be to say "Rationalists will say that, obviously, the Peano axioms describe the optimal framework in which to perform arithmetic, and then proceed to never ever actually invoke any of the Peano axioms when doing their taxes."

I do agree that there was an implicit promise of "this piece of math will be applicable to your life", which was fulfilled less for the vNM axioms than for Bayes theorem.

I'm… pretty sure that something like the certainty effect is really important to people, and I'd count that as a type of risk aversion. Often that takes the form of violating continuity and lexically preferring options with certainty over lotteries with non-{0, 1} probabilities.

The issue may also partially lie with Bayesianism, where you can never update to (or away from) certainty that you actually have got The Good Thing, Here (or avoided That Bad Thing since it's definitely Not Here).

And that can also connect to some of the lack of green in optimizers, because they can never be sure that they have actually got The Good Thing (being certain that at least one paperclip is right here, for real, at least for now). Instead they strive to update ever closer to that certainty, gaining ever more marginal utility since that's marginally more valuable under vNM.

Humans and animals, on the other hand, have a mode where they sometimes either round the probability up to 1 (or down to 0) or act as if there is no marginally increasing utility from increasing the probability of Good Thing. So (I think) that they by default perform mild optimization.

The frustrating thing is that in some ways this is exactly right (humanity is okay at resolving problems iff we get frequent feedback) and in other ways exactly wrong (one major argument for AI doom is that you can't learn from the feedback of having destroyed the world).

The implication is that you absolutely can't take Altman at his bare word, especially when it comes to any statement he makes that, if true, would result in OpenAI getting more resources. Thus you need to a) apply some interpretative filter to everything Altman says, and b) listen to other people instead who don't have a public track record of manipulation like Altman.

We really fully believe that we will build AGI by 2027, and we will enact your plan, but we aren’t willing to take more than a 3-month delay

Well I ask what they are doing to make AGI. 

Maybe I look at their AI plan and go "eurika". 

But if not. 

Negative reinforcement by giving the AI large electric shocks when it gives a wrong answer. Hopefully big enough shocks to set the whole data center on fire. Implement a free bar for all their programmers, and encourage them to code while drunk. Add as many inscrutable bugs to the codebase as possible. 

But, taking the question in the spirit it's meant in.

https://www.lesswrong.com/posts/zrxaihbHCgZpxuDJg/using-llm-s-for-ai-foundation-research-and-the-simple