LessWrong 2.0 Reader

Seems like a rational prioritization to me if they were in an important moment of thought and didn't want to disrupt it. (Noting of course that 'walking on it' was not intentional and was caused by forgetting it was there.)

This sounds like you're saying that they made a rational prioritization and then, separately from that, forgot that it was there. But those two events are not separate: the forgetting-and-then-walking-on-it was a predictable consequence of the earlier decision to ignore it and instead focus on work. I think if you model the first decision as a decision to continue working and to also take on a significant risk of hurting your feet, it doesn't seem so obviously rational anymore. (Of course it could be that the thought in question was just so important that it was worth the risk. But that seems unlikely to me.)

As the OP says, a "normal person might stop and remove all the glass splinters". Most people, in thinking whether to continue working or whether to clean up the splinters, wouldn't need to explicitly consider the possibility that they might forget about the splinters and step on them later. This would be incorporated into the decision-making process implicitly and automatically, by the presence of splinters making them feel uneasy until they were cleaned up. The fact that this didn't happen suggests that the OP might also ignore other signals relevant to their well-being. 

The fact that the OP seems to consider this event a virtue to highlight in the title of their post, is also a sign that they are systematically undervaluing their own well-being in a way that to me seems very worrying.

Also, I would feel pretty bad if someone wrote a comment like this after I posted something. (Maybe it would have been better as a PM)

Probably most people would. But I think it's also really important for there to be clear, public signals that the community wants people to take their well-being seriously and doesn't endorse people hurting themselves "for the sake of the cause". 

The EA and rationalist communities are infamous for having lots of people burning themselves out through extreme self-sacrifice. If someone makes a post where they present the act of working until their feet start bleeding as a personal virtue, and there's no public pushback to that, then that sends the implicit signal that the community endorses that reasoning. That will then contribute to unhealthy social norms that cause people to burn themselves out. The only way to counteract that is by public comments that make it clear that the community wants people to take care of themselves, even if that makes them (temporarily) less effective.

To the OP: please prioritize your well-being first. Self-preservation is one of the instrumental convergent drives [? · GW]; you can only continue to work if you are in good shape.

Thanks for the insightful comment!

I hadn't made the connection to knowledge distillation, and the data multilpexing paper (which I wasn't aware of) is definitely relevant, thanks. I agree that our results seem very odd in this light.

It is certainly big news if OA fine-tuning doesn't work as it's supposed to. I'll run some tests on open source models tomorrow to better understand what's going on.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?

Oh, that seems easy enough. People might think that they are safe as long as they don't write as much as I or Scott do under a few names, but that's not true. If you have any writing samples at all, you just stick the list of them into a prompt and ask about similarity. Even if you have a lot of writing, context windows are now millions of tokens long, so you can stick an entire book (or three) of writing into a context window.

And remember, the longer the context window, the more that the 'prompt' is simply an inefficient form of pretraining, where you create the hidden state of an RNN for millions of timesteps, meta-learning the new task, and then throw it away. (Although note even there that Google has a new 'caching' feature which lets you run the same prompt multiple times, essentially reinventing caching RNN hidden states.) So when you stick corpuses into a long prompt, you are essentially pretraining the LLM some more, and making it as capable of identifying a new author as it is capable of already identifying 'gwern' or 'Scott Alexander'.

So, you would simply do something like put in a list of (author, sample) as well as any additional metadata convenient like biographies, then 'unknown sample', and ask, 'rank the authors by how likely they are to have written that final sample by an unknown author'.

This depends on having a short list of authors which can fit in the prompt (the shorter the samples, the more you can fit, but the worse the prediction), but it's not hard to imagine how to generalize this to an entire list. You can think of it as a noisy sorting problem or a best-arm finding problem. Just break up your entir e list of n authors into groups of m, and start running the identification prompt, which will not cost n log n prompts because you're not sorting the entire list, you are only finding the min/max (which is roughly linear). For many purposes, it would be acceptable to pay a few dozen dollars to dox an author out of a list of a few thousand candidates.

djb admonishes us to always to remember to ask about amortized or economies of scales in attacks, and that's true too here of course in stylometric attacks. If we simply do the obvious lazy sort, we are throwing away all of the useful similarity information that the LLM could be giving us. We could instead work on embedding authors by similarity using comparisons. We could, say, input 3 authors at a time, and ask "is author #1 more similar to #2, or #3?" Handwaving the details, you can then take a large set of similarity rankings, and infer an embedding which maximizes the distance between each author while still obeying the constraints. (Using expectation maximization or maybe an integer solver, idk.) Now you can efficiently look up any new author as a sort of nearest-neighbors lookup problem by running a relatively few comparison prompts and homing in on the set of author-points a new author is nearest, and use that small set for a final direct question.

(All this assumes you are trying to leverage a SOTA LLM which isn't directly accessible. If you use an off-the-shelf LLM like a LLaMA-3, you would probably do something more direct like train a triplet loss on the frozen LLM using large text corpuses and get embeddings directly, making k-NN lookups effectively free & instantaneous. In conclusion, text anonymity will soon be as dead as face anonymity.)

Here are two arguments for low-hanging algorithmic improvements.

First, in the past few years I have read many papers containing low-hanging algorithmic improvements.  Most such improvements are a few percent or tens of percent.  The largest such improvements are things like transformers or mixture of experts, which are substantial steps forward.  Such a trend is not guaranteed to persist, but that’s the way to bet.

Second, existing models are far less sample-efficient than humans.  We receive about a billion tokens growing to adulthood.  The leading LLMs get orders of magnitude more than that.  We should be able to do much better.  Of course, there’s no guarantee that such an improvement is “low hanging”.  

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?

Yesterday Sam Altman stated (perhaps in response to the Vox article that mentions your decision) that "the team was already in the process of fixing the standard exit paperwork over the past month or so. if any former employee who signed one of those old agreements is worried about it, they can contact me and we'll fix that too."

I notice he did not include you in the list of people who can contact him to "fix that", but it seems worth a try, and you can report what happens either way.

The diversity of opinion in the AI existential safety community is at least as big (and is probably even larger, which is natural given that the field is much younger, with its progress being much less certain), but, in addition to that, the diversity is less obfuscated, because it does not have anything resembling the Transformer-based LLM highly successful center around which people can consolidate.

I still don't see how you can validate any alignment ideas without having a lot of compute to test them out on precursor models. Or how you can validate them without training misaligned models that won't be released to the public. That's why I have almost 0 faith in any of these smaller players. Maybe it's good for small players to publish research so big labs can try to repro them. In that case, I still don't see why you would leave your lab to do that. Your idea is much more likely to be reproduced if you influence training runs at a lab. I take this as part of why for example Neel Nanda joined an AI lab.

But is it actually that unquestionable? Even with Microsoft backing OpenAI, Google should have always been ahead of OpenAI, if it were just a matter of raw compute.

Since hardware jumps are exponential, even if they don't do "yolo-runs" like OpenAI does where they dedicate a large portion of their compute to risky ideas, just wait a few years until their GPUs/TPUs get better and a gpt-4 sized model is much cheaper to train. I expect Google to race ahead of OpenAI sooner or later, at least until Stargate is finished.  Arguably their demos have been more technically impressive, even if OpenAI's demos are shinier looking. 
 

I think that non-standard architectural and algorithmic breakthroughs can easily make smaller players competitive, especially as inertia of adherence to "what has been proven before" will inhibit the largest players.

Do these exist? My model is that most ideas are just come up with experimentally validating papers and hypothesis through trial-and-error (i.e. compute), and OpenAI wasn't a small player in terms of compute. They also used standard architectures and algorithms. They just took more risks than Google. But a risky "yolo run" now is a "standard training run" in a few years.

Even in one or two years, we'll find that the models can do a lot more involved tasks than they can do now. For example, you could imagine having the models carry out a whole coding project instead of it giving you one suggestion on how to write a function. You could imagine the model taking high-level instructions on what to code and going out on its own, writing any files, and testing it, and looking at the output. It might even iterate on that a bit. So just much more complex tasks.

OK, so we are likely to have that (I don't think he is over-optimistic here), and the models are already very capable of discussing AI research papers and exhibit good comprehension of those papers (that's one of my main use cases for LLMs: to help me understand an AI research paper better and faster). And they will get better at that as well.

This really does not sound like AGI to me (or at least highly depends on what a coding project means here) and prediction+stock markets don't buy that this will be AGI either.  This just sounds like a marginally better GPT-4. I'd expect AGI to be as transformative as people expect AGI to be when it can automate hardware research.

This seems a bit odd given past literature on LLMs. As I've noted before, you can do inner-monologue problems specifically via knowledge-distillation somewhat analogous to your finetuning, and it's also possible to ask models to solve multiple problems simultaneously analogous to your base task (or do various kinds of speculative or parallelized decoding at a lower level). There is enormous computational waste and slack, and capacity to spare for multiple problems. So it not working for the OA "finetuning" of GPT-3.5 is unexpected: I can't think of any previous results aimed at making forward passes do more which failed completely (although ofc maybe they just don't get reported or I didn't happen to read them etc).

I notice this is not the first time I've left a puzzled comment on a post where the authors failed to make GPT-3.5 do something via OA "finetuning" that it seemed like it definitely should have been capable of after finetuning or which non-OA models did do... And the common ingredient seems like the OA "finetuning".

I'm not aware of any experiments by third parties demonstrating that OA "finetuning" works like it's supposed to. Maybe someone should do that before more people try to do AI safety research predicated on the assumption that using OA's "finetuning" is telling you anything meaningful about LLMs in general, rather than being like, say, trying to understand LLM poetry by looking at ChatGPT's rhymes or LLM linguistic knowledge by asking one to spell words.

The other common optimization process people generally refer to is "markets".  In the same sense that "evolution" is what happens when variation and selection combine, "market" is what happens when multiple traders repeatedly choose how to exchange things.