LessWrong 2.0 Reader

This is very interesting, and thanks for sharing. 

• One thing that jumps out at me is they used a instruction format to prompt base models, which isn't typically the way to evaluate base models. It should be reformatted to a completion type of task. If this is redone, I wonder if the performance of the base model will also increase, and maybe that could isolate the effect further to just RLHF.
• I wonder if this has anything to do with also the number of datasets added on by RLHF (assuming a model go through supervised/instruction finetuning first, and then RLHF), besides the algorithm themselves.
• Another good model to test on is https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 which only has instruction finetuning it seems as well.

This makes some interesting predictions re: some types of trauma: namely, that they can happen when someone was (probably even correctly!) pushing very hard towards some important goal, and then either they ran out of fuel just before finishing and collapsed, or they achieved that goal and then - because of circumstances, just plain bad luck, or something else - that goal failed to pay off in the way that it usually does, societally speaking. In either case, the predictor/pusher that burned down lots of savings in investment doesn't get paid off. This is maybe part of why "if trauma, and help, you stronger; if trauma, and no help, you get weaker".

No one has picked up the true origin of OpenAI yet. If you dig in to it, you will see some revealing declarations and emails. The whole idea for a non-profit open AI organization and the commitment to share the benefits with humanity by the name of Open AI came through theft. https://archive.ph/1wEOX  

I got the bidding idea from Kaj, and “if the mind is a group” is my preferred metaphor/simplification of multi-agent models of mind (writing about this soon). This metaphor naturally implies reputation, as I realized yesterday while working with a client. I don't know if there’s a name for the reputation idea; it may be original

"If a model trained on synthetic data is expected to have good performance out of distribution (on real-world problems) then I think that it would also be expected to have high performance at assessing whether it's in a simulation."

Noosphere89, you have marked this sentence with a "disagree" emoji. Would you mind expanding on that? I think it is a pretty important point and I'd love to see why you disagree with Ben.

Ah, yep, this makes sense to me.

Yeah, this is a good point, which doesn't seem addressed by any idea so far.

Chess is like a bounded, mathematically described universe where all the instrumental convergence stays contained, and only accomplishes a very limited instrumentality in our universe (IE chess programs gain a limited sort of power here by being good playmates).

LLMs touch on the real world far more than that, such that MCTS-like skill at navigating "the LLM world" in contrast to chess sounds to me like it may create a concerning level of real-world-relevant instrumental convergence.

OP did the work to collect these emails and put them into a post. When people do work for you, you shouldn't punish them by giving them even more work.

Oh wow, this is almost exactly how I model my internal mind. I didn't realize it was a real thing other people has arrived at. Is there a name for this?