LessWrong 2.0 Reader

"bottleneck" of the CoT tokens.  Whatever needs to be passed along from one sequential calculation step to the next must go through this bottleneck.

Does it, though? The keys and values from previous forward passes are still accessible, even if the generated token is not.

So the CoT tokens are not absolute information bottlenecks. But yes, replacing the token by a dot reduces the number of serial steps the model can perform (from mn to m+n, if there are m forward passes and n layers).

Fwiw I basically think you are right about the agentic AI overhang and obviously so. I do think it shapes how one thinks about what's most valuable in AI alignment.

How sure are you that OKcupid is a significantly better product for the majority of people (as opposed to a niche group of very online people)?

oh, sorry, I thought slatestar codex wrote something about it and you were saying that's where it comes from

Wouldn't automatically abolish all requirements; maybe I'm not good enough in searching but to the degree I'm not an outlier:

• With internet we have reviews, but they're not always trustworthy, and even if they are, understanding/checking/searching reviews is costly, sometimes very costly.
• There is value in being able to walk up to the next-best random store for a random thing and being served by a person with a minimum standard of education in the trade. Even for rather trivial things.

This seems underappreciated here.

Flower safety isn't a thing. But having the next-best florist to for sure be a serious florist person to talk to, has serious value. So, I'm not even sure for something like flowers I'm entirely against any sort of requirements.

So it seems to me more a question of balance what exactly to require in which trade, and that's a tricky one, but in some places I lived seems to have been handled mostly +- okay. Admittedly simply according to my shallow glance at things.

Lived also in countries that seem more permissive, requiring less job training, but clearly prefer the customer experience in those that regulate, despite higher prices.

Then, I wouldn't want the untrained/unexamined florist to starve or even simply become impoverished. But at least in some countries, social safety net mostly prevents that.

I was brainstorming sabotage-related threats with friends:

• Sandbag on a few safety research tasks, especially techniques to catch scheming and stuff that's high-level or has poor feedback loops
  • Plausible; maybe not so bad
• Sandbag on ~all safety research tasks
  • Easy to catch; catching obvious sandbagging has safety benefits + maybe we can posttrain it away
• Sandbag/lie when validating that a model/deployment/whatever is safe
  • idk
• When automating AI R&D, backdoor the model
  • (this is takeover-attempt-y, not sandbagging-y)
  • idk
• When deployed to automate/improve security (especially for the model's own weights): both sandbagging and code-backdooring/hacking/etc.
  • idk
• Sandbagging on AI R&D to help another lab win the race (!?)
  • Implausible
• ??

Are people quite worried about safety research sabotage? Why? Do they think a (say) 20% hit to automated safety research is a huge deal, or do they expect a larger hit?

I happened to read a Quanta article about equivalence earlier, and one of the threads is the difficulty of a field applying a big new concept without the expository and distillation work of putting stuff into textbooks/lectures/etc.

That problem pattern-matches with the replication example, but well-motivated at the front end instead of badly-motivated at the back end. It still feels like exposition and distillation are key tasks that govern the memes-in-the-field passed among median researchers.

I strongly suspect the crux of the replication crisis example is that while there are piles of exposition and distillation for probability/statistics, they are from outside whatever field experiences the problem, and each field needs its own internal completion of these steps for them to stick.

I think this should be a third specialization for every scientific field. We have theorists, we have experimentalists, and to this we should add analysts. Their work would specialize in establishing the statistical properties of experimental designs and equipment in the field on the one side, and the statistical requirements to advance various theories on the other side.

Oh. I linked the wrong thing. I would down vote this too. Sorry about setting an expectation and then not fulfilling it.

Edit:  I fixed the link at the end of the post.

It sucks that I have to wait a week before posting anything again though because I made a simple mistake. I guess I'll just have to hope I don't mess up again in the future.

Claim: memeticity in a scientific field is mostly determined, not by the most competent researchers in the field, but instead by roughly-median researchers. [...] Sure, the most competent people in the field may recognize the problems, but the median researchers don’t, and in aggregate it’s mostly the median researchers who spread the memes.

This assumes the median researchers can't recognize who the competent researchers are, or otherwise don't look to them as thought leaders.

I'm not arguing that this isn't often the case, just that it isn't always the case. In engineering, if you're more competent than everyone else, you can make cooler shit. If you're a median engineer trying to figure out which memes to take on and spread, you're going to be drawn to the work of the more competent engineers because it is visibly and obviously better.

In fields where distinguishing between bad research and good research has to be done by knowing how to do good research, rather than "does it fly or does it crash", then the problem you describe is much more difficult to avoid. I argue that the difference between the fields which replicate and those which don't is as much about the legibility of the end product as it is about the quality of the median researcher.

A few more thoughts:

It's plausible that for both humans and orcas the relevant selection pressure mostly came from social dynamics, and it's plausible that there were different environmental pressures.

Actually my guess would be that it's because intelligence was environmentally adaptive [LW(p) · GW(p)], because my intuitive guess would be that group selection^[1] is significant enough over long timescales which would disincentivize intelligence if it's not already (almost) useful enough to warrant the metabolic cost, unless the species has a lot of slack.

So an important question is: How adaptive is high intelligence?

In general I would expect that selection pressure for intelligence was significantly stronger in humans, but maybe for orcas it was happening over a lot longer time window, so the result for orcas could still be more impressive.

From what I observed about orca behavior I'd perhaps say a lower bound of their intelligence might roughly be like human 15 year olds or so. So up to that level of intelligence there seem to be benefits that allow orcas to use more sophisticated hunting techniques.

But would it be useful for orcas to be significantly smarter than humans? My prior intuition would've been that probably not very much.

But I think observing the impressive orca brains mostly screens this off: I wouldn't have expected orcas to evolve to be that smart, and I similarly strongly wouldn't have expected them to have that impressive brains, and seeing their brains updates me that there had to be some selection pressure to produce that.

But the selection pressure for intelligence wouldn't have needed to be that strong compared to humans for making the added intelligence worth the metabolic cost, because orcas are large and their neurons make up a much smaller share of their overall metabolic consumption.

And that there is selection pressure is not totally implausible in hindsight:

• Orcas hunt very collaboratively, and maybe there are added benefits from coordinating their attacks better. (Btw, orcas live in matrilines, and I'd guess that from an evolutionary perspective the key thing to look at is how well a matriline performs, not individuals, but not sure. So there would be high selection for within-matriline cooperation (and perhaps communication!).)
• Some/(many?) Orca sub-species prey on other smart animals like dolphins or whales, and maybe orcas needed to be significantly smarter to be able to outwit the defensive mechanisms they learn to adapt.

But overall I know way too little about orca hunting techniques to be able to evaluate those.

1. ^{^}

   I mean group selection that could potentially be on a level of species where species go extinct. Please lmk if that's actually called differently.