LessWrong 2.0 Reader

Yep, that sounds sensible. I sometimes use consumer reports in my usual method for buying something in product class X. My usual is: 
1) Check what's recommended on forums/subreddits who care about the quality of X. 
2) Compare the rating distribution of an instance of X to other members of X. 
3) Check high quality reviews. This either requires finding someone you trust to do this, or looking at things like consumer reports. 
 

The LLM analog of in vino veritas

I expect any tests to show unambiguously that it's "not being replaced at all and citations[/mentions] chaotically swirling". If I understand Evans correctly, these were all random eminent figures he picked, not selected to be falling out of fashion - and they do seem to be a pretty broad sample of the "old prestigious standard names" space.

The stand mixer is a clever analogy; I didn't previously have experience with the separation thing.

I presume you've seen Is Clickbait Destroying Our General Intelligence?, and probably Hanson's cultural evolution / cultural drift frame. I wonder if you're familiar with Callard's Distant Signals paradigm [ transcript available on episode page ], which I think is the most illuminating of the three.

Besides just the cost of ~instantaneous ~omnicast communication dropping to ~zero, I see a role for the fall of the gold standard in all this. See e.g. U.S. per capita energy usage since ~1970, international fertility since ~1970. My theory [ which I really need to make a more legible graphic for ] is that when people don't "own their money" and have to track the effects of distant inflation-adjusts from the Fed, inflation volatility [ the destructive macroeconomic thing this OP on /r/badeconomics is saying couldn't possibly be happening due to the fall of the gold standard ] goes way up. Incentives in the market for ideas are ultimately material [yes, virtual status goods influence material wealth, but it also goes the other way around], so the market for materials influences the market for ideas, and vice versa, in a vicious spiral of decline. Is the theory.

Update: back up to 50% chance. 

Noting Microsoft’s cancelling of data center deals. And the fact the ‘AGI’ labs are still losing cash, and with DeepSeek are competing increasingly on a commodity product. 

Despite the simplicity prior, “this nontrivial program implements the behavior I want” almost always fails to be true because the program contains bugs. Even after you fix the bugs, it still almost always fails to be true because it contains even more bugs. 

False scientific theories don’t seem to fail in quite the same ways, and I’m not sure how much of that is due to differences in the search processes versus the search spaces (e.g. physics seems quite different from Python). 

Another work on fuzzing is Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internals by @joshc [LW · GW], @kh4dien [LW · GW], and Severin Field. Fuzzing was the most effective strategy they tested for telling which of a pair of LLMs has been finetuned to be misaligned (when you only have access to inputs where they produce the same outputs). 

A thing that might be worth trying: quantize the deceptive models down, and see what that does to their truthfulness.

Hypothesis: acting deceptively is a more complex behavior for an LLM than being truthful. Thus, anything that cripples an LLM's ability to act in complex ways is going to make them more truthful. Quantization would have that effect too.

That method might, then, lose power on more capable LLMs, or in case of deeper deceptive behaviors. Also if you want to check for deception in extremely complex tasks - LLM's ability to perform the task might fall off a cliff long before deception does.

I did the same during my last year of high school and first year in university! I even got a second hand trouser press so I could make sure the creases were sharp. 

However, I stopped it in the end. I think most observers thought I was either some sort of neoreactionary or just a plain weirdo. Almost nobody asked me why I was wearing it (this was Scandinavia, so pretty normal to not ask other people why they are doing weird things. We are a bunch of quiet, private people). At some point I realized I was using mental energy by the entire thing, being less able to relax in social situations.

Then, I changed to wearing a shirt + sweater + tweed jacket, with slightly longer hair and more stubble. This still gave me a certain style satisfaction, but in more muted way. And finally, by now, I always just dress to stand out as little as possible, since I have found that not thinking about my own appearance gives me more energy for interacting socially with others. Also, after having been in full dress to a couple of renaisance fairs, I started realizing that modern people are actually doing a lot of work to make sure they all wear period appropriate clothing, and it started to seem that me not doing so was like a reenactor putting a 14th century houbbelande on when everyone else was wearing 15th century clothing. 

The 200K token context window is a significant bottleneck.

Gemini Pro has a 2 million token context window, so I assume it would do significantly better. (I wonder why no other model has come close to the Gemini context window size. I have to assume not all algorithmic breakthroughs are replicated a few months later by other models.)

I agree that it can be possible to turn such a system into an agent. I think the original comment is defending a stronger claim that there's a sort of no free lunch theorem: either you don't act on the outputs of the oracle at all, or it's just as much of an agent as any other system.

I think the stronger claim is clearly not true. The worrying thing about powerful agents is that their outputs are selected to cause certain outcomes, even if you try to prevent those outcomes. So depending on the actions you're going to take in response to its outputs, its outputs have to be different. But the point of an oracle is to not have that property -- its outputs are decided by a criterion (something like truth) -- that is independent of the actions you're going to take in response^[1]. So if you respond differently to the outputs, they cause different outcomes. Assuming you've succeeded at building the oracle to specification, it's clearly not the case that the oracle has the worrying property of agents just because you act on its outputs.

I don't disagree that by either hooking the oracle up in a scaffolded feedback loop with the environment, or getting it to output plans, you could extract more agency from it. Of the two I think the scaffolding can in principle easily produce dangerous agency in the same way long-horizon RL can, but that the version where you get it to output a plan is much less worrrying (I can argue for that in a separate comment if you like).