LessWrong 2.0 Reader

I suggest making anonymity compulsary

It’s an interesting idea, but the track records of the grantees are important information, right? And if the track record includes, say, a previous paper that the funder has already read, then you can’t submit the paper with author names redacted.

Also, ask people seeking funding to make specific, unambiguous, easily falsiable predictions of positive outcomes from their work. And track and follow up on this!

Wouldn’t it be better for the funder to just say “if I’m going to fund Group X for Y months / years of work, I should see what X actually accomplished in the last Y months / years, and assume it will be vaguely similar”? And if Group X has no comparable past experience, then fine, but that equally means that you have no basis for believing their predictions right now.

Also, what if someone predicts that they’ll do A, but then realizes it would be better if they did B? Two possibilities are: (1) You the funder trust their judgment. Then you shouldn’t be putting even minor mental barriers in the way of their pivoting. Pivoting is hard and very good and important! (2) You the funder don’t particular trust the recipient’s judgment, you were only funding it because you wanted that specific deliverable. But then the normal procedure is that the funder and recipient work together to determine the deliverables that the funder wants and that the recipient is able to provide. Like, if I’m funding someone to build a database of AI safety papers, then I wouldn’t ask them to “make falsifiable predictions about the outcomes from their work”, instead I would negotiate a contract with them that says they’re gonna build the database. Right? I mean, I guess you could call that a falsifiable prediction, of sorts, but it’s a funny way to talk about it.

I love this quick takes thing:

• Low-key a better version of twitter
• I can blurb because I don't have the attention span to form thoughts long enough for a post
• Maybe one day I will collect enough thoughts here to get a LLM to slop out a post

I figured out how to not forget my keys when going out by sticking a bright post-it to the door that says "keys?"

All it took was realising that I am not infact "built different" and forget my keys all the time. This week was high-stakes because flatmate is out, and wouldn't be able to bail me out of being stuck outside.

Whether this sticks (ha!) or I start to ignore the post-it remains to be seen.

"QC: the water usage argument against LLMs is extremely bad faith..."

Rule of thumb, if you're reading some article and the water usage is measured in gallons, it's bullsh*t. This isn't even that, ml, seriously?! My very normal suburban house water bill doesn't even measure in gallons, it measures in hundreds of cubic feet (at about $5/per).  People do not realize how vast the quantities of water humans regularly divert and move around (and how very little of it is required to directly sustain human life).

Absolutely. In adversarial setting (XZ backdoor) there's no point in relaxing accountability.

Well, but you don't necessarily know if a setting is adversarial, right? And a process that starts by assuming everyone had good intentions probably isn't the most reliable way to find out.

it's exactly the case when one would expect blameless postmortems to work: The employer and the employee are aligned - neither of them wants the plane to crash and so the assumption of no ill intent should hold.

Not necessarily fully aligned, since e.g. the captain might benefit from getting to bully his first officer, or from coming to work drunk every day, or might have theatre tickets soon after the scheduled landing time, or.... Obviously he doesn't want to crash, but he might have different risk tolerances than his employer.

(Back of the envelope: google says United Airlines employs ~10,000 pilots. If a pilot has a 50-year career, then United has 200 pilot careers every year. Since 2010, they've had about 3000 careers and 9 accidents, no fatalities. That's about 0.3% chance of accident per career. An individual pilot might well want to make choices that have a higher-than-that chance of accident over their career.)

Something was swept under the carpet here and that may have been avoided if the investigation was careful not to assign blame.

Stipulated, but... suppose that the investigators had run a blameless process, the captain had told the truth ("I disabled the EGPWS because it would have been distracting while I deliberately violated a bunch of regulations") with no risk of getting fired, and they found some rot in the culture there. Can they do anything about it? It seems like if the answer is "yes", and if anyone benefits from the current culture, then we're back at people having incentives to lie.

Formatting is still kind of bad, and is affecting readability. It's been a couple of posts in a row now with long wall of text paragraphs. I feel like you changed something? And you should change it back. :)

I interpreted Habryka's comment as making two points, one of which strikes me as true and important (that it seems hard/unlikely for this approach to allow for pivoting adequately, should that be needed), and the other of which was a misunderstanding (that they don't literally say they hope to pivot if needed).

But success for most things doesn't require just one correct solution among k attempts, right? For the majority of areas without easily checkable solutions, higher odds of getting it right on the first try or fres tries is both very useful and does seem like evidence of reasoning. Right? Or am I missing something?

Reducing the breadth of search is a substantial downside if it's a large effect. But reliably getting the right answer instead of following weird paths most of which are wrong seems like the essence of good reasoning.

We believe that, even without further breakthroughs, this work can almost entirely mitigate the risk that we unwittingly put misaligned circa-human-expert-level agents in a position where they can cause severe harm.

Sam, I'm confused where this degree of confidence is coming from? I found this post helpful for understanding Anthropic's strategy, but there wasn't much argument given about why one should expect the strategy to work, much less to "almost entirely" mitigate the risk!

To me, this seems wildly overconfident given the quality of the available evidence—which, as Aysja notes [LW(p) · GW(p)], involves auditing techniques like e.g. simply asking the models themselves to rate their evil-ness on a scale from 1-10! I can kind of understand evidence like this informing your background intuitions and choice of research bets and so forth, but why do you think it justifies this much confidence that you'll catch/fix misalignment?

Notably, he was wrong about that.