LessWrong 2.0 Reader

Back in May, when the Crowdstrike bug happened, people were posting wild takes on Twitter and in my signal groupchats about how Crowdstrike is only used everywhere because the government regulators subject you to copious extra red tape if you try to switch to something else.

Here’s the original claim:

Microsoft blamed a 2009 antitrust agreement with the European Union that they said forced them to sustain low-level kernel access to third-party developers.[286][287][288] The document does not explicitly state that Microsoft has to provide kernel-level access, but says Microsoft must provide access to the same APIs used by its own security products.[287]

This seems consistent with your understanding of regulatory practices (“they do not give a rats ass what particular software vendor you use for anything”), and is consistent with the EU’s antitrust regulations being at fault—or at least Microsoft’s cautious interpretation of the regulations, which indeed is the approach you want to take here.

(very naive take) I would suspect this is medium-easily automatable by making detailed enough specs of existing hardware systems & bugs in them, or whatever (maybe synthetically generate weak systems with semi-obvious bugs and train on transcripts which allows generalization to harder ones). it also seems like the sort of thing that is particularly susceptible to AI >> human; the difficulty here is generating the appropriate data & the languages for doing so already exist ?

What about just not pursuing a PhD and instead doing what OP did? With the PhD you potentially lose #1 in 

I actually think that you can get great results doing research as a hobby because

• it gives you loads of slack, which is freedom to do things without constraints. In this context, I think slack is valuable because it allows you to research things outside of the publishing mainstream.
• and less pressure.

I think these two things are crucial for success. The slack allows you to look at risky and niche ideas are more likely to yield better research rewards if they are true, since surprising results will trigger further questions.

Also, since you are more likely to do better at topics you enjoy, getting money from a day job allows you to actually purse your interests or deviate from your supervisor’s wishes. Conversely, it also allows you to give up when you’re not enjoying something.

which is where much of the impact comes from, especially if you subscribe to a multiplicative view of impact [EA · GW].

First of all, macroscopical indistinguishability is not fundamental physical property - branching indifference is additional assumption, so I don't see how it's not as arbitrary as branch counting.

You're right it's not a fundamental physical property - the overall philosophical framework here is that things can be real - as emergent entities - without being fundamental physical properties. Things like lions, and chairs are other examples.

But more importantly, branching indifference assumption is not the same as informal "not caring about macroscopically indistinguishable differences"! 

This is how Wallace defines it (he in turn defines macroscopically indistinguishable in terms of providing the same rewards). It's his term in the axiomatic system he uses to get decision theory to work. There's not much to argue about here? 

As Wallace showed, branching indifference implies the Born rule implies you almost shouldn't care about you in a branch with a measure of 0.000001 even though it may involve drastic macroscopic difference for you in that branch. You being macroscopic doesn't imply you shouldn't care about your low-measure instances.

Yes this is true. Not caring about low-measure instances is a very different proposition from not caring about macroscopically indistinguishable differences. We should care about low-measure instances in proportion to the measure, just as in classical decision theory we care about low-probability instances in proportion to the probability.

Sometimes people express concern that AIs may replace them in the workplace. This is (mostly) silly. Not that it won't happen, but you've gotta break some eggs to make an industrial revolution. This is just 'how economies work' (whether or not they can / should work this way is a different question altogether). 

The intrinsic fear of joblessness-resulting-from-automation is tantamount to worrying that curing infectious diseases would put gravediggers out of business.

There is a special case here, though: double digit unemployment (and youth unemployment, in particular) is a major destabilizing force in politics. You definitely don't want an automation wave so rapid that the jobless and nihilistic youth mount a civil war, sharply curtailing your ability to govern the dangerous technologies which took everyone's jobs in the first place.

As AI systems become more expensive, and more powerful, and pressure to deploy them profitably increases, I'm fairly concerned that we'll see a massive hollowing out of many white collar professions, resulting in substantial civil unrest, violence, chaos. I'm not confident that we'll get (i.e.) a UBI (or that it would meaningfully change the situation even if we did), and I'm not confident that there's enough inertia in existing economic structures to soften the blow.

The IMF estimates that current tech (~GPT 4 at launch) can automate ~30% of human labor performed in the US. That's a big, scary number. About half of these, they imagine, are the kinds of things you always want more of anyway, and that this complementarity will just drive production in that 15% of cases. The other 15%, though, probably just stop existing as jobs altogether (for various reasons, I think a 9:1 replacement rate is more likely than full automation, with current tech).

This mostly isn't happening yet because you need an ML engineer to commit Serious Time to automating away That Job In Particular. ML engineers are expensive, and usually not specced for the kind of client-facing work that this would require (i.e. breaking down tasks that are part of a job, knowing what parts can be automated, and via what mechanisms, be that purpose-built models, fine-tuning, a prompt library for a human operator, some specialized scaffolding...). There's just a lot of friction and lay-speak necessary to accomplish this, and it's probably not economically worth it for some subset of necessary parties (ML engineers can make more elsewhere than small business owners can afford to pay them to automate things away, for instance).

So we've got a bottleneck, and on the other side of it, this speculative 15% leap in unemployment. That 15% potential leap, though, is climbing as capabilities increase (this is tautologically true; "drop in replacement for a remote worker" is one major criteria used in discussions about AI progress).

I don't expect 15% unemployment to destabilize the government (Great Depression peak was 25%, which is a decent lower bound on 'potentially dangerous' levels of unemployment in the US). But I do expect that 15% powder keg to grow in size, and potentially cross into dangerous territory before it's lit.

Previously, I'd actually arrived at that 30% number myself (almost exactly one year ago), but I had initially expected:

1. Labs would devote substantial resources to this automation, and it would happen more quickly than it has so far.
2. All of these jobs were just on the chopping block (frankly, I'm not sure how much I buy the complementarity argument, but I am An Internet Crank, and they are the International Monetary Fund, so I'll defer to them).

These two beliefs made the situation look much more dire than I now believe it to be, but it's still, I claim, worth entertaining as A Way This Whole Thing Could Go, especially if we're hitting a capabilities plateau, and especially if we're doubling down on government intervention as our key lever in obviating x-risk.

[I'm not advocating for a centrally planned automation schema, to be clear; I think these things have basically never worked, but would like to hear counterexamples. Maybe just like... a tax on automation to help staunch the flow of resources into the labs and their surrogates, a restructuring of unemployment benefits and retraining programs and, before any of that, a more robust effort to model the economic consequences of current and future systems than the IMF report that just duplicates the findings of some idiot (me) frantically reviewing BLS statistics in the winter of 2023.]

OK but your original comment reads like you're offering things not mattering cosmically as a reason for thinking MWI doesn't change anything (if that's not a reason, then you haven't given any reason, you've just stated your view). And I think that's a good argument - if you have general reasons that are independent of specific physics to think nothing matters (cosmically), then it will follow that nothing matters in MWI as well. I was responding to that argument.

Because this is what intelligence is - picking actions that lead to better outcomes. Pursuing current goal will ensure good results in one future, preparing for every goal will ensure good results in many more futures.

On the side - could you elaborate why you think "relu better than sigmoid" is a "weird trick", if that is implied by this question?

The reason that I thought to be commonly agreed is that it helps with the vanishing gradient problem (this could be shown from the graphs).

I always assume when thinking about future dangerous models that they have access to some sort of black-box memory. Do we think there is a non-negligible chance that an AI that doesn't have hidden memory, only English-language CoT, will be able to evade our monitoring and execute a rouge deployment? (Not a rhetorical question, there might be a way I haven't thought of.)

So I think that assuming the AI being stateless when thinking about future risk is not a good idea, as I think the vast majority of the risk comes from AIs for which this assumption is not true. I think this makes the post's claim significantly weaker, as the untrusted AI can just learn which of its attacks get replaced, and improve its attacks based on this data. 

It still needs to avoid getting top 1% suspiciousness score and getting audited though, as it would be dangerous to the AI to ever get caught writing a text that clearly intends to jailbreak the monitor. So I agree that even an AI with black-box memory will still be severely constrained in how many and how egregious jailbreak attempts it can try.

Here is a thing that I think would be cool to analyze sometime: How difficult would it have been for AI systems to discover and leverage historical hardware-level vulnerabilities, assuming we had not discovered them yet. Like, it seems worth an analysis to understand how difficult things like rowhammer, or more recent speculative execution bugs like Spectre and Meltdown would have been to discover, and how useful they would have been. It's not an easy analysis, but I can imagine the answer coming out obviously one way or another if one engaged seriously with the underlying issue.