LessWrong 2.0 Reader

The model you propose isn't related to the underlying reality, which involves tracing back your personal history. You've asserted that you can group epochs of history into buckets and assign %age numbers for how many humans you'd likely be descended from, and that this is a reasonable method for approximating an answer to this question. That is not in fact obvious. There's a lot of underlying assumptions, some of which you call out on that guesstimate page, and some I've called out here.

So it's reasonable to spot check the claim via a separate method, as I did above and got an answer many sigmas removed from yours. That's an indication that this isn't a very good method for estimating the answer, and I'm not inclined to tweak numbers to get it to fit my preconceived notion of the right answer.

in some way, bureaucracy design is the exact opposite of machine learning. while the goal of machine learning is to make clusters of computers that can think like humans, the goal of bureaucracy design is to make clusters of humans that can think like a computer

I'm sorry, I literally don't understand what you're saying here. What does "care about long-run power for similar values" mean? Do you care about maximizing your own power?

If we're talking about Darwinian contexts, systems which optimize for long-run power are in fact often selected out. Long-run benefit is of no utility unless short-term survival is taken care of, and long-run and short-term needs are often at odds.

So from a behavioral perspective I expect that you get systems which are optimizing for short-term survival. Indeed, I think this point is trivial and one which you probably agree with. What I'm saying is that short-term survival and long-run power are not necessarily correlated, and I think that is the crux.

Let's take an example that is not rigorously worked out and is probably wrong in some details, but can serve to illustrate. Long-run power in humans is derivative of social structure: the leaders of the tribe control the tribe's collective resources. If you want power in human society, you need to rise to the top of our social structures, and the optimal ways of doing that are generally not nice.

But why do we have social structures at all? Why are we organized as tribes? Because we are social animals who prefer the company of others. Being with others beats striking out on your own because, generally speaking, other people in the tribe are nice. Niceness creates an environment in which sycophantic power seeking pays off, but only because there is severe evolutionary pressure towards niceness in the first place. [In the environment which gave rise to humans, not as a general statement.]

I'd also add that a high fraction of these costs won't be increased if you improve cyber crime productivity (by e.g. 10%). As in, maybe a high fraction of the costs are due to the possiblity of very low effort cyber crime (analogous to the cashier case).

And Fabien's original motivation was more closely related to this.

Bonus: post-train on similar tasks

I don't think that 'post-train on similar tasks' should be considered just a bonus. I think that that's a key part of adequate safety testing. Fine-tuning on similar tasks has a substantial history in ML literature when it comes to evaluating the max capability of a general model on a specific task. It is pretty standard to report variations of: zero-examples (aka zero-shot), n-examples (aka n-shot), 1-attempt, n-attempts (with a resolution scheme such as majority solution gets submitted), fine-tuning on similar task (or subset of the examples for this task). 

This isn't some weird above-and-beyond demand, it's a standard technique used for assessing capabilities. I would go so far as to say that I would suspect that someone who didn't try this didn't actually want to elicit the full capabilities of the model.

The justification for fine-tuning not being a part of the reported assessment of general purpose models is that you want to measure what users will be expected to experience as they interact with the model. But even closed-weight API-only models often offer a fine-tuning API. And definitely if you are trying to assess the risk of the weights being stolen, you need to consider fine-tuning.

There are (at least) two different meanings of "costing" in large-scale economic impact thinking.  The narrow meaning is "actual amount spent on this topic".  The more common (because it's a bigger number) meaning is "how much bigger would the economy be in the counterfactual world that doesn't have this feature".

The article linked from Wikipedia says

The damage cost estimation is based on historical cybercrime figures including recent year-over-year growth, a dramatic increase in hostile nation-state sponsored and organized crime gang hacking activities, and a cyberattack surface which will be an order of magnitude greater in 2025 than it is today.

Cybercrime costs include damage and destruction of data, stolen money, lost productivity, theft of intellectual property, theft of personal and financial data, embezzlement, fraud, post-attack disruption to the normal course of business, forensic investigation, restoration and deletion of hacked data and systems, reputational harm, legal costs, and potentially, regulatory fines.

Which puts it in the second category - most of these costs are NOT direct expenses, but indirect and foregone value.   That doesn't make it wrong, exactly, just not comparable to "real" measures (which GDP and GPP isn't either, but it's more defensible).

It's extremely unclear whether LLM adoption and increasing capabilities will shift the equilibrium between attack and defense on these fronts.  Actually, it's almost certain that it will shift it, but it's uncertain how much and in what direction, on what timeframes.

It's further unclear whether legislation can slow the attacks more than they hinder defense.  

Mostly, it's not a useful estimate or model for reasoning about decisions.

Important bit of history: Kevin Esvelt (head of SecureBio, who I have been working for) invented the Gene Drive back in 2013. He says he was at first scared that the idea could be misused, and thought carefully about the offense-defense balance before concluding that it was safely defense-dominant. He says that he shared his idea only after concluding it was safe.

The reason SecureBio is concerned about other biological threats, and the role AI might play in them, is that there are a wide range of things it is possible to do with genetic engineering. Some of these, unlike Gene Drives, are highly offense dominant and would be really bad for humanity if deployed. 

So, in conclusion, I myself have been excited for Gene Drives to get deployed for years now. I think that anti-malaria-hosting-mosquito Gene Drives are very valuable and very safe. I am in fact pretty mad that there has been so much delay, and so many lives negatively impacted by society's excess caution on this front.

Why are we so overly cautious about things which have lots of evidence to support them being safe, and at the same time so reckless about other things which are genuinely dangerous? I feel frustrated and upset about humanity's poor collective risk assessment and decision making.

There should be a dropdown menu at the left side in the input box (opposite the "submit" button).

I don't know the answer to how much cybercrime is really costing, but I think your economic analysis is not accurately tracking "what GDP means".

Arms length financial transactions of "money points for services or goods" operates on the basis of scarcity, monopoly pricing power, and other power concerns that are locally legible inside of bilateral exchanges between reasonable agents.

GDP does not track the "reserve price" of consumers of computational services, where conditional on a computing service hypothetically being monopolistically priced, the person would hypothetically pay a LOT for that service.

Various surveys and a bit of logic suggest that people would hypothetically pay thousands or in many cases even tens of thousands of dollars for access to the internet even though the real cost is much much less.

By contrast, GDP just measures the "true scarcity... and lawful evil induced scarcity" part of the economy (mushed together and swirled around, so the DMCA makes hacking printer ink cartridges full of producer-added malware illegal, rather than subsidizing such heroic hacking work, as would occur under benevolent governance, and so on).

Linus Torvalds is probably owed a "debt of gratitude", by Earth, on the order of many billions, and possibly trillions, but he gave away Linux and has never been paid anything like that amount, and so the value he created and gave away does not show up in GDP. (Not just him, there's a whole constellation of rarely sung heroes and moderately happy dudes who were part of a hobbyist ecosystem that created the modern digital world between 1970 and 2010 and gave it away for free).

On a deeper level, the inability to measure or encourage the "post-scarcity" or "public goods" part of the human "economy" (if you can even call it an "economy" when it doesn't run on bilateral arms-length self-interested deals) is part of why such goods are underproduced by default, in general, and have been underproduced for all of human history.

Within this frame, it seems very plausible that the computational consumer surplus that cybercriminals attack is worth huge amounts of money to protect, even though it was acquired very cheaply from people like Linus.

Presumably humans are not yet in "private scarcity-based equilibrium" with the economics of computation processes?

In the long run it might be reasonable to expect the "a la carte computer security situation" (where every technical system becomes a game of whack-a-mole fighting many very specific ways to ruin everything in the computational commons) to devolve until most uses of most computer processes have almost no consumer surplus, because the costs of paying for a la carte help with computer security almost perfectly balances against the consumer surplus from using "essentially free compute".

This would not happen if good computer security practices arise that can somehow preserve the existing (and probably massive) consumer surplus around computers such that "using the internet and computers in general in a safe way is very cheap because computer security itself is easy to get right and spread around as a public good with nearly no marginal cost".

Like... hypothetically the government could make baseline "secure and super valuable" computing systems.

But it doesn't.

A private ad-based surveillance and propaganda corporation "solved search and created lots of billionaires" NOT the library of congress.

The NSA tries to make sure that most consumer hardware and software is insecure so that the <0.5% of consumer buyers that happen to be mobsters or terrorists can be spied on, rather than putting out open source defensive software for everyone.

People like Aaron Swartz and Moxie did, mostly for free, the thigns that a benevolent government would do if a benevolent government existed.

But no actively benevolent governments exist.

In Anathem, Neil Stephenson (who is very smart, in a very fun way) posits a giant science inquisition that prevents technological advancement (leading to AGI or nukes or bioweapons or what have you) and lets humanity "experience the current tech scale" for thousands of years with instabilities factored out and only locally stable cultural loops retained...

...in that world it is just taken for granted that 99.999% of the internet is full of auto-generated lies called "bogons" that are put out by computer security companies so as to force consumers to pay monthly subscriptions for expensive bogon filtering software that make their handheld jeejaws only really good for talking with close personal friends or business associates. It is just normal to them, for the internet to exist and be worthless, like it is normal to us for lies in ads and on the news to be the default.

Anathem's future contains no wikipedia, because wikipedia is like linux: insanely valuable, yet not scarce, with very few dollars directed to it in ways that ensures (1) it isn't hacked from the outside and (2) the leadership doesn't ruin it for personal or ideological profit from the inside.

Anathem offers us a bleak "impossible possible future" but not the bleakest.

Things probably won't happen that way because that exact way of stabilizing human civilization is unlikely, but Anathem honestly grapples with the broader issue where information services are (1) insanely valuable and (2) also nearly impossible for the market to properly price.