LessWrong 2.0 Reader

I don't know the answer to how much cybercrime is really costing, but I think your economic analysis is not accurately tracking "what GDP means".

Arms length financial transactions of "money points for services or goods" operates on the basis of scarcity, monopoly pricing power, and other power concerns that are locally legible inside of bilateral exchanges between reasonable agents.

GDP does not track the "reserve price" of consumers of computational services, where conditional on a computing service hypothetically being monopolistically priced, the person would hypothetically pay a LOT for that service.

Various surveys and a bit of logic suggest that people would hypothetically pay thousands or in many cases even tens of thousands of dollars for access to the internet even though the real cost is much much less.

By contrast, GDP just measures the "true scarcity... and lawful evil induced scarcity" part of the economy (mushed together and swirled around, so the DMCA makes hacking printer ink cartridges full of producer-added malware illegal, rather than subsidizing such heroic hacking work, as would occur under benevolent governance, and so on).

Linus Torvalds is probably owed a "debt of gratitude", by Earth, on the order of many billions, and possibly trillions, but he gave away Linux and has never been paid anything like that amount, and so the value he created and gave away does not show up in GDP. (Not just him, there's a whole constellation of rarely sung heroes and moderately happy dudes who were part of a hobbyist ecosystem that created the modern digital world between 1970 and 2010 and gave it away for free).

On a deeper level, the inability to measure or encourage the "post-scarcity" or "public goods" part of the human "economy" (if you can even call it an "economy" when it doesn't run on bilateral arms-length self-interested deals) is part of why such goods are underproduced by default, in general, and have been underproduced for all of human history.

Within this frame, it seems very plausible that the computational consumer surplus that cybercriminals attack is worth huge amounts of money to protect, even though it was acquired very cheaply from people like Linus.

Presumably humans are not yet in "private scarcity-based equilibrium" with the economics of computation processes?

In the long run it might be reasonable to expect the "a la carte computer security situation" (where every technical system becomes a game of whack-a-mole fighting many very specific ways to ruin everything in the computational commons) to devolve until most uses of most computer processes have almost no consumer surplus, because the costs of paying for a la carte help with computer security almost perfectly balances against the consumer surplus from using "essentially free compute".

This would not happen if good computer security practices arise that can somehow preserve the existing (and probably massive) consumer surplus around computers such that "using the internet and computers in general in a safe way is very cheap because computer security itself is easy to get right and spread around as a public good with nearly no marginal cost".

Like... hypothetically the government could make baseline "secure and super valuable" computing systems.

But it doesn't.

A private ad-based surveillance and propaganda corporation "solved search and created lots of billionaires" NOT the library of congress.

The NSA tries to make sure that most consumer hardware and software is insecure so that the <0.5% of consumer buyers that happen to be mobsters or terrorists can be spied on, rather than putting out open source defensive software for everyone.

People like Aaron Swartz and Moxie did, mostly for free, the thigns that a benevolent government would do if a benevolent government existed.

But no actively benevolent governments exist.

In Anathem, Neil Stephenson (who is very smart, in a very fun way) posits a giant science inquisition that prevents technological advancement (leading to AGI or nukes or bioweapons or what have you) and lets humanity "experience the current tech scale" for thousands of years with instabilities factored out and only locally stable cultural loops retained...

...in that world it is just taken for granted that 99.999% of the internet is full of auto-generated lies called "bogons" that are put out by computer security companies so as to force consumers to pay monthly subscriptions for expensive bogon filtering software that make their handheld jeejaws only really good for talking with close personal friends or business associates. It is just normal to them, for the internet to exist and be worthless, like it is normal to us for lies in ads and on the news to be the default.

Anathem's future contains no wikipedia, because wikipedia is like linux: insanely valuable, yet not scarce, with very few dollars directed to it in ways that ensures (1) it isn't hacked from the outside and (2) the leadership doesn't ruin it for personal or ideological profit from the inside.

Anathem offers us a bleak "impossible possible future" but not the bleakest.

Things probably won't happen that way because that exact way of stabilizing human civilization is unlikely, but Anathem honestly grapples with the broader issue where information services are (1) insanely valuable and (2) also nearly impossible for the market to properly price.

In this comment you talked about a scenario where a perfect map of London was created by “an (extremely unlikely) accident”. I think that’s an “edge case of representation”, right?

I think it clearly wasn't a case of representation, in the same way a random string clearly doesn't represent anything, nor a cloud that happens to look like something. Those are not edge cases; edge cases are arguably examples where something satisfies a vague predicate partially, like chess is "sporty" to some degree. ("Life" is arguably also vague because it isn't clear whether it requires a metabolism, which isn't present in viruses.)

Again, a central case of representation would be:

(A) The map has lots and lots of features that straightforwardly correspond to features of London

(B) The reason for (A) is that the map was made by an optimization process that systematically (though imperfectly) tends to lead to (A), e.g. the map was created by a human map-maker who was trying to accurately map London

(C) The map is in fact useful for navigating London

I think maps are not overly central cases of representation.

(A) is clearly not at all required for clear cases of representation, e.g. in case of language (abstract symbols) or when a fist is used to represent a gas station, or a stone represents a missing chess piece, etc. That's also again the difference between the sock and the contents of the suitcase. The latter concept doesn't resemble the thing it represents, like a sock, even though it may refer to one.

Regarding (C), representation also doesn't have to be helpful to be representation. A secret message written in code is in fact optimized to be maximally unhelpful to most people, perhaps even to everyone in case of an encrypted diary. And the local explaining the way to the next gas station with the help of his hands clearly means something regardless of whether someone is able to understand what he means. Yes, maps specifically are mostly optimized to be helpful, but that doesn't mean representation has to be helpful. It's similar to portraits: they tend to be optimized for representing someone, resembling someone, and for being aesthetically pleasing. Which doesn't mean representation itself requires resemblance or aesthetics.

I agree that something like (B) might be a defining feature of representation, but not quite. Some form of optimization seems to be necessary (randomly created things don't represent anything), but optimization for (A) (resemblance) is necessary only for things like maps or pictures, but not for other forms of representation, as I argued above.

And then there are edge cases, like the Gettier problem, or vague impressions, or partially-correct ideas, or hearsay, or vibes, or Scott/Yvain, or latent variables that are somehow helpful for prediction but in a very hard-to-interpret and indirect way, or habits, etc. Those are edge-cases of map-territory correspondence. And arguing about what if anything they’re “really” representing is a waste of time, just as arguing about whether a virus is “really” alive is a waste of time.

The Gettier problem is a good illustration of what I think is the misunderstanding here: Gettier cases are not supposed to be edge cases of knowledge. Just like the randomly created string "Eliezer Yudkowsky" is not an edge case of representation. Edge cases have to do with vagueness, but the problem with defining knowledge is not that the concept may be vague. Gettier cases are examples where justified true belief is intuitively present but knowledge is intuitively absent. Not partially present like in edge cases. That is, we would apply the first terms (justification, truth, belief) but not the latter (knowledge). Vagueness itself is often not a problem for definition. A bachelor is clearly an unmarried man, but what may count as marriage, and what separates men from boys, is a matter of degree, and so too must be bachelor status.

Now you may still think that finding something like necessary and sufficient conditions for representation is still "a waste of time" even if vagueness is not the issue. But wouldn't that, for example, also apply to your attempted explication of "intention"? Or "awareness"? In my opinion, all those concepts are interesting and benefit from analysis.

reasonably publicly accessible by an ordinary person from sources other than a covered model or covered model derivative

Seems like it'd pretty obviously cover information generated by non-covered models that are routinely used by many ordinary people (as open source image models currently are).

As a sidenote, I think the law is unfortunately one of those pretty cursed domains where it's hard to be very confident of anything as a layman without doing a lot of your own research, and you can't even look at experts speaking publicly on the subject since they're often performing advocacy, rather than making unbiased predictions about outcomes.  You could try to hire a lawyer for such advice, but it seems to be pretty hard to find lawyers who are comfortable giving their clients quantitative (probabilistic) and conditional estimates.  Maybe this is better once you're hiring for e.g. general counsel of a large org, or maybe large tech company CEOs have to deal with the same headaches that we do.  Often your best option is to just get a basic understanding of how relevant parts of the legal system work, and then do a lot of research into e.g. relevant case law, and then sanity-check your reasoning and conclusions with an actual lawyer specialized in that domain.

Joshua Achiam is Head of Mission Alignment ("working across the company to ensure that we get all pieces (and culture) right to be in a place to succeed at the mission"), this is not technical AI alignment. 

I want to draw a clear distinction between the hypothesis I mention in the OP (your 'causal' explanation) and the 'acausal' explanation you mention here. You were blending the two together during our conversation, but I think there are important differences.

In particular, I am interested in the question: does o1 display any "backwards causation" from answers to chain-of-thought? Would its training result in chain-of-thought that is optimized to justify a sort of answer it tends to give (eg, hallucinated URLs)?

This depends on details of the training which we may not have enough information on (plus potentially complex reasoning about the consequences of such reasoning).

Update with two new responses:

I think this is 10 generals, 1 petrov, and one other person (either the other petrov or a citizen, not sure, wasn't super rigorous)

Yeah that makes sense. I think I underestimated the extent to which "warning shots" are largely defined post-hoc, and events in my category ("non-catastrophic, recoverable accident") don't really have shared features (or at least features in common that aren't also there in many events that don't lead to change).

Great! I'd love to have included a remark that one, as a human, might anticipate forward-chainy/rational reasoning in these systems because we're often taking the "thought" metaphor seriously/literally in the label "chain-of-thought", rather than backwardy/rationalization "reasoning". 

But since it is is at least somewhat intelligent/predictive, it can make the move of "acausal collusion" with its own tendency to hallucinate, in generating its "chain"-of-"thought". That is, the optimization to have chain-of-thought in correspondence with its output can work in the backwards direction, cohering with bad output instead of leading to better output, a la partial agency [? · GW].

(Admittedly human thoughts do a lot of rationalization as well. So maybe the mistake is in taking directionality implied by "chain" too seriously?)

Maybe this is obvious [LW · GW], but it could become increasingly reckless to not notice when you're drawing the face of "thoughts" or "chains" on CoT shoggoth-movements . You can be misled into thinking that the shoggoth is less able to deceive than it actually is.

Less obvious but important: in the reverse direction, drawing  "hacker faces" on the shoggoth, as in the case of the Docker hack (section 4.2.1), can mislead into thinking that the shoggoth "wants" to or tends to hack/undermine/power-seek more than it actually, independently does. It seems at least somewhat relevant that the docker vulnerability was exploited for a challenge that was explicitly about exploiting vulnerability. Even though it was an impressive meta-hack, one must wonder how much this is cued by the prompt and therefore is zero evidence for an autonomy telos---which is crucial for the deceptive optimizer story---even though mechanistically possible. 

(The word "independently" above is important: if it takes human "misuse"/participation to trigger its undermining personas, we also might have more of a continuous shot at pausing/shutdown or even corrigibilty.)

I was going to post this as a comment, but there's also an answer here: I'd say calling o1 "deceptive" could be as misleading as calling it aligned if it outputs loving words. 

It has unsteady referentiality, at least from the POV of the meanings of us life-forms. Even though it has some closeness to our meanings and referentiality, the quantity of the unsteadiness of that referentiality can be qualitative. Distinguishing "deceptively aligned mesa-optimizer" from "the tentacles of the shoggoth I find it useful to  call 'words' don't work like 'words' in some annoying ways" is important, in order to protect some of that (quantitatively-)qualitative difference. Both for not dismissing risks and for not hallucinating them.

Apparently the current funding round hasn't closed yet and might be in some trouble, and it seems much better for the world if the round was to fail or be done at a significantly lower valuation (in part to send a message to other CEOs not to imitate SamA's recent behavior). Zvi saying that $150B greatly undervalues OpenAI at this time seems like a big unforced error, which I wonder if he could still correct in some way.

Also from WSJ