LessWrong 2.0 Reader

Yes! Which side do you want to be on? Want to do it in person, or in this comment thread?

OpenAI CFO Sarah Friar warns us that the next model will only be about one order of magnitude bigger than the previous one.

The question is whether she's tlking parameter count, nominal training flops, or actual cost. In general, GPT generations so far have been roughly one order of magnitude apart in parameter count and training cost, and roughly two orders of magnitude in nominal training flops (parameter count x training tokens). Since she's a CFO, and that was a financial discussion, I assume she natively thinks in terms of training cost, so the 'correct' answer to her is one order of magnitude not two, so my suspicion is that she's actually talking in terms of parameter count. So I don't think she's warning us of anything, I think she's just projecting a straight line on a logarithmic plot. I.e. business as usual at OpenAI.

The point of made up numbers is that they are a helpful tool for teasing out some implicit information from your intuitions, which is often better than not doing that at all, but, it's important that they are useful in a pretty different way from numbers-you-empirically-got-from-somewhere, and thus it's important that they be clearly labeled as made up numbers that Tsvi made up numbers.

See: If it's worth doing, it's worth doing with Made Up Statistics

During this particular tutorial, Julia tried to explain Bayes’ Theorem to some, er, rationality virgins. I record a heavily-edited-to-avoid-recognizable-details memory of the conversation below:

Julia: So let’s try an example. Suppose there’s a five percent chance per month your computer breaks down. In that case…
Student: Whoa. Hold on here. That’s not the chance my computer will break down.
Julia: No? Well, what do you think the chance is?
Student: Who knows? It might happen, or it might not.
Julia: Right, but can you turn that into a number?
Student: No. I have no idea whether my computer will break. I’d be making the number up.
Julia: Well, in a sense, yes. But you’d be communicating some information. A 1% chance your computer will break down is very different from a 99% chance.
Student: I don’t know the future. Why do you want to me to pretend I do?
Julia: (who is heroically nice and patient) Okay, let’s back up. Suppose you buy a sandwich. Is the sandwich probably poisoned, or probably not poisoned?
Student: Exactly which sandwich are we talking about here?

In the context of a lesson on probability, this is a problem I think most people would be able to avoid. But the student’s attitude, the one that rejects hokey quantification of things we don’t actually know how to quantify, is a pretty common one. And it informs a lot of the objections to utilitarianism – the problem of quantifying exactly how bad North Korea shares some of the pitfalls of quantifying exactly how likely your computer is to break (for example, “we are kind of making this number up” is a pitfall).

The explanation that Julia and I tried to give the other student was that imperfect information still beats zero information. Even if the number “five percent” was made up (suppose that this is a new kind of computer being used in a new way that cannot be easily compared to longevity data for previous computers) it encodes our knowledge that computers are unlikely to break in any given month. Even if we are wrong by a very large amount (let’s say we’re off by a factor of four and the real number is 20%), if the insight we encoded into the number is sane we’re still doing better than giving no information at all (maybe model this as a random number generator which chooses anything from 0 – 100?)

This depends on many things (one's skills, one's circumstances, one's preferences and inclinations (the efficiency of one's contributions greatly depends on one's preferences and inclinations)).

I have stage 4 cancer, so statistically, my time may be more limited than most. I’m a PhD student in Computer Science with a strong background in math (Masters).

In your case, there are several strong arguments for you to focus on research efforts which can improve your chances of curing it (or, at least, of being able to maintain the situation for a long time), and a couple of (medium strength?) arguments against this choice.

For:

• If you succeed, you'll have more time to make impact (and so if your chance of success is not too small, this will contribute to your ability to maximize your overall impact, statistically speaking).

• Of course, any success here will imply a lot of publicly valuable impact (there are plenty of people in a similar position health-wise, and they badly need progress to occur ASAP).

• The rapid development of applied AI models (both general purpose models and biology-specific models) creates new opportunities to datamine and juxtapose a variety of potentially relevant information and to uncover new connections which might lead to effective solutions. Our tools progress so fast that people are slow to adapt their thinking and methods to that progress. So new people with fresh outlook have reasonable shots (of course, they should aim for collaborations). In this sense, your PhD CS studies and your strong math is very helpful (a lot of the relevant models are dynamic systems, timing of interventions is typically not managed correctly as far as I know (there are plenty of ways to be nice to particularly vulnerable tissues by timing the chemo right and thus being able to make it more effective, but this is not a part of the standard-of-care yet as far as I know), and so on).

• You are likely to be strongly motivated and to be able to maintain strong motivation. At the same time you'll know that it is the result that counts here, not the effort, and so you will be likely to try your best to approach this in a smart way, not in a brute force effort way.

Possibly against:

• The psychological implications of working on your own life-and-death problem are non-trivial. One might choose to embrace them or to avoid them.

• Focusing on "one's own problem" might be compatible or not very compatible with this viewpoint you once expressed: https://www.lesswrong.com/posts/KFWZg6EbCuisGcJAo/immortality-or-death-by-agi-1?commentId=QYDvovQZevDmGtfXY [LW(p) · GW(p)]

(Of course, there are plenty of other interesting things one can do with this background (PhD CS studies and strong math). For example, one might decide to disregard the health situation and to dive into technical aspects of AI development and AI existential safety issues, especially if one's estimate of AI timelines yields really short timelines.)

Do we really want to curate a post featuring a made up graph of entirely made up numbers? Maybe I entirely missed the point of leading off with that figure.

Quick updates:

• I worried this was a loophole: "the Trust Agreement also authorizes the Trust to be enforced by the company and by groups of the company’s stockholders who have held a sufficient percentage of the company’s equity for a sufficient period of time." An independent person told me it's a normal Delaware law thing and it's only relevant if the Trust breaks the rules. Yay! This is good news, but I haven't verified it and I'm still somewhat suspicious (but this is mostly on me, not Anthropic, to figure out).
• I started trying to deduce stuff about who holds how many shares. I think this isn't really doable from public information, e.g. some Series C shares have voting power and some don't and I don't know how to tell which investors have which. But I got a better understanding of this stuff and I now tentatively think the Transfer Approval Threshold is super reasonable. Yay!
  • Also even if I understood the VC investments perfectly I'd be confused by the (much larger?) investments by Amazon and Google
  • If you're savvy in this and want to help, let me know and I'll share my rough notes

(On the other hand, it's pretty concerning that the Trust hasn't replaced Jason, who left in December, nor Paul, who left in April. Also note that it hasn't elected a board member to fill Luke's seat.)

I guess my LTBT-related asks for Anthropic are now:

• Commit that you'll promptly announce publicly if certain big LTBT-related changes happen
• Publish the Trust Agreement (and any other relevant documents), or at least let a non-Anthropic person I trust read it (under an NDA if you want) and publish whether they have big concerns

(This isn't high-priority for me; I just get annoyed when Anthropic brags about its LTBT without having justified that it's great.)

Unless I misread, it said "mRNA" before.

Short-term gasoline availability after a natural disaster is a good example.  Very high prices do not actually lead to more supply, because transportation constraints are binding, and the road crews don't get any of the increase in prices.  In a model world, it's good, the gasoline vendors pay some of their earnings to the road crews to prioritize their (very valuable) goods.  In the real world, the supply chain is too long and interconnected for price signals to change much in terms of availability.

Over longer timeframes, and for more normal goods and situations, the model is very powerful, and price fixing is extremely harmful.  I'm not supporting rent control in any fashion.  In fact, I'm not exactly supporting gasoline price limits in an emergency, just pointing out that the usual arguments for price flexibility may not apply.

.

Is this a link post or is this a post original to LW? Something about the style makes me think this is copied from somewhere else (and I would want to link to it), but I might be wrong.

I don't really care about preserving status-quo humanity

By non-extinction I don't mean freezing the status quo of necessarily biological Homo sapiens on Earth, though with ASI I expect that non-extinction in particular keeps most individual people alive indefinitely in the way they individually choose. I think this is a more natural reading of non-extinction (as a sense of non-doom) than a perpetual state of human nature preserve.

By doom I mean the universe gets populated by AI with no moral worth

So losing cosmic wealth is sufficient to qualify an outcome as doom, as in Bostrom's existential risk. What if Earth literally remains untouched by machines, protected from extinction level events but otherwise left alone, is it still doom in this sense? What if aliens that hold moral worth but currently labor under an unkind regime exterminate humanity, but then the aliens themselves spread to the stars (taking them for themselves) and live happily ever after, is that still not a central example of doom?

My point is that the term is highly ambiguous, resolution criteria for predictions that involve it are all over the place, so it's no good for use in predictions or communication. There is illusion of transparency where people keep expecting it to be understood, and then others incorrectly think that they've understood the intended meaning. Splitting doom into extinction and loss of cosmic wealth seems less ambiguous.