I agree. One problem of grade inflation is that we lose the ability to measure excellence.

If you have a scale e.g. from 1 to 5, where the average grade is 3, if you somehow hire a magical teacher, you could see an improvement of the mean student, let's say from 3 to 2. Then you might conclude that the teacher does something right, and maybe try to replicate that.

But if instead the average grade is 1, hiring the same magical teacher would... leave the average grade at 1. Are all students "okay" in the subject? That's 1. Are they excellent? Also 1. Do half of them participate at international competitions and bring medals to their school? Still 1. From the perspective of a research which only looks at these numbers, no change.

If the average grade is 1, then basically all you can see is changes among the worst performing students. Which... is also an important thing, but it is a different thing; the interventions that improve grades in this group do not necessarily work for anyone else, e.g. teaching to the test during the "No Child Left Behind" policy.

*

A frequently proposed alternative is grading on the curve, which in my opinion has the same fundamental problem. If you merely sort the students in the classroom by their raw score, and then assign a different grade to each 20% of the class... that allows you to distinguish the individuals within the classroom, but the average of the classroom remains constant (this time, by definition), so you cannot see which teaching interventions were useful.

It also creates weird incentives, because you are de facto punished for being in the same class with smart students, so from the perspective of grade optimization it would be better for you to join a worse class.

This might be avoided by using standardized tests, and calibrating them on the entire population. Even then, we would have another problem, which is selecting what kinds of problems go to the test. For example, if you add a few insanely difficult questions, then the winners would be those who know things far beyond what is taught at school. But if you only use "age-appropriate" questions, then the best students cannot get beyond 100%, and they might actually lose some points for attention deficit. I guess my point here is that even if we use normalized tests, we cannot treat the contents of the test as a black box, because differently constructed tests measure different things.

An underlying problem is, basically, what is the goal of the school education? If it is producing the solid average, then excellence is beyond that goal, so it's kinda fair if we exclude it. But if the goal is producing excellence, then clearly such goal cannot be achieved for most of the population. If we say that both goals are desired (excellence for those who can achieve it; the solid fundamentals for everyone else), then perhaps the grading system should reflect it somehow. There should be a grade that makes it intuitively very clear that it means e.g. "this student has all the solid fundamentals, but none of the excellence". Because there will be many students like this.

Maybe something like in video games: F, D, C, B, A, S, SS, SSS? With "A" corresponding to the current best grade, and getting an "S" requires you to do something beyond normal school duties (e.g. participating at a math olympiad), so there would be nothing wrong if there is no S-graded student in the entire school. We could even make it a requirement that grades F-A can be freely awarded by the local teacher, but getting an S and beyond requires some kind of independent exam.

(I am nearly certain this is actually a LLM model posting.)

My AI predictions

(I did not carefully think about my predictions. I just wanted to state them somewhere because I think it's generally good to state stuff publicly.)

(Do not assume my future self will still make the same predictions as I am now.)

(I do not comment on my underlying models for deriving the predictions. Please make no assumptions about those just because other people with some models might make similar predictions or so. My reasons might be different.)

TLDR: I don't know.

Timelines

Conditional on no strong governance success that effectively prevents basically all AI progress, and conditional on no huge global catastrophe happening in the meantime:

How long until the earth gets eaten? 10th/50th/90th percentile: 3y, 12y, 37y.

How long until an AI reaches Elo 4000 on codeforces? 10/50/90: 9mo, 2.5y, 11.5y

How long until an AI is better at math research than the best human mathmatician according to the world's best mathematicians? 10/50/90: 2y, 7.5y, 28y

Takeoff Speed

• I'm confident (94%) that it is easier to code an AI on a normal 2020 laptop that can do Einstein-level research at 1000x speed, than it is to solve the alignment problem very robustly^[1].^[2]
  • AIs might decide not to implement the very efficient AGIs in order to scale safer and first solve their alignment problem, but once a mind has solved the alignment problem very robustly, I expect everything to go extremely quickly.
• However, the relevant question is how fast AI will get smarter shortly before the point where ze^[3] becomes able to solve the alignment problem (or alternatively until ze decides making itself smarter quickly is too risky and it should cooperate with humanity and/or other similarly smart AIs currently being created to solve alignment).
  • So the question is: Will we get to this point by incremental progress that yields smallish improvements (=slow), or by some breakthrough that when scaled up can rush past the human intelligence level very quickly (=fast)?
    • I'm very tentatively leaning more towards the "fast" side, but i don't know.
    • I'd expect (80%) to see at least one more paradigm shift that is at least as big as the one from LSTMs to transformers. It's plausible to me that the results from the shift will come faster because we have greater computer overhang. (Though also possible it will just take even more compute.)
      • It's possible (33%) that the world ends within 1 year of a new major discovery^[4]. It might just very quickly improve inside a lab over the course of weeks without the operators there really realizing it^[5], until it then sectretly exfiltrates itself, etc.
        • (Btw, smart people who can see the dangerous implications of some papers proposing something should obviously not publicly point to stuff that looks dangerous (else other people will try it).)

1. ^{^}

   Hard to define what I mean by "very robustly", but sth like "having coded an AI program s.t. a calibrated mind would expect <1% of expected value loss if run, compared to the ideal CEV aligned superintelligence".

2. ^{^}

   I acknowledge this is a nontrivial claim. I probably won't be willing to invest the time to try to explain why if someone asks me now. The inferential distance is quite large. But you may ask.

3. ^{^}

   ze is the AI pronoun.

4. ^{^}

   Tbc, not 33% after the first major discovery after transformers, just after any.

5. ^{^}

   E.g. because the AI is in a training phase and only interacts with operators sometimes where it doesn't tell them everything. And in AI training the AI practices solving lots and lots of research problems and learns much more sample-efficient than transformers.

What is it that you know, that leads you to think that "SGD just doesn't "want" to teach LLMs agency"?

Mostly the fact that it hasn't happened already, on a merely "base" model. The fact that CoTs can improve models' problem-solving ability has been known basically since the beginning, but there's been no similar hacks found for jury-rigging agenty or insightful characters. (Right? I may have missed it, but even janus' community doesn't seem to have anything like it.)

But yes, the possibility that none of the current training loops happened to elicit it, and the next dumb trick will Just Work, is very much salient. That's where my other 20% are at.

You can split your brain and treat LLMs differently, in a different language. Rather, I can and I think most people could as well

Your comment seems to imply that the conjecture's "99%" is about circuits $C$ for which $P\left(C\right)$ is false. Otherwise, it would be impossible for a $V$ to miss 100% of random reversible circuits without missing the circuits $C$ for which $P\left(C\right)$ is true. In the conjecture, should "random reversible circuits $C$" be read as "random reversible circuits $C$ for which $P\left(C\right)$ is false"? It does not change much indeed, but I might have to correct my presentation of the conjecture here [LW · GW].

Thx, sounds very useful! 
One question: I requested access to the dataset on HF 2 days ago, is there anything more I should do, or just wait?
 

Oh, you're using AdamW everywhere? That might explain the continuous training loss increase after each spike, with AdamW needing time to adjust to the new loss landscape...

Lower learning rate leads to more spikes? Curious! I hypothesize that... it needs a small learning rate to get stuck in a narrow local optimum, and then when it reaches the very bottom of the basin, you get a ~zero gradient, and then the "normalize gradient vector to step size" step is discontinuous around zero.

Experiments springing to mind are:
1. Do you get even fewer spikes if you increase the step size instead?
2. Is there any optimizer setup at all that makes the training loss only ever go down?
2.1. Reduce the step size whenever an update would increase the training loss?
2.2. Use gradient descent instead of AdamW?

I'm not sure. I remember playing a bunch of games, like pokemon heart gold, lego starwars, and some other pokemon game where you were controlling little pokemon in 3rd person instead of controlling a human who threw pokeballs (anyone know that game? )

And like, I didn't speak English when I played them. So I had to figure out everything by just pressing random buttons and seeing responses. And this makes it a lot more difficult. Like I could open my "inventory" (didn't know what that was) and then use a "healing potion" (didn't know what that was), and then because my pokemon was at full health already, I would think the healing potion was useless, or think that items in inventory only cause text to appear on the screen, but that they don't have any effect on the actaul, and then I'd believe this until I accidentally clicked the inventory and randomly saw a change, or had failed a level so many times that I was getting desperate and just manually doing exhaustive search over all the actions.

But like, I'm very confident I was more action efficient than claude is. Mostly because like, if I enter a battle, and like fail 5 times more or less in the same way, you start to think something is awry, and start doing different stuff. And also just because, certain things become automatic after a short while, like moving around. For claude it takes the same amount of time each time. So if you're failing at a specific point in a battle, the fact that that point is responsible for you overall failing to progress, becomes very obvious, because anything other than that becomes automatic and trivial and you just do it instantly.

You have a point, although I don't think having a genuine feeling of despair is a hugely important variable.  As the story goes:

[Method actor Dustin] Hoffman had a grueling scene coming up, where his character hadn’t slept in three days, and Hoffman told [Sir Laurence] Olivier that to prepare for the scene, he too hadn’t slept for 72 hours.

“My dear boy,” replied Olivier, “why don’t you try acting?”

But more generally, even if you want to teach some kind of set of skills and resilience for dealing with things like "sitting still for hours a day", "doing hours of boring homework-like stuff", "obeying lots of orders from authority figures", etc., you can deliberately learn and practice each of them, and ramp up the difficulty more quickly, and probably reach a higher level in 8 months than most people reach in 12 years.  U.S. Army boot camp is apparently 10 weeks, for example.