LessWrong 2.0 Reader

I agree that this is plausibly a real important difference, but I do not think that it is obvious.

The most recent augmentative technological change was the industrial revolution. It has reshaped virtually every every activity. It allowed for the majority of the population to not work in agriculture for the first time since the agricultural revolution.

The industrial revolution centered on energy. Having much cheaper, much more abundant energy allowed humans to use that energy for all sorts of things. 

If fusion ends up being similar in cost to existing electricity production, it will be a substitutional technology. This is the thing that we are working on now (well, also making it work at all). People who work in fusion focus on this because it is the reasonable near/medium term projection. If fusion ends up being substantially cheaper, it will be an augmentative technology. It is not at all clear that this will happen, because we can't know how the costs will change between the first and thousandth fusion power plant.

Notably, we don't know if foom is going to be a thing either.

The narrative around the technology is at least as important as what has happened in the technology itself. The fusion community could frequently talk about how incredible the industrial revolution was, and how it powered Britain to global dominance for two centuries. A new source of energy might do the same thing ! But this is more hype than we feel we ought to offer, and the community's goal is not to create a dominant superpower.

Even if foom is going to happen, things would look very different if the leaders credibly committed to helping others foom if they are first. I don't know if this would be better or worse from a existential risk perspective, but it would change the nature of the race a lot.

I want to be able to describe agents that do not have (vNM, geometric, other) rational preferences because of incompleteness or inconsistency but self-modify to become so. 

Eg. In vNM utility theory there is a fairly natural weakening one can do which is ask for a vNM-style representation theorem after dropping transitivity.

[ Incidentally, there is some interesting math here having to do with conservative vs nonconservative vector fields and potentials theory all the way to hodge theory. ]

does JB support this ? 

Im confused since in vNM we start with a preference order over probability distributions. But in JB irs over propositions?

I agree, but it is important to note that the authors of the paper disagree here.

(It's somewhat hard for me to tell if the crux is more that they don't expect that everyone would get AI aligned to them (at least as representatives) even if this was technical feasible with zero alignment tax or if the crux is that even if everyone had single-single aligned corrigible AIs representing their interests and with control over their assets and power that would still result in disempowerment. I think it is more like second thing here.)

So Zvi is accurately representing the perspective of the authors, I just disagree with them.

I do think basically none of this makes sense if you don't have some particular flavor of ambitious goals. If you don't have ambitious goals that (probably) depend on leveling up a bunch, then yeah, don't do that. (unless you intrinsically value the leveling up)

If you sort-of-kind-of have ambitious goals, but you're not totally bought into them, or have mixed feelings about them, then you probably want to somehow resolve that (which I don't think is a "deliberate practice" shaped problem, more like a therapy/emotional processing problem, or a "just find a fun project or better work environment that makes things more naturally motivating" problem)

Trump, when asked whether DeepSeek posed a security risk, says "No. … The AI we're talking about will be a lot less expensive than people originally thought. That's a good thing. I view that as a very good development, not a bad development." – link.

It might be at this point just an underinvestment in robotics, compared to other AI.

Admittedly, Gato didn't have positive transfer, unlike all the other robotic elements.

The first thing that comes up when I look at this is I'm not that sure what your goals are, and I'm not sure whether the sort of thing I'm getting at in this post is an appropriate tool. 

You say:

I suspect that if I set myself a weekly challenge of editing a sequence from found footage that pertained to a pseudo-random topic of theme that this might possibly pay dividends in terms that generalize to metacognition.

This sounds like you're seeing the metacognition as more like a terminal goal, than an instrumental goal (which I think doesn't necessarily make sense).

I do think metacognition is generally useful, but in an established domain like video-editing or self-promotion in a fairly understood field, there are probably object-level skills you can learn that pay off faster than metacognition. (Most of the point of metacognition there is to sift out the "good" advice from the bad).

I want to separate out...

• purposefully practicing metacognition
• purposefully practicing particular object level skills, such as videoediting or self-promotion (which involves figuring out what the subskills are that you can get quicker feedback on)
• purposefully practice "purposeful practice", such that you get better at identifying subskills in various (not-necessarily-metacognition-y) domains.

...as three different things that might (or might not) be the right thing for you.

Right now I can't really tell what your goal is, so I would first just ask "what is it you are trying to achieve?" 1-3 years from now, how would you know if [whatever kind of practice you did] turned out to work? (I think it's helpful to imagine "what would an outside observe watching a video recording see happening differently")

I find the pessimistic interpretation of the results a bit odd given considerations like those in https://www.lesswrong.com/posts/i2nmBfCXnadeGmhzW/catching-ais-red-handed [LW · GW]. 

with a bunch of reflexes to eg stop and say “that doesn’t sound right” or “I think I’ve gone wrong, let’s backtrack and try another path”

Shannon Sands says he's found a backtracking vector in R1:

https://x.com/chrisbarber/status/1885047105741611507

Apropos of very low-latency LLMs and revisiting this topic a little: what does this imply about DRL robotics, rather than animals? Will DRL NNs have to have brains as big as humans in order to run superhuman humanoid robots?

One possible implication is that Portia-like NNs are possible for robotics in general. Robotics may be quite 'easy' in that sense.

It is striking that when we look at NN parameter/FLOPS-counts, we generally do not see 'large' robotics, vision, or sound models, but LLMs; the largest pure-vision models like PaLI-X are <100b-parameters, the largest robotics are usually <10b, with Gato 1's ~1b having been, if anything, unusually large because of all the other stuff it was doing. (I'm very behind on the robotics literature so maybe there are now much larger 100b-parameter models as they move into the 'foundation model' multi-modal/task scaling paradigm, but I'd bet that there still are none >1,000b.) Even sound/image/video generative models, which would be expected to be much larger than necessary for robotics tasks, are often small enough to run on a single consumer GPU, still. And these are usually trained with scaling laws now, so these are compute-optimal sizes and it is not just that they are wildly under-parameterized (the way almost all models were pre-2020).

So, if robotics is intrinsically easy, but animal brains do not show this because of their latency requirements, which forces them into misleadingly expensive brains, the implication is that we can do robotics by lifting the limitations of biological brains, like being forced to learn in realtime, in the real world, one animal at a time, without any sharing.

We should be able to train deep but small NNs in silico: turning all animal problems into Portia problems, if you will, pausing the simulation to let the NNs think & act for as long as necessary to plan the right action, and only then letting time flow to see what happens, and reset it to try again.

We remove all burdens of wallclock time or caloric consumption or childhood development, which are powerful general robotic controllers, and only then use these teacher-models to optimize low-latency controllers. The wider low-latency student models will be easier to train when they simply must imitate the teacher in a supervised-learning setting instead of RL from scratch, and so the size should be a lot better. (If nothing else, the student models can't 'die' if they make a mistake like breaking a latency constraint, so this learning setting is way easier than an animal's task.)

On a related note, it is also striking how far down in size LLMs can be pushed. You can get good reasoning out of tiny billion-parameter LLMs trained hard enough on high-quality-enough data, and the 'densifying experience curve' is steady and rapid (halving period of ~4 months), so we can expect that at some point we may have superhuman reasoning LLMs in the billion or sub-billion parameter range... which are just very, very ignorant, perhaps even more ignorant than you or me, of all the real-world knowledge & text that a proper LLM has. We can't train those from scratch, but we can train trillion-parameter LLMs to suck in all the text in the world, and then exhale training data for small fast cheap models [LW · GW].

So it seems that Moravec's Paradox remains undefeated: as difficult as we find the abstract intellectual capabilities like the process of doing math or reasoning, so difficult we struggle to even write them down to train LLMs on, they are not intrinsically difficult and do not require big expensive NNs.