LessWrong 2.0 Reader

Humans face a version of this all the time - different contradictory wants with different timescales and impacts.  We don't have and certainly can't access a legible utility function, and it's unknown if any intelligent agent can (none of the early examples we have today can).

So the question as asked is either trivial (it'll depend on the willpower and rationality of the agent whether they optimize for the future or the present), or impossible (goals don't work that way).

Why SPY over QQQ?

One argument against is that I think it’s coming soon, and I have a 40 year history of frothing technological enthusiasm, often predicting things will arrive decades before they actually do. 😀

The aesthetics have even been considered carefully, although oddly this has not extended to dress (as far as I have seen).

I remember there being some dress instructions/suggestions for last years Bay solstice. I think we where told to dress in black, blue and gold.

These criticisms are often made of “market dominant minorities”, to use a sociologist’s term for what American Jews and Indian-Americans have in common. Here’s a good short article on the topic: https://scholarship.law.duke.edu/cgi/viewcontent.cgi?article=5582&context=faculty_scholarship

Randomly read this comment and I really enjoyed it, Turn it into a post? (I understand how annoying structuring complex thoughts coherently can be but maybe do a dialogue or something? I liked this.)

I largely agree with a lot of the missing things in people's views of utility functions and so I think you expressed some of that in a pretty good deeper way.

When we get into acausality and evertt branches I think we're going a bit off-track. I can think computational intractability and observer bias is something interesting to bring up but I always find it never leads anywhere. Quantum Mechanics is fundamentally observer invariant and so positing something like MWI is a philosophical stance (that is supported by occam's razor) but it is still observer dependent, what if there are no observers?

(Pointing at Physics as Information Processing)

Do you have any specific reason why you're going into QMech when talking about brain-like AGI stuff?

I have an AI agent that wrote myself; I use it on average 5x per week over the last 6 months. I think it's moderately useful. I mostly use it for simple shell tasks that would otherwise require copy-pasting back and forth with claude.ai.

My guess is that the big AI companies don't think the market for this is big enough to be worth making a product out of it.

I'm not confident one way or another.

I think my key crux is that in domains where there is a way to verify that the solution actually works, RL can scale to superhuman performance, and mathematics/programming are domains that are unusually easy to verify/gather training data for RL performance, so with caveats it can become rather good at those specific domains/benchmarks like millennium prize evals, but the important caveat is I don't believe this transfers very well to domains where verifying isn't easy, like creative writing.

I'm bearish on that. I expect GPT-4 to GPT-5 to be palatably less of a jump than GPT-3 to GPT-4, same way GPT-3 to GPT-4 was less of a jump than GPT-2 to GPT-3. I'm sure it'd show lower loss, and saturate some more benchmarks, and perhaps an o-series model based on it clears FrontierMath, and perhaps programmers and mathematicians would be able to use it in an ever-so-bigger number of cases...

I was talking about the 1 GW systems that would be developed in late 2026-early 2027, not GPT-5.

@johnswentworth [LW · GW] FWIW, GPQA Diamond seems much harder than GPQA main to me, and current models perform well on it. I suspect these models beat your performance on GPQA diamond if you're allowed 30 mins per problem. I wouldn't be shocked if you beat them, but that's because you're unusually broadly knowledgeable about science, not because you're smart.

I personally get wrecked by GPQA chemistry, get ~50% on GPQA biology if I have like 7 minutes per problem (which is notably better than their experts from other fields get, with much less time), and get like ~80% on GPQA physics with less than 5 minutes per problem. But GPQA Diamond seems much harder.

This isn’t crazy— people have tried related techniques.  But it needs more details thought out. 

In the chess example, the AIs start out very stupid, being wired at random.  But in a game between two idiots, moving at random, eventually someone is going to win.  And then you reinforce the techniques used by the winner, and de-reinforce the ones used by the loser.  In any encounter, you learn, regardless of who wins.  But in an encounter between a PM and a programmer, if the programmer fails, who gets reinforced?  It might be because the programmer is dumb, and should be de-reinforced.  But it might be because the PM is dumb, and asked for something impossible or far beyond what can be done, in which case it should be de-reinforced.  But it might be because the PM came up with a task just barely beyond the programmer’s ability, which is good and should be reinforced.  We somehow need to keep the PM producing problems which are hard but possible.  Maybe the programmer could be tasked with coming up with either a solution or a proof of impossibility?  

AlphaGo had a mechanism which tracked how important each move was.  It was trained to predict the probability that white would win, on each position encountered in the game.    Moves where this probability swung wildly were given a larger weight in reinforcement.  This was important for concentrating training on decisive moves, allowing the extraction of information from each move instead of each game. It’s not clear if this is possible in the programming task.