LessWrong 2.0 Reader

As far as I understand from the article, the LLM generated five hypotheses that make sense. One of them is the one that the team has already verified but hadn’t yet published anywhere and another one the team hadn’t even thought of but consider worth investigating.

Assuming the five are a representative sample rather than a small human-curated set of many more hypotheses, I think that’s pretty impressive.

if the LLM generates enough hypotheses, and you already know the answer, one of them is likely to sound like the answer.

I don’t think this is true in general. Take any problem that is difficult to solve but easy to verify and you aren’t likely to have an LLM guess the answer.

To make an analogy, which involves accusations of baby killing in a completely unrelated context, imagine an abortion opponent who calls someone a baby killer in a context where it's not obvious he's talking about abortion. When questioned on this, he replies "I sincerely think abortion is baby killing, so my description was accurate, to the best of my knowledge". What's wrong with his reasoning?

What's wrong with his reasoning is that if you're trying to honestly and accurately communicate, you need to describe someone's position in uncontroversial terms. It doesn't matter whether you personally think abortion is baby killing; you're communicating with people who don't, and it's not your place to say "well, if I assume my side of this controversial issue is right, then my description is accurate". If you want your description to be accurate, describe it in ways that would give an accurate impression even to people who disagree with you.

"He thinks terrorists should have access... well, that's true because I personally think he has no good or principled way to exclude terrorists" is at worst just that kind of deception. (At best, of course, it's pedantry after all.)

i'm happy to grant that the 0.1% is just a fermi estimate and there's a +/- one OOM error bar around it. my point still basically stands even if it's 1%.

i think there are also many factors in the other direction that just make it really hard to say whether 0.1% is an under or overestimate.

for example, market capitalization is generally an overestimate of value when there are very large holders. tesla is also a bit of a meme stock so it's most likely trading above fundamental value.

my guess is most things sold to the public sector probably produce less economic value per $ than something sold to the private sector, so profit overestimates value produced

the sign on net economic value of his political advocacy seems very unclear to me. the answer depends strongly on some political beliefs that i don't feel like arguing out right now.

it slightly complicates my analogy for elon to be both the richest person in the us and also possibly the most influential (or one of). in my comment i am mostly referring to economic-elon. you are possibly making some arguments about influentialness in general. the problem is that influentialness is harder to estimate. also, if we're talking about influentialness in general, we don't get to use the 0.1% ownership of economic output as a lower bound of influentialness. owning x% of economic output doesn't automatically give you x% of influentialness. (i think the majority of other extremely rich people are not nearly as influential as elon per $)

That's a reasonable suspicion but as a counterpoint there might be more low-hanging fruit in biomedicine than math, precisely because it's harder to test ideas in the former. Without the need for expensive experiments, math has already been driven much deeper than other fields, and therefore requires a deeper understanding to have any hope of making novel progress.

edit: Also, if I recall correctly, the average IQ of mathematicians is higher than biologists, which is consistent with it being harder to make progress in math.

and yet, the richest person is still only responsible for 0.1%* of the economic output of the united states.

Musk only owns 0.1% of the economic output of the US but he is responsible for more than this, including large contributions to

• Politics
• Space
  • SpaceX is nearly 90% of global upmass
  • Dragon is the sole American spacecraft that can launch humans to ISS
  • Starlink probably enables far more economic activity than its revenue
  • Quality and quantity of US spy satellites (Starshield has ~tripled NRO satellite mass)
• Startup culture through the many startups from ex-SpaceX employees
• Twitter as a medium of discourse, though this didn't change much
• Electric cars probably sped up by ~1 year by Tesla, which still owns over half the nation's charging infrastructure
• AI, including medium-sized effects on OpenAI and potential future effects through xAI

Depending on your reckoning I wouldn't be surprised if Elon's influence added up to >1% of Americans combined. This is not really surprising because a Zipfian relationship would give the top person in a nation of 300 million 5% of the total influence.

you might expect that the butterfly effect applies to ML training. make one small change early in training and it might cascade to change the training process in huge ways.

at least in non-RL training, this intuition seems to be basically wrong. you can do some pretty crazy things to the training process without really affecting macroscopic properties of the model (e.g loss). one very well known example is that using mixed precision training results in training curves that are basically identical to full precision training, even though you're throwing out a ton of bits of precision on every step.

On the contrary, it seems like this is the area where we should be able to best apply RL, since there is a clear reward signal. 

Is there? It's one thing to verify whether a proof is correct; whether an expression (posed by a human!) is tautologous to a different expression (also posed by a human!). But what's the ground-truth signal for "the framework of Bayesian probability/category theory is genuinely practically useful"?

This is the reason I'm bearish on the reasoning models even for math. The realistic benefits of them seem to be:

1. Much faster feedback loops on mathematical conjectures.
2. Solving long-standing mathematical challenges such as Riemann's or P vs. NP.
3. Mathematicians might be able to find hints of whole new math paradigms in the proofs for the long-standing challenges the models generate.

Of those:

• (1) still requires mathematicians to figure out which conjectures are useful. It compresses hours, days, weeks, or months (depending on how well it scales) of a very specific and niche type of work into minutes, which is cool, but not Singularity-tier.
• (2) is very speculative. It's basically "compresses decades of work into minutes", while the current crop of reasoning models can barely solve problems that ought to be pretty "shallow" [LW(p) · GW(p)] from their perspective. Maybe Altman is right, the paradigm is in its GPT-2 stage, and we're all about to be blown away by what they're capable of. Or maybe it doesn't scale past the frontier of human mathematical knowledge very well at all, and the parallels with AlphaZero are overstated. We'll see.
• (3) is dependent on (2) working out.

(The reasoning-model hype is so confusing for me. Superficially there's a ton of potential, but I don't think there's been any real indication they're up to the real challenges still ahead.)

Thanks so much for your thoughts Dave. 

I agree humans have always been misaligned, and that in many ways we have made significant advancements in alignment over long time frames. However, I think few would deny that any metric approximating alignment would be quite volatile over shorter time frames or specific populations, which creates periods of greater risk.

I agree that there must be something new that increases that existential risk to justify significant concern. You identified bioweapons as one example, which I agree is a risk, but not the specific one I am concerned about. 

The new factors I am concerned about are:

• The vastly increased ability of small misaligned actors to significantly alter, manipulate, or undermine humans' capacities for alignment. This seems largely tied to social media. As evidence, I would point to the sharp increase in social divisions in the US in recent years.
• The introduction of AI that allows individuals to project their misaligned will and power without having to involve or persuade other individuals who previously would have exerted some degree of influence toward realignment 

It seems to be putting the cart before the horse to be spending so much time, money, effort, and thought on AI Alignment, while our alignment as humans is so poor. In my mind, understanding the nature and roots of our misalignment, and identifying how to use technology to increase our alignment rather than undermine it, seems to me to be an obvious prerequisite to being able to trust ourselves to use powerful AI in ways that don't decrease alignment. While recent years may have presented conditions that were especially effective at exploiting vulnerabilities in our capacities for maintaining alignment, those vulnerabilities have always been and always will be a risk, so we will always be the weakest link in the Alignment equation until we put serious effort into elevating ourselves to the same standards we expect to hold AI to.

This is one of the arguments in favor of the AGI project.

Could you please explain the argument in favor of the AGI project that you had in mind here, so I don't risk making incorrect assumptions? I apologize I'm not as familiar with other perspectives as I'd like to be yet. Also, I'd love to hear your take on my additional thoughts. Thank you for engaging, I find the dialogue very helpful.

I'm talking about this graph:

What are the calculations used for this graph. Text says to see the appendix but the appendix does not actually explain how you got this graph.

I think a lot of such people can be "cured" by high certainty of being caught, not by severity of punishment

This stems from a misunderstanding of how the career-criminal mind works. They don't really care about being caught. They remember how out of the last 40 or so times they walked into Walmart and left with ~$100 in unpaid merchandise they only got caught half the time and the other half of the time they got let off with time served of 10-20 days. Either they get away with it or they gotta wait a couple weeks before they get to try again. Not a big deal either way.

So much of the crime plaguing modern America is open and obvious and even caught on camera. It's just that the criminal justice system refuses to punish repeat petty offenders.

What punishment do you think someone who has been convicted of stealing 15 times before should get on his 16th conviction?