LessWrong 2.0 Reader

I think I'm confused here.
Is it fair to say that o3 does math and coding better than the average SWE?
If this is true, then I really don't understand why it hasn't made all the headlines. 
Any explanation?

Assuming positive defense/offense balance can be achieved in principle, what would an AGI-powered defense look like?

Just realized I forgot to mention this: I really like how the interactive handled the Bonus Objective, i.e. if the player is thinking along the right lines their character automatically makes the in-universe sensible/optimal decision for them (which means you can set up a fair Bonus Objective for players who don't live in that universe and so don't have all the context).

I was already sold on singularity. For what it's worth I found the post and comments very helpful for why you would want to take the sun apart in the first place and why it would be feasible and desirable for superintelligent and non-superintelligent civilization (Turning the sun into a smaller sun that doesn't explode seems nicer than having it explode. Fusion gives off way more energy than lifting the material. Gravity is the weakest of the 4 forces after all. In a superintelligent civilization with reversible computers, not taking apart the sun will make readily available mass a taut constraint).

If you can use 1kg of hydrogen to lift x>1kg of hydrogen using proton-proton fusion, you are getting exponential bulidup, limited only by "how many proton-proton reactors you can build in Solar system" and "how willing you are to actually build them", and you can use exponential buildup to create all necessary infrastructure.

Here is an example which I believe is directionally correct, it took me roughly 20 minutes to come up with it. The prompt is "how do living systems create meaning "?:

1. My life feels like it has meaning (sensory-motor behavior and conceptual intentional aspects). Looking at it through an evolutionary perspective, it is highly likely that meaning assignment is the way through which living systems survived. Thus, there has to be some base biological level at which meaning is created through cell-cell communication/ bioelectricity/ biochemistry /biosensoring etc.
2. Life is just made of atoms. Atoms are just automata. This implies, there is no meaning at the atom level and thus it cannot pop at a higher levels through emergence or some shit. You are delusional to believe there is some meaning assignment in life.
3. Meaning is something that is defined through the language that we speak. It is well known that different cultures have different words and conceptual framing which implies that meaning is different in different cultures. Meaning thus only depends on language.
4. Meaning is just a social construct and we can define anything to have meaning. Thus it doesn't matter what you find meaningful since it is just something you inherited through society and parenting.

I believe points 1-3 are fine, point 4 is kinda shaky.

1. Doesn't have to expend the energy. It's about reshaping the matter to machines. Computers take lots of mass-energy to constitute them, not to power them.
2. Things can go 6 orders of magnitude faster due to intelligence/agency, it's not highly unlikely in general.
3. I agree that in theory the arguments here could be better. It might require knowing more physics than I do, and has the "how does Kasparov beat you at chess" problem.

Thank you for putting this together.

Something I find interesting is that even many of the highest-profile skeptics of AI progress are surprisingly bullish (from an objective perspective).

For example, Yann LeCun has said we might get to AGI within a decade or two, and even Gary Marcus has gone on record saying "I do think we will eventually reach AGI (artificial general intelligence), and quite possibly before the end of this century."

"Before the end of this century" might seem pessimistic, but you'd think a true pessimist would say it will take centuries or millennia or even never happen at all. Almost no one seems to be saying that.

Management consulting firms have lots of great ideas on slide design: https://www.theanalystacademy.com/consulting-presentations/

Some things they do well:

• They treat slides as documents that can be understood standalone (this is even useful when presenting, as not everyone is following every word)
• They employ a lot of hierarchy to help make the content skimmable (helpful for efficiency)
• They put conclusions / summaries / action items up front, details behind (helpful for efficiency, especially in a high trust environments)

Additional thoughts:

• More than 3 bars/colors is fine
• I recommend using horizontal bars on some of those slides, so the labels are written in the same direction as the bars - lets you fill space more efficiently
• Put sentences / verbs in titles; noun titles like "Summary" or "Discussion" are low value
• If you're measuring deltas between two things, compute the error bar on the delta, don't compute the error bars on the two things; consider coloring by statistical significance (e.g., continuous color scale over range of standard errors of differences of the mean)
• In addition to agenda, it can be helpful to start with objectives - why are you here and what are you hoping to get from them? are you trying to inform them? get advice on something specific? get advice on something broad?
• Can help to include real data / real prompts / real model outputs - harder to fool yourself when you look at real data instead of relying on abstract metrics and intentions
• It's fine to have crummy slides - don't waste 1 hour of your time to save 5 minutes of your audience's time - the slides should serve you, not the other way around