LessWrong 2.0 Reader

I added in two more tags (Redwood Research and AI Risk Skepticism), but I don't know how to properly format them. As it stands, there are extra lines before both of my entries. I am unsure of how to remove them.

Patenting an invention necessarily discloses the invention to the public. (In fact, incentivizing disclosure was the rationale for the creation of the patent system.) That is a little worrying because the main way the non-AI tech titans (large corporations) have protected themselves from patent suits has been to obtain their own patent portfolio, then entering into cross-licensing agreements with the holders of the other patent portfolios. Ergo, the patent-trolling project you propose could incentivize the major labs to disclose inventions that they are currently keeping secret, which of course would be bad.

Although patent trolling is a better use of money IMHO than most of things designed to reduce AI extinction risk that have been funded so far, lobbying the US and UK governments to nationalize their AI labs is probably better because a government program can be a lot more strict than private organizations can be in prohibiting researchers and engineers from sharing insights and knowledge outside the program. In particular, if the nationalized program is sufficiently like top-secret defense programs, long prison sentences are the penalty for a worker to share knowledge outside of the program.

Also, note that the patent troll needs to be well-funded (with a warchest of 100s of millions of dollars preferably) before it starts to influence the behavior of the AI labs although if it starts out with less and gets lucky, it might manage to get the needed warchest by winning court cases or arriving at settlements with the labs.

Math proofs are math proofs, whether they are in plain English or in Lean. Contemporary LLMs are very good at translation, not just between high-resource human languages but also between programming languages (transpiling), from code to human (documentation) and even from algorithms in scientific papers to code. Thus I wouldn't expect formalizing math proofs to be a hard problem in 2025.

However I generally agree with your line of thinking. As wassname wrote above (it's been quite obvious for some time but they link to a quantitative analysis), good in-silico verifiers are indeed crucial for inference-time scaling. However for the most of real-life tasks there's either no decent, objective verifiers in principle (e. g., nobody knows right answers to counterfactual economics or history questions) or there are very severe trade-offs in verifier accuracy and time/cost (think of wet lab life sciences: what's the point of getting hundreds AI predictions a day for cheap if one needs many months and much more money to verify them?)

I think that smartphones should be banned during school, but in the way that you can bring them to school, put them in some soundproof box, then take them out after the lessons are over. There is no reason to have a phone during the lessons, but coordinating with children in the afternoon is useful. Also, if my kids go to some activity right after school (without going home first), if they were not allowed to bring the phone to school, it means they also can't have it at the afternoon activity.

The hard problem is, how do you differentially get the screen time you want? At some point yes you want to impose a hard cap, but if I noticed my children doing hacking things, writing programs, messing with hardware, or playing games in a way that involved deliberate practice, or otherwise making good use, I would be totally fine with that up to many hours per day. The things they would naturally do with the screens? Largely not so much.

I think that if you teach your children how to use the computer while they are too young to install their own software, you will give them good habits. It is worse if they get their first computer advice from their classmates, because that will probably be about the things you don't want them to do.

My kids started at kindergarten age with Tux Paint, and later with other graphical programs (they both love to paint, both on paper and in computer). There was a time when they watched idiotic movies on YouTube, but then they stopped doing that (I am not sure why; maybe it simply got boring) and moved on to Pocket Platformer, Inkscape, and recently Scratch. They haven't discovered social networks yet.

With time, first we didn't have any limits, the kids got bored after a while. But then the time spent at the screen kept increasing, so we set a limit to one hour a day. That sometimes seemed like not enough, especially during weekends, so after a few experiments, we ended up with an "economy" where the kids get one hour a day for free, and can gain extra time by playing outside or helping at home.

I think my kids are probably on a good way to become hackers. I have friends who didn't let their kids use the computer until they were about nine years old (which is when they started having computer science lessons at school). Now, one friend's child only plays Minecraft, the other friend's child only plays Roblox, and there is no way to convince them to use the computer for any other purpose. Who knows, they may surprise me later.

When my kids are old enough to get smartphones, I probably won't allow them to install games or social networks there. I think it is better to do there things on the computer, because when you walk away from the computer it is over, and a turned-off computer cannot bother you with notifications.

Question: if I'm considering an isolated system (~= "the entire universe"), you say that I can swap between state-vector-format and matrix-format via

. But later, you say...

If $H_S$ is uncoupled to its environment (e.g. we are studying a carefully vacuum-isolated system), then we still have to replace the old state vector picture $|\varphi ⟩\in H$ by a (possibly rank $>1$) density matrix $\rho \in {\text{Op}}_H$...

But if Invalid LaTeX $\rho := \ket\phi \bra\phi: TeX parse error: Undefined control sequence \ket, how could it ever be rank>1?

(Perhaps more generally: what does it mean when a state is represented as a rank>1 density matrix? Or: given that the space of possible $\rho$s is much larger than the space of possible Invalid LaTeX $\ket\phi: TeX parse error: Undefined control sequence \kets, there are sometimes (always?) multiple $\rho$s that correspond to some particular Invalid LaTeX $\ket\phi: TeX parse error: Undefined control sequence \ket; what's the significance of choosing one versus another to represent your system's state?)

Also having the four suits each having a different multiplier might be fun?

Yeah, setups where (for instance) Clubs are worth 2x, Hearts and Diamonds are worth 1x, and Spades are worth 1/2x would (I expect) accelerate the effect. The example in Planecrash talks about multipliers like 1.3 or 1.1 where the evaluation is closer, which I turned to an integer multiplier to make the math doable in an average person's head. 

I have a more complicated and playtested version of Jellychip I mean to publish in a few days :)

Yep, that seems right and that does seem suboptimal. 

I think checking for escaping the island at the end of game would fix that since people still need to survive ten turns. Alternately, raising the amount of Boat needed would stretch that out, and more playtesting could figure out what the right target is.

. . . Hrm. What if escaped players still need food and water for the duration of the game, and then have to save up if they want to escape early? Not needing shelter gives a gentle encouragement to go as soon as they can.

Introspection is really interesting! This example where language models respond with the HELLO pattern (and can say they do so) is actually just one example of language models being able to articulate their implicit goals, and more generally to out-of-context reasoning. 

I'm glad my suggestion was helpful! 

(I continue to be quite unsure how to think about saving for retirement and kids college.)

In normal worlds, I think you are in excellent shape, with how your greater than $2 million net worth compares to median of around $100,000 and mean of around $800,000 net worth for households in their 40s in the US. Also, I think you have greater net worth than more than 99% of households in the world. If you let your taxable account go to zero, then you would likely have to pay less for college, because often the retirement accounts are not included in those calculations. If you didn't add anything more to your retirement account and it just grew 8% per year for twenty years, you would have ~$4 million. Then with the rule of thumb of drawing 4% per year during retirement, you would have an annual income of ~$160k. And that would not be counting any Social Security, income from home downsizing, inheritance, etc.
As for AGI worlds, as many have pointed out, we could be all dead or all rich, so it wouldn't matter. One person pointed out that wealth at the singularity might allow you to buy galaxies, but at least if you're altruistic, the impact of reducing existential risk is many orders of magnitude greater. Others have pointed out that even if humanity on average is rich, there may not be UBI. As long as one has significant net worth, this should grow at least initially, and then the cost of living should fall, so you should be fine. However, for people without any net worth, especially outside countries that benefit a lot from AI, there is reason to be worried. I personally think that even if there is not universal UBI, the rich of the world would not allow the poor to starve en masse. But if you want to do better than just not starving, then having a modest amount of net worth I think is quite prudent.

Would bet on this sort of strategy working; hard agree that ends don't justify the means and see that kind of justification for misinformation/propaganda a lot amongst highly political people. (But above examples are pretty tame.)