LessWrong 2.0 Reader

So far Claude 3.7 is the only non-reasoning model I've tried that answers this correctly. All reasoning models did as well.

Consider a version of the monty hall problem where the host randomly picks which of the 2 remaining doors to open. It reveals a goat. What should you do?

If you're planning to actually do the experiments it suggests, or indeed act on any advice it gives in any way, then it's an agent.

Suppose you are  writing  a simulation. You keep optimizing it and hardcoding some stuff and handle different cases more efficiently and everything. One day your simulation becomes efficient enough so that you can run big enough grid for long enough, and there develops life. Then intelligent life. Then they tried to figure out the physics of their universe, and they succeed! But, oh wait, their description is extremely short but completely computationally intractable. 

Can you say that they actually figured out in what kind of universe they are in already, or should you wait for when they discover another million lines of code of optimization for it? Should you create giant sparkling letters "congratulations, you figured that out" or wait for more efficient formulation? 

The AI system accepts all previous feedback, but it may or may not trust anticipated future feedback. In particular, it should be trained not to trust feedback it would get by manipulating humans (so that it doesn't see itself as having an incentive to manipulate humans to give specific sorts of feedback).

I will call this property of feedback "legitimacy". The AI has a notion of when feedback is legitimate, and it needs to work to keep feedback legitimate (by not manipulating the human).

Legitimacy is good - but if an AI that's supposed to be intent-aligned to the user would find that it has an "incentive" to purposefully manipulate the user in order to get particular feedback from the user, unless it pretends that it would ignore that feedback, it's already misaligned and that misalignment should be dealt with directly IMO - this feels to me like a band-aid over a much more serious problem.

The mundane prompts were blocked 0% of the time. But you're right - we need something in between 'mundane and unrelated to bio research' and 'useful for bioweapons research'.

But I'm not sure what - here we are looking at lab wetwork ability. It seems that that ability is inherently dual-use.

Thanks!

Your write up was useful to me. 

I don’t think Tor scales in current form because it relies on altruistic donors to provide bandwidth. I agree there may be a way to scale it that doesn’t rely on altruism. 

I agree you’re pointing at an important problem. Namely when there’s a large structure aimed at achieving some task for users, and it deliberately does it poorly, some of our best solutions are to ensure  low cost-of-exit for users and allow for competing alternatives. 

This can be slow and wasteful as millions of people need to be fired, billions of dollars of equipment lost etc everytime a large company dies and is outcompeted. In the worst case this is entire countries and continents dying a slow death while their citizens are poached by other countries or left with an inferior quality of life. 

If there were incentives to fix large structures from the inside or alternatively, a way solve large tasks without requiring large top-down structures, that might improve the status quo. 

Would you invest your own money in such a project?

If I were a billionaire I might. 

I also have (maybe minor, maybe not minor) differences of opinion with standard EA decision-making procedures of assigning capital across opportunities. I think this is where our crux actually is, not on whether giant cities can be built with reasonable amounts of funding. 

And sorry I won’t be able to discuss that topic in detail further as it’s a different topic and will take a bunch of time and effort.  

I love this post.

1. Another important detail to track is what the leader says in private versus what they say in public. Typically you may want to first acquire data and attempt to trigger these cascades in private and in smaller groups, before you try triggering them across your nation or planet. 

2. I also think the Internet is going to shift these dynamics, by forcing private spheres of life to shrink or even become non-existent, and by increasing the number of events that are in public and therefore have potential to trigger these cascades. 

For example someone might just be experimenting with an unusual ideology or relationship or lifestyle in their private life, but if they and 50 others end up posting about this on YouTube, and a bunch of people end up reviewing it favourably, it could quickly become a worldwide phenomena practised by millions. 

A corollary is that if someone powerful wants their current ideas to not be threatened by competing ideas they may need to even more aggressively shut down anyone trying anything else in public, even if it’s only being tried at a small scale. Shutting down the more popular ideas might just mean more public attention gets freed up and directed to the less popular ideas. 

I am not sure a monopoly on violence is sufficient to shut down ideas in today’s world. People still smuggle hard drives and phones into North Korea for example. A real-time system to smuggle information could change the dynamics of military coups in the future. Note that this information could even include information about people’s preferences, who sides with who, trustworthy video proofs, leaks of private information of the elites etc. 

P.S. Shameless plug but I’m currently a full-time researcher trying to figure this stuff out. (Implications of the internet.) Feel free to read my website and reach out if interested. 

We'll likely record the lectures at least. The homework problems might be saved for future use, we'll see what makes sense.

Right now we have no concrete plans to run future iterations of this course. We have discussed recording and releasing the lectures however. Also, Vanessa is gauging interest for a summer mentorship program [LW · GW], which can be a good opportunity for a deeper dive into these topics.