LessWrong 2.0 Reader

No current AI system could generate a research paper that would receive anything but the lowest possible score from each reviewer

Is it true in case of o3?

Looking like I'll not have figured this out before the time limit despite the extra time, what I have so far:

I feel like this is a bit incorrect. There are imaginable things that are smarter than humans at some tasks, smart as average humans at others, thus overall superhuman, yet controllable and therefore possible to integrate in an economy without immediately exploding into an utopian (or dystopian) singularity. The question is whether we are liable to build such things before we build the exploding singularity kind, or if the latter is in some sense easier to build and thus stumble upon first. Most AI optimists think these limited and controllable intelligences are the default natural outcome of our current trajectory and thus expect mere boosts in productivity.

We can leave theology. It is not so important. I am more concerned with the questions of finitism and infinitism in relation to paradox of sets.

Finitism is logically consistent. However, it seems to me that it suffers from the same problem as the ontological proof of the existence of God. It is an attempt to make a global prediction about the nature of the Universe based on a small thought experiment. Predictions like "Time cannot be infinite", "Space cannot be infinite" follow directly from finitism. It turns out that we make these predictions based on our mathematical problems with the paradox of sets. At the same time, the paradox of sets itself resembles the paradox "I'm telling a lie now". and, it seems, should look for a solution somewhere in the same area. If we think off the cuff, it seems to me naively that the very concept of "ordinary set" is composed in such a way as to lead to paradoxes. This is the problem of the concept of "ordinary set". This is not the problem of the existence/non-existence of physical infinity.

Oh, okay. I don't really understand this topic. But as far as I know, not all mathematicians are finitists. So it seems that the proofs of finitism are not flawless.

On the other hand, how is the problem of the set paradox solved in cosmological infinitism? Something like "The Infinite Universe may exist, but it is forbidden to talk about it as an object"? Because any attempt to do so will bring you back to the set paradox, if you take it seriously. "Talk about any particular part of the Universe as much as you like, but don't even think about the Universe as a whole"? This risks forming a somewhat patchwork model of the worldview. "It may exist, but you cannot think about it intelligently and rationally." One is reminded of Zeno's attempts to prove that one cannot think about motion without contradictions.

Thank you for your question!

I agree that the simulations need to have sufficient complexity. Indeed, that was one of main motivations I became interested in creating multi-objective benchmarks in the past. Various AI safety toy problems seemed to me so much simplified that they lacked essential objectives and other decisive nuances. This motivation is still very much one of my main driving motivations.

That being said, complexity has also downsides:
1) The complexity introduces confounding factors. When a model fails such a benchmark, it is not clear whether it was because it did not have required perceptual capabilities (so it is a capabilities problem), or it is using a model/framework that is unsuitable for alignment (so it is an alignment problem). 
2) Running the simulations will be more time consuming and it would make the research elitist in the sense that various people would not be able to afford it.

My plan is to try to start with preference towards simple, but not simpler than necessary. And then gradually make it more complex. That means trying to use the gridworlds and introducing as many symbols as is needed to represent the important objectives, objects, other concepts and phenomena, and their interactions.

I believe symbolic approaches should not be entirely dismissed. As a illustrative metaphor, I am thinking of books - they contains symbols, yet we consider them as a cornerstone of our civilization. Similarly to the current dilemma with benchmarks, we may then worry whether books are too simple and symbol based - or perhaps one should prefer watching movies instead, since they represent reality in more detail. But would that claim be necessarily true? It does not seem so obvious after all.

In case more complexity is needed, there are currently at least five ideas:
1) Adding more feature layers to the gridworld. I did not mention it before, but the observation format already supports multiple concurrent observable layers on top of each other. One of the layers could be for example facial expressions, or any other observable and unobservable metrics relevant to objects they accompany.
2) Adding textual messages between agents as a side panel to the gridworlds.
3) Making the environment bigger, so there are more objects and more phenomena.
4) Making the environment bigger and making also the objects bigger so that they cover multiple cells in the grid. Thus the objects will become composite, consisting of sub-parts with their own dynamics.
5) Using some other framework, for example Sims.

Curious, how do these thoughts and considerations land with you?

The value extractable is rent on both the land and the improvement. LVT taxes only the former. E.g. if land can earn $10k/month after an improvement of $1mm, and if interest is 4.5%, and if that improvement is optimal, a 100% LVT is not $10k/mo but $10k/mo minus $1mm*0.045/12=$3,750. So 100% LVT would be merely $6,250.

If your improvement can't extract $6.3k from the land, preventing you from investing in that improvement is a feature, not a bug.

Very powerful AIs may very well be created in order to defend the current capitalist system.

Like the most plausible proposal for what distinguishes bounded tool AI vs dangerous AI is that dangerous AI does adversarial/minimax-like reasoning whereas bounded tool AI mostly just assumes the world will allow it to do whatever it tries, so it doesn't need to try very hard.

This means the main people who will be forced to create dangerous AI are the ones working in hardcore adversarial contexts, which will especially be the military and police (as well as their opponents, including rogue states and gangsters). But the military and police have as their primary goal to maintain the current system.

I probably shouldnt have used the free energy terminology. Does complexity accuracy tradeoff work better ?

To be clear, I very much dont mean these things as a metaphor. I am thinking there may be an actual numerical complexity - accuracy that is some elaboration of Watanabe s "free energy" formula that actually describes these tendencies.

You can't sell the improvements if they're tied to land that is taxed higher than the improvements bring in (due to mistakes in improvement or changed environment that has increased the land value and the improvements haven't stayed optimal).    The land is taxed at it's full theoretical value, less than the improvements bring in, and the improvements are literally connected to it.

If you fail to pay the LVT you can presumably sell the improvements. I don't think there's an inefficiency here -- you shouldn't invest in improving land if you're not going to extract enough value from it to pay the LVT, and this is a feature, not a bug (that investment would be inefficient).