steven0461's Shortform Feed

Suppose you have an AI powered world stabilization regime. Suppose somebody makes a reasonable moral argument about how humanity's reflection should proceed, like "it's unfair for me to have less influence just because I hate posting on Facebook". Does the world stabilization regime now add a Facebook compensation factor to the set of restrictions it enforces? If it does things like this all the time, doesn't the long reflection just amount to a stage performance of CEV with human actors? If it doesn't do things like this all the time, doesn't that create a serious risk of the long term future being stolen by some undesirable dynamic?

Inexorability of AI-enacted events doesn't intrude on decisions and discoveries of people written in those events. These decisions from the distant future may determine how the world preparing to reflect on them runs from the start.

How do they deal with model uncertainty (unknown unknowns)?

Additionally I think it's not real because if there would be such warming through feedback effects there's enough time to do heavy geoengineering. Geoengineering has it's own risks but it's doable in "runaway warming" scenarios.

I think the Metaculus crowd median is among the highest-quality predictions out there. Especially when someone goes through all the questions where they're confident the median is off, and makes comments pointing this out. I used to do this, some months back when there were more short term questions on Metaculus and more questions where I differed from the community. When you made a bunch of comments of this type a month back on Metaculus, that covered most of the 'holes', in my opinion, and now there are only a few questions where I differ from the median prediction.

Another source of predictions is from the IARPA Geoforecasting Challenge, where if you're competing you have access to hundreds of MTurk human predictions through an API. The quality of the predictions are not as great, and there are some questions where the MTurk crowd is way off. But they do have a question on whether Iran will execute or be targeted in a national military attack.

I agree that it's quite possible to beat the best publicly available forecasts. I've been wanting to work together on a small team to do this (where I imagine the same set of people would debate and make the predictions). If anyone's interested in this, I'm datscilly on Metaculus and can be reached at [my name] at gmail.

Maybe Good Judgement Open? I don't know how they actually get their probabilities though.

I think one could greatly outperform the best publicly available forecasts through collaboration between 1) some people good at arguing and looking for info and 2) someone good at evaluating arguments and aggregating evidence. Maybe just a forum thread where a moderator keeps a percentage estimate updated in the top post.

I would trust the aggregation algorithm on Metaculus more than an average (mostly because its performance is evaluated against an average). So I think that's usually pretty decent.

Sounds like failing at charity, not trying to figure out what thinking produced a claim/question/behavior and misinterpreting it as a result. In your example, there is an implication of difficulty with noticing the obvious, when the correct explanation is most likely having a different objective, which should be clear if the question is given half a thought. In some cases, running with the literal meaning of a claim as stated is actually a misinterpretation, since it differs from the intended meaning.

Another thing I feel like I see a lot on LW is disagreements where there's a heavy thumb of popularity or reputational costs on one side of the scale, but nobody talks about the thumb, and it makes it hard to tell if people are internally trying to correct for the thumb or if they're just substituting the thumb for whatever parts of their reasoning or intuition they're not explicitly talking about, and a lot of what looks like disagreement about the object level arguments that are being presented may actually be disagreement about the thumb. For example, in the case of the parent comment, maybe such a thumb is driving judgments of the relative values of oranges and pears.

I don't think there's a general solution. Eliezer's old quote "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." applies to social movements and discussion groups just as well. It doesn't matter if you're on the right or the wrong side - you have attention and resources that the war can use for something else.

There _may_ be an option to be on neither side, and just stay out. Most often, that's only available to those with no useful resources, or that can plausibly threaten both sides.

In most domains people don't make arguments because they either think they aren't strong or because making the argument would lose them social status.

The cases where an argument carries with it real danger are relatively small, and in most of those cases it should be possible to know that you are in a problematic area. In those cases, you should make arguments first nonpublically with people who you consider to be good judges of whether those arguments should be made publically.

I paint a stylized case of some type of situation where the question arises, and where my gut feeling tells me it may often be better to release the argument than to hide it, for the sake of long-term social cohesion and advancement:

You're part of an intellectual elite, with their own values/biases, and you consider hiding a sensible argument (say, on a political topic) because commoners, given their separate values/biases, would risk to act to it in a way that goes counter your agenda. You might likely not release the argument thus.

In the long-run this can backfire. The only way for society to advance is by reducing the gap between the elite and commoners. Commoners understand if they are regularly fed biased info by the elite; and the less seriously the elite engages, the less commoners will trust and be able to be hauled into more nuanced ways of thinking.

In short, in this stylized case: Intellectual honesty, even risking an immediate harm to your values, may likely enough pay in the long-term. Lifting the level of the discussion, by bringing up rational arguments for both 'sides', is important, especially in democratic systems where you eventually anyways rely on a common understanding of the world.

Maybe this does not generalize well, and my hunch is wrong: in the long-run we're all dead, and the level of public discussions is often so low that adding an argument in one direction is often just ammunition being exploited without impacting much along other dimensions.

There's still a big gap between Betfair/Smarkets (22% chance Trump president) and Predictit/FTX (29-30%). I assume it's not the kind of thing that can be just arbitraged away.

Huh. I am curious to hear explanations if anyone has one.

Nordhaus seems to miss the point here, indeed. He does statistics purely on historic macro-economic data. In these, there could not be even a hint of the singularity we'd here talk about - and that also he seems to refer to in his abstract (imho). This core singularity effect of self-accelerating, nearly infinitely fast intelligence improvement, once a threshold is crossed, is almost by definition invisible in present data: Only after this singularity, we expect things to get weird, and visible in economic data.

Bit sad to see the paper as is. Nordhaus has written seminal contributions in integrated environmental-economic modelling in the resources/climate domain. And also for singularity questions, good economic analysis modelling explicitly substitutabilities between different types of productive capital, resources, labor, information processing, could be insightful, I believe, and at least I have not yet stumbled upon much in that regard. There is a difficulty to imagine a post-singularity world at all; but interesting scenarios could probably be created, trying to formalize more casual discussions.

I haven't thought about it much, but it seems like the fraction of far future stuff isomorphic to the 21st century is probably fairly negligible from a purely utilitarian viewpoint, because the universe is so big that even using 1/1,000,000 of it for simulations would be a lot of simulations, and why would the far future want that many simulations of the 21st century? It doesn't seem like a good use of resources to do that many duplicate historical simulations in terms of either instrumental value or terminal value.