LessWrong 2.0 Reader

I’m trying to train my brain to answer the question "what am I trying to do?" before I unlock my phone. If I can't answer the question, maybe that "mental speed bump" makes it easier to put my phone back down.

Hello Everyone!

It has been a while since I posted around here, but I was happy to check things out. It seems like things have gone quiet around here. Which is sad. There have been so many positive discussions around here. One of the gems I have found, as of late, is a great place called Planksip.org. Planksip is a sort of new online home for philosophy (for those who are into that sort of thing), and they are having some interesting discussions. For those who write philosophy or want to publish a book, they do offer that as a service. If you are interested in writing and being seen on philosophy topics, they have a pretty good email list. They publish on the Ghost platform (which is rather new) so it makes it really easy.  I would encourage everyone to stop on by Planksip, and check it out if you want to write/publish more on philosophy. 

As a non-physicist I kind of had the idea that the reason I was taught Newtonian mechanics in high school was that it was assumed I wasn't going to have the time, motivation, or brainpower to learn some kind of fancy, real university version of it, so the alternate idea that it's useful for intuition-building of the concepts is novel and interesting to me.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?

(I'm the OP)

I'm not trying to say "it's bad to large sums of money to any group because humans have a tendency to to seek power." 

I'm saying "you should be exceptionally cautious about giving large sums of money to a group of humans with the stated goal of constructing an AGI."

You need to weight any reassurances they give you against two observations:

1. The commonly observed pattern of individual humans or organisations seeking power (and/or wealth) at the expense of the wider community. 
2. The strong likelihood that there will be an opportunity for organisations pushing ahead with AI research to obtain incredible wealth or power.

So, I'm not trying to say "humans seek power therefore giving any group of humans money is bad". 

It's that "humans seek power" and, in the specific case of AI companies, there may be incredibly strong rewards for groups that behave in a self-interested way.

The general idea I'm working off is that you need to be exceptionally skeptical of seemingly altruistic statements and commitments made by humans when there are exceptionally lucrative incentives to break these commitments at a later point in time (and limited ways to enforce the original commitment).

"In particular, it emphasized the importance of distributing AI broadly;¹ our current view is that this may turn out to be a promising strategy for reducing potential risks"

Yes, I'm interpreting the phrase "may turn out" to be treating the idea with more seriousness than it deserves. 

Rereading the paragraph, it seems reasonable to interpret it as politely downplaying it, in which case my statement about Open Phil taking the idea seriously is incorrect.

Well, I guess describing a model of a computably enumerable theory, like PA or ZFC counts. We could also ask for a model of PA that's nonstandard in a particular way that we want, e.g. by asking for a model of $PA+\neg Con\left(PA\right)$, and that works the same way. Describing a reflective oracle has low solutions too, though this is pretty similar to the consistent guessing problem. Another one, which is really just a restatement of the low basis theorem, but perhaps a more evocative one, is as follows. Suppose some oracle machine $T$ has the property that there is some oracle that would cause it to run forever starting from an empty tape. Then, there is a low such oracle.

(Technically, these aren't decision problems, since they don't tell us what the right decision is, but just give us conditions that whatever decisions we make have to satisfy. I don't know what to say instead; this is more general then e.g. promise problems. Maybe I'd use something like decision-class problems?)

All these have a somewhat similar flavour by the nature of the low basis theorem. We can enumerate a set of constraints, but we can't necessarily compute a single object satisfying all the constraints. But the theorem tells us that there's a low such object.

I don't know what the situation is for subsets of the digits of Chaitin's constant. Can it be as hard as the halting problem? You might try to refute this using some sort of incompressibility idea. Can it be low? I'd expect not, at least for computable subsets of indices of positive density. Plausibly computability theorists know about this stuff. They do like constructing posets of Turing degrees of various shapes, and they know about which shapes can be realized between $0$ and the halting degree $0^{\prime }$. (E.g. this paper.)

That is great to hear, but I find it probable they’ll be ignored/lobbied against/gamed when it goes against business interests.

"and seek amateur advice"

well said!

Yeah, there's a sort of trick here. The natural question is uniform--we want a single reduction that can work from any consistent guessing oracle, and we think it would be cheating to do different things with different oracles. But this doesn't matter for the solution, since we produce a single consistent guessing oracle that can't be reduced to the halting problem.

This reminds me of the theory of enumeration degrees, a generalization of Turing degrees allowing open-set-flavoured behaviour like we see in partial recursive functions; if the answer to an oracle query is positive, the oracle must eventually tell you, but if the answer is negative it keeps you waiting indefinitely. I find the theory of enumeration degrees to be underemphasized in discussion of computability theory, but e.g. Odifreddi has a chapter on it all the way at the end of Classical Recursion Theory Volume II.

The consistent guessing problem isn't a problem about enumeration degrees. It's using a stronger kind of uniformity--we want to be uniform over oracles that differently guess consistently, not over a set of ways to give the some answers, but to present them differently. But there is again a kind of strangeness in the behaviour of uniformity, since we get equivalent notions if we do or do not ask that a reduction between sets $A$, $B$ be a single function that uniformly enumerates $A$ from enumerations of $B$, so there might be some common idea here. More generally, enumeration degrees feel like they let us express more naturally things that are a bit awkward to say in terms of Turing degrees--it's natural to think about the set of computations that are enumerable/${\Sigma }_1$ in a set--so it might be a useful keyword.