Posts
Comments
No I don't think so because people could just airgap the GPUs.
Weaker AI probably wouldn't be sufficient to carry out an actually pivotal act. For example the GPU virus would probably be worked around soon after deployment, via airgapping GPUs, developing software countermeasures, or just resetting infected GPUs.
This discussion is a nice illustration of why x-riskers are definitely more power-seeking than the average activist group. Just like Eskimos proverbially have 50 words for snow, AI-risk-reducers need at least 50 terms for "taking over the world" to demarcate the range of possible scenarios. ;)
Nice overview, I agree but I think the 2016-2021 plan could still arguably be described as "obtain god-like AI and use it to take over the world"(admittedly with some rhetorical exaggeration, but like, not that much)
I would be happy to take bets here about what people would say.
Sure, I DM'd you.
I think making inferences from that to modern MIRI is about as confused as making inferences from people's high-school essays about what they will do when they become president
Yeah, but it's not just the old MIRI views, but those in combination with their statements about what one might do with powerful AI, the telegraphed omissions in those statements, and other public parts of their worldview e.g. regarding the competence of the rest of the world. I get the pretty strong impression that "a small group of people with overwhelming hard power" was the ideal goal, and that this would ideally be controlled by MIRI or by a small group of people handpicked by them.
I think they talked explicitly about planning to deploy the AI themselves back in the early days(2004-ish) then gradually transitioned to talking generally about what someone with a powerful AI could do.
But I strongly suspect that in the event that they were the first to obtain powerful AI, they would deploy it themselves or perhaps give it to handpicked successors. Given Eliezer's worldview I don't think it would make much sense for them to give the AI to the US government(considered incompetent) or AI labs(negligently reckless)
It wasn't specified but I think they strongly implied it would be that or something equivalently coercive. The "melting GPUs" plan was explicitly not a pivotal act but rather something with the required level of difficulty, and it was implied that the actual pivotal act would be something further outside the political Overton window. When you consider the ways "melting GPUs" would be insufficient a plan like this is the natural conclusion.
doesn't require replacing existing governments
I don't think you would need to replace existing governments. Just block all AI projects and maintain your ability to continue doing so in the future via maintaining military supremacy. Get existing governments to help you, or at least not interfere, via some mix of coercion and trade. Sort of a feudal arrangement with a minimalist central power.
"Taking over" something does not imply that you are going to use your authority in a tyrannical fashion. People can obtain control over organizations and places and govern with a light or even barely-existent touch, it happens all the time.
Would you accept "they plan to use extremely powerful AI to institute a minimalist, AI-enabled world government focused on preventing the development of other AI systems" as a summary? Like sure, "they want to take over the world" as a gist of that does have a bit of an editorial slant, but not that much of one. I think that my original comment would be perceived as much less misleading by the majority of the world's population than "they just want to do some helpful math uwu" in the event that these plans actually succeeded. I also think it's obvious that these plans indicate a far higher degree of power-seeking(in aim at least) than virtually all other charitable organizations.
(..and to reiterate, I'm not taking a strong stance on the advisability of these plans. In a way, had they succeeded, that would have provided a strong justification for their necessity. I just think it's absurd to say that the organization making them is less power-seeking than the ADL or whatever)
Are you saying that AIS movement is more power-seeking than environmentalist movement that spent 30M$+[...]
I think that AIS lobbying is likely to have more consequential and enduring effects on the world than environmental lobbying regardless of the absolute size in body count or amount of money, so yes.
"MIRI default plan" was "to do math in hope that some of this math will turn out to be useful".
I mean yeah, that is a better description of their publicly-known day-to-day actions, but intention also matters. They settled on math after it became clear that the god AI plan was not achievable(and recently, gave up on the math plan too when it became clear that was not realistic). An analogy might be an environmental group that planned to end pollution by bio-engineering a microbe to spread throughout the world that made oil production impossible, then reluctantly settled for lobbying once they realized they couldn't actually make the microbe. I think this would be a pretty unusually power-seeking plan for an environmental group!
Are you sure [...] et cetera are less power-seeking than AI Safety community?
Until recently the MIRI default plan was basically "obtain god-like AI and use it to take over the world"("pivotal act"), it's hard to get more power-seeking than that. Other wings of the community have been more circumspect but also more active in things like founding AI labs, influencing government policy, etc., to the tune of many billions of dollars worth of total influence. Not saying this is necessarily wrong but it does seem empirically clear that AI-risk-avoiders are more power-seeking than most movements.
let’s ensure that AGI corporations have management that is not completely blind to alignment problem
Seems like this is already the case.
Possibly related: Stochastic Collapse: How Gradient Noise Attracts SGD towards sparser subnetworks"
Seconded.
Makes sense. But I think the OP is using the term to mean something different than you(centrally math and puzzle solving)
Hmm, but don't puzzle games and math fit those criteria pretty well?(I guess if you're really trying hard at either there's more legitimate contact with reality?) What would you consider a central example of a nerdy interest?
I wonder if "brains" of the sort that are useful for math and programming are neccessarily all that helpful here. I think intuition-guided trial and error might work better. That's been my experience dealing with chronic-illness type stuff.
I think she meant he was looking for epistemic authority figures to defer to more broadly, even if it wasn't because he thought they were better at math than him.
Some advanced meditators report that they do perceive experience as being basically discrete, flickering in and out of existence at a very high frequency(which is why it might appear continuous without sufficient attention). See e.g. https://www.mctb.org/mctb2/table-of-contents/part-i-the-fundamentals/5-the-three-characteristics/
Tangentially related: some advanced meditators report that their sense that perception has a center vanishes at a certain point along the meditative path, and this is associated with a reduction in suffering.
performance gap of trans women over women
The post is about the performance gap of trans women over men, not women.
I don't know enough about hormonal biology to guess a specific cause(some general factor of neoteny, perhaps??). It's much easier to infer that it's likely some third factor than to know exactly what third factor it is. I actually think most of the evidence in this very post supports the 3rd-factor position or is equivocal - testosterone acting as a nootropic is very weird if it makes you dumber, that men and women have equal IQs seems not to be true, the study cited to support a U-shaped relationship seems flimsy, that most of the ostensible damage occurs before adulthood seems in tension with your smarter friends transitioning after high school.
I buy that trans women are smart but I doubt "testosterone makes you dumber" is the explanation, more likely some 3rd factor raises IQ and lowers testosterone.
I think using the universal prior again is more natural. It's simpler to use the same complexity metric for everything; it's more consistent with Solomonoff induction, in that the weight assigned by Solomonoff induction to a given (world, claw) pair would be approximately the sum of their Kolmogorov complexities; and the universal prior dominates the inverse square measure but the converse doesn't hold.
If you want to pick out locations within some particular computation, you can just use the universal prior again, applied to indices to parts of the computation.
If you're running on the non-time-penalized solomonoff prior[...]a bunch of things break including anthropic probabilities and expected utility calculations
This isn't true, you can get perfectly fine probabilities and expected utilities from ordinary Solmonoff induction(barring computability issues, ofc). The key here is that SI is defined in terms of a prefix-free UTM whose set of valid programs forms a prefix-free code, which automatically grants probabilities adding up to less than 1, etc. This issue is often glossed over in popular accounts.
certain aspects of MWI theory (like how you actually get the Born probabilities) are unresolved
You can add the Born probabilities in with minimal additional Kolmogorov complexity, simply stipulate that worlds with a given amplitude have probabilities given by the Born rule(this does admittedly weaken the "randomness emerges from indexical uncertainty" aspect...)
Having briefly looked into complexity science myself, I came to similar conclusions -- mostly a random hodgepodge of various fields in a sort of impressionistic tableau, plus an unsystematic attempt at studying questions of agency and self-reference.
That is, I think humans generally (though not always) attempt to avoid death when credibly threatened, even when they're involved in a secret conspiracy to overthrow the government.
This seems like a misleading comparison, because human conspiracies usually don't try to convince the government that they're perfectly obedient slaves even unto death, because everyone already knows that humans aren't actually like that. If we imagine a human conspiracy where there is some sort of widespread deception like this, it seems more plausible that they would try to continue to be deceptive even in the face of death(like, maybe, uh, some group of people are pretending to be fervently religious and have no fear of death, or something)
Statements can be epistemically legit or not. Statements have content, they aren't just levers for influencing the world.
I mean it's epistemically legitimate for him to bring them up. They are in fact evidence that Scott holds hereditarian views.
Now, regarding the "overall" legitimacy of calling attention to someone's controversial views, it probably does have a chilling effect, and threatens Scott's livelihood which I don't like. But I think that continuing to be mad at Metz for his sloppy inference doesn't really make sense here. Sure, maybe at the time it was tactically smart to feign outrage that Metz would dare to imply Scott was a hereditarian, but now that we have direct documentation of Scott admitting exactly that, it's just silly. If you're still worried about Scott getting canceled (seems unlikely at this stage tbh) it's better to just move on and stop drawing attention to the issue by bringing it up over and over.
But was Metz acting as a "prosecutor" here? He didn't say "this proves Scott is a hereditarian" or whatever, he just brings up two instances where Scott said things in a way that might lead people to make certain inferences....correct inferences, as it turns out. Like yeah, maybe it would have been more epistemically scrupulous if he said "these articles represent two instances of a larger pattern which is strong Bayesian evidence even though they are not highly convincing on their own" but I hardly think this warrants remaining outraged years after the fact.
How is Metz's behavior here worse than Scott's own behavior defending himself? After all, Metz doesn't explicitly say that Scott believes in racial iq differences, he just mentions Scott's endorsement of Murray in one post and his account of Murray's beliefs in another, in a way that suggests a connection. Similarly, Scott doesn't explicitly deny believing in racial iq differences in his response post, he just lays out the context of the posts in a way that suggests that the accusation is baseless(perhaps you think Scott's behavior is locally better? But he's following a strategy of covertly communicating his true beliefs while making any individual instance look plausibly deniable, so he's kinda optimizing against "locally good behavior" tracking truth here, so it seems perverse to give him credit for this)
"For my friends, charitability -- for my enemies, Bayes Rule"
ZMD: Looking at “Silicon Valley’s Safe Space”, I don’t think it was a good article. Specifically, you wrote,
In one post, [Alexander] aligned himself with Charles Murray, who proposed a link between race and I.Q. in “The Bell Curve.” In another, he pointed out that Mr. Murray believes Black people “are genetically less intelligent than white people.”
End quote. So, the problem with this is that the specific post in which Alexander aligned himself with Murray was not talking about race. It was specifically talking about whether specific programs to alleviate poverty will actually work or not.
So on the one hand, this particular paragraph does seem like it's misleadingly implying Scott was endorsing views on race/iq similar to Murray's even though, based on the quoted passages alone, there is little reason to think that. On the other hand, it's totally true that Scott was running a strategy of bringing up or "arguing" with hereditarians with the goal of broadly promoting those views in the rationalist community, without directly being seen to endorse them. So I think it's actually pretty legitimate for Metz to bring up incidents like this or the Xenosystems link in the blogroll. Scott was basically using a strategy of communicating his views in a plausibly deniable way by saying many little things which are more likely if he was a secret hereditarian, but any individual instance of which is not so damning. So I feel it's total BS to then complain about how tenuous the individual instances Metz brought up are -- he's using it as an example or a larger trend, which is inevitable given the strategy Scott was using.
(This is not to say that I think Scott should be "canceled" for these views or whatever, not at all, but at this stage the threat of cancelation seems to have passed and we can at least be honest about what actually happened)
This seems significantly overstated. Most subjects are not taught in school to most people, but they don't thereby degrade into nonsense.
Why should Michael Burry have assumed that he had more insight about Avant! Corporation than the people trading with him?
Because he did a lot of research and "knew more about the Avant! Corporation than any man on earth"? If you have good reason to think that you're the one with an information advantage trades like this can be rational. Of course it's always possible to be wrong about that, but there are enough irrational traders out there that it's not ruled out. Also note that it's not actually needed that your counterparties are irrational on average, it's enough that there are irrational traders somewhere in the broader ecosystem, as they can "subsidize" moderately-informed trading by others(which you can take advantage of in individual cases)
An amended slogan that more accurately captures the phenomenon the post is trying to point to would be "Conditional on your trade seemingly not creating value for your counterparty, your trade likely wasn't all that good".
Not sure how much I believe this myself, but Jacob cannell has an interesting take that social status isn't a "base drive" either, but is basically a proxy for "empowerment", influence over future states of the world. If that's true it's perhaps not so surprising that we're still well-aligned, since "empowerment" is in some sense always being selected for by reality.
I would tell the students that any compactly specified model has to rely on a certain amount of "common-sensical" interpretation on their part, such that they need to evaluate what "counts" as a legitimate application of the theory and what does not. I'd argue this by analogy to their daily lives where interpretation of this sort is constantly needed to make sense of basic statements. Abstractly, this arises due to reality having a lot of detail which needs to be dynamically interpreted by a large parallel model like their brain and can't be handled by a very compact equation or statement, so they need to act as a "bridge" between the compact thing and actual experiments. (Indeed, this sort of interpretation is so ubiquitous that real students would almost never make this kind of mistake, at least not so blatantly) There's also something to be said about how most of our evidence that a given world-model is true necessarily comes from the extended social web of other scientists, but I would focus on the more basic error of interpretation first.
It's a counterexample to a single step of reasoning(large multiverse of people --> God), it doesn't have to be globally a valid theory of reality. And clearly the existence of an imaginable multiverse satisfying a certain property makes it more plausible that our actual multiverse might satisfy the same property. (As an analogy, consider math, where you might want an object satisfying properties A and B. Constructing an object with property A makes it more plausible that you might eventually construct one with both properties)
Can you in practice use set theory to discover something new in other branches or math, or does it merely provide a different (and less convenient) way to express things that were already discovered otherwise?
The value of set theory as a foundation comes more from being a widely-agreed upon language that is also powerful enough to express pretty much everything mathematicians can think up, rather than as a tool for making new discoveries. I think it's worth learning at least at a shallow level for this reason, if you want to learn advanced math.
Well, UDASSA is false https://joecarlsmith.com/2021/11/28/anthropics-and-the-universal-distribution
Did you notice that I linked the very same article that you replied with? :P I'm aware of the issues with UDASSA, I just think it provides a clear example of an imaginable atheistic multiverse containing a great many possible people.
I think the cardinality should be Beth(0) or Beth(1) since finite beings should have finite descriptions, and additionally finite beings can have at most Beth(1)(if we allow immortality) distinct sequences of thoughts, actions, and observations, given that they can only think, observe, act, in a finite number of ways in finite time, so if you quotient by identical experiences and behaviors you get Beth(0) or Beth(1)(you might think we can e.g. observe a continuum amount of stuff in our visual field but this is an illusion, the resolution is bounded). The Bekenstein bound also implies physically limited beings in our universe have a finite description length.
There could be a God-less universe with Beth 2 people, but I don't know how that would work
I don't think it's hard to imagine such a universe, e.g. consider all possible physical theories in some formal language and all possible initial conditions of such theories. This might be less simple to state than "imagine an infinitely perfect being" but it's also much less ambiguous, so it's hard to judge which is actually less simple.
SIA gives reason to think you should assign a uniform prior across possible people
My perspective on these matters is influenced a lot by UDASSA, which recovers a lot of the nice behaviors of SIA at the cost of non-uniform priors. I don't actually think UDASSA is likely a correct description of reality, but it gives a coherent pictures of what an atheistic multiverse containing a great many possible people could look like.
I don't think the anthropic argument works. I have some technical objections to discussion of the set of possible people(I think Beth(0) or Beth(1) at most are more plausible cardinalities, I don't think we have to assume a uniform prior over possible people which means we don't need to assign a 0% probability to any particular being's existence in the absence of uniform existence) but more basically, I just don't see why God makes much of a difference as to the plausibility of any particular ontological arrangement. If you think God might create a universe with Beth(2) people, why couldn't there be a God-less universe with the same cardinality of people? If you think God might create a proper Class of people, why couldn't there be a God-less Proper Universe with the same people? Conversely, if modal realism undermines induction, doesn't a God-created set of all people undermine it in the same way? These universes might sound pretty "wild" and so appear implausible without intelligent design, but on a description-length perspective, "having beth(2) people and no God" or whatever can be specified pretty compactly. You might appeal to the infallibility, etc. of God to explain away paradoxes, but I think this is essentially invoking a "get out of paradoxes free" card without doing any explanatory work.
I also don't think I would lose as the gatekeeper(against a human), and would be willing to make a similar bet if anyone's interested.
I think twitter is still the closest thing to a global town square. This post by Tyler Cowen is good on the topic.
Subsist? Sustain? Self-actualize? Start?