Posts
Comments
(There was already a linkpost.)
What are the most noteworthy sections to read? (Looks like you forgot to bold them.) Thanks!
The Amazon link in the post is for the third (and latest) edition, only $28. Your other links are for the second edition, except the Harvard link's dead.
Did you forget to bold the particularly noteworthy sections in the table of contents?
More than a 76% pay cut, because a lot of the compensation at Google is equity+bonus+benefits; the $133k minimum listed at your link is just base salary.
I'd thought it was a law of nature that quiet norms for open plans don't actually work; it sounds like you've found a way to have your cake and eat it too!
That's fair; thanks for the feedback! I'll tone down the gallows humor on future comments; gotta keep in mind that tone of voice doesn't come across.
BTW a money brain would arise out of, e.g., a merchant caste in a static medieval society after many millennia. Much better than a monkey brain, and more capable of solving alignment!
Beren, have you heard of dependent types, which are used in Coq, Agda, and Lean? (I don't mean to be flippant; your parenthetical just gives the impression that you hadn't come across them, because they can easily enforce integer bounds, for instance.)
Thanks for the great back-and-forth! Did you guys see the first author's comment? What are the main updates you've had re this debate now that it's been a couple years?
The paper's first author, beren, left a detailed comment on the ACX linkpost, painting a more nuanced and uncertain (though possibly outdated by now?) picture. To quote the last paragraph:
"The brain being able to do backprop does not mean that the brain is just doing gradient descent like we do to train ANNs. It is still very possible (in my opinion likely) that the brain could be using a more powerful algorithm for inference and learning -- just one that has backprop as a subroutine. Personally (and speculatively) I think it's likely that the brain performs some highly parallelized advanced MCMC algorithm like Hamiltonian MCMC where each neuron or small group of neurons represents a single 'particle' following its own MCMC path. This approach naturally uses the stochastic nature of neural computation to its advantage, and allows neural populations to represent the full posterior distribution rather than just a point prediction as in ANNs."
One of his subcomments went into more detail on this point.
Re open plan offices: many people find them distracting. I doubt they're a worthwhile cost-saving measure for research-focused orgs; better to have fewer researchers in an environment conducive to deep focus. I could maybe see a business case for them in large orgs where it might be worth sacrificing individual contributors' focus in exchange for more legibility to management, or where management doesn't trust workers to stay on task when no one is hovering over their shoulder, but I hope no alignment org is like that. For many people open plan offices are just great, of course, and I think it can be hard for them to grok how distracting they can be for people on the autism spectrum, to pick a not-so-random example. :) But I like the idea of looking for ways to increase efficiency!
I meant I don't think the CEV of ancient Rome has the same values as ancient Rome. Looks like your comment got truncated: "what is good if they were just"
Is there a command-line tool for previewing how a "markdown+LaTeX" text file would render as a LW draft post, for those of us who prefer to manipulate text files using productivity tools like (neo)vim and git?
Ah right, because Clippy has less measure, and so has less to offer, so less needs to be offered to it. Nice catch! Guess I've been sort of heeding Nate's advice not to think much about this. :)
Of course, there would still be significant overhead from trading with and/or outbidding sampled plethoras of UFAIs, vs the toy scenario where it's just Clippy.
I currently suspect we still get more survival measure from aliens in this branch who solved their alignment problems and have a policy of offering deals to UFAIs that didn't kill their biological boot loaders. Such aliens need not be motivated by compassion to the extent that aboriginals form a Schelling bloc, handwave appendagewave. (But we should still play to win, like they did.)
Paperclips vs obelisks does make the bargaining harder because clippy would be offered fewer expected paperclips.
My current guess is we survive if our CEV puts a steep premium on that. Of course, such hopes of trade ex machina shouldn't affect how we orient to the alignment problem, even if they affect our personal lives. We should still play to win.
Roman values aren't stable under reflection; the CEV of Rome doesn't have the same values as ancient Rome. It's like a 5-year-old locking in what they want to be when they grow up.
Locking in extrapolated Roman values sounds great to me because I don't expect that to be significantly different than a broader extrapolation. Of course, this is all extremely handwavy and there are convergence issues of superhuman difficulty! :)
Yes it would, at least if you mean their ancient understanding of morals.
Not on mobile, in my experience.
I think it would be helpful to note at the top of the post that it's crossposted here. I initially misinterpreted "this blog" in the first sentence as referring to LW.
This idea keeps getting rediscovered, thanks for writing it up! The key ingredient is acausal trade between aligned and unaligned superintelligences, rather than between unaligned superintelligences and humans. Simulation isn't a key ingredient; it's a more general question about resource allocation across branches.
Too much power, I would assume. Yet he didn't kill Bo Xilai.
Why the downboats? People new to LW jargon probably wouldn't realize "money brain" is a typo.
Nitpick: maybe aligned and unaligned superintelligences acausally trade across future branches? If so, maybe on the mainline we're left with a very small yet nonzero fraction of the cosmic endowment, a cosmic booby prize if you will?
"Booby prize with dignity" sounds like a bit of an oxymoron...
You have a money brain? That's awesome, most of us only have monkey brains! 🙂
What does "corrupt" mean in this context? What are some examples of noncorrupt employers?