Evidence other than evolution for optimization daemons?
post by Liam Donovan (liam-donovan) · 2019-04-21T20:50:18.986Z · LW · GW · 6 commentsThis is a question post.
Contents
Answers 12 rohinmshah None 6 comments
The idea of consequentialist agents arising in sufficiently strong optimizing systems intuitively makes sense to me. However, I don't have a good mental model of the differences between a world where optimization daemons can arise and a world where they can't (i.e. what facts about the world provide Bayesian evidence for the concept of ODs). The only example I've seen is the evolution of humans, but I find it concerning that I can't make any other predictions about the world based on the idea of ODs.
What other Bayesian evidence/potential intuition pumps exist for the possibility of optimization daemons arising?
[removed discussion of religion to make the question more clear/straightforward]
Answers
I think a lot of the intuition right now is "there is an argument that inner optimizers will arise by default; we don't know how likely it is but evolution is one example so it's not non-negligible".
For the argument part, have you read More [AF · GW] realistic tales of doom [AF · GW]? Part 2 is a good explanation of why inner optimizers might arise.
↑ comment by Liam Donovan (liam-donovan) · 2019-04-22T16:33:24.666Z · LW(p) · GW(p)
I have read that post; it makes sense, but I'm not sure how to distinguish "correct" from "persuasive but wrong" in this case without other evidence
6 comments
Comments sorted by top scores.
comment by Rob Bensinger (RobbBB) · 2019-04-21T21:02:01.510Z · LW(p) · GW(p)
"Catholicism predicts that all soulless optimizers will explicitly represent and maximize their evolutionary fitness function" is a pretty unusual view (even as Catholic views go)! If you want answers to take debates about God and free will into account, I suggest mentioning God/Catholicism in the title.
More broadly, my recommendation would be to read all of https://www.lesswrong.com/rationality [? · GW] and flag questions and disagreements there before trying to square any AI safety stuff with your religious views.
Replies from: liam-donovan, liam-donovan↑ comment by Liam Donovan (liam-donovan) · 2019-04-21T21:33:14.083Z · LW(p) · GW(p)
Regarding the first part of your comment: If I understand the quoted section correctly, I don't think I know enough about biology or theology to confidently take a position on that view. Is the observed behavior of some soulless optimizer (e.g. intelligent non-human primates) significantly different from what one would expect if they only maximized inclusive genetic fitness? If so, that would definitely answer my question.
↑ comment by Liam Donovan (liam-donovan) · 2019-04-21T21:13:52.014Z · LW(p) · GW(p)
Thank you for the prompt response to a poorly-worded question!
I'm not particularly interested in answers that take God/free will into account; I was just hoping to find evidence/justifications for the existence of optimization daemons other than evolution. It sounds like my question would be clearer and more relevant if I removed the mention of religion?
comment by shminux · 2019-04-22T00:07:58.135Z · LW(p) · GW(p)
Would you consider a MENACE Tic-tac-toe matchbox-based optimizer an OD?
Replies from: liam-donovan↑ comment by Liam Donovan (liam-donovan) · 2019-04-22T03:39:23.967Z · LW(p) · GW(p)
I don't understand why it would be -- it looks like MENACE is just a simple physical algorithm that successfully optimizes for winning tic-tac-toe. I thought the idea of an OD was that a process optimizing for goal A hard enough could produce a consequentialist* agent that cares about a different goal B. What is the goal B here (or am I misunderstanding the concept)?
*in the sense Christiano uses "consequentialist"
Replies from: shminux