Should I do it?
post by MrLight · 2020-11-19T01:08:10.506Z · LW · GW · 7 commentsThis is a question post.
Contents
Should I press it or not? None Answers 4 ChristianKl None 7 comments
Shall we bite the apple?
Hello community, nice to meet you here.
For a long time I have been tormented by a question that I can't seem to answer alone.
Let's talk hypothetically. Let's assume that I have developed a method that can be mapped in a small program. This method, once started, will develop an AGI by itself.
The program works independently, evolves and gets better from minute to minute. Not just better with one problem, but with any number of problems. The algorithm is able to rewrite and adapt itself. The algorithm follows this path of continuous improvement until it ends in an AGI.
This algorithm consists of several individual components, which together give this algorithm the capabilities described above. Imagine that I wrote and tested each of these individual components separately from each other in an alpha release (error driven design). And further imagine, each of these parts delivered the expected result.
The only thing left is to write these components in a faster language like C ++ or Assembler, bring them together and press 'Start'.
My dilemma is that I cannot ensure that the process can be stopped once it has started. I'm curious.
So I sit here and think. Should I do it and provide evidence? Or should I just wait, do nothing and watch someone else come up with the idea how to do it?
Or should I disclose the process without a program? Just publish it so that everyone can read it? A gift to humanity. The question is justified when you consider the risks.
But if you want to test it in a safe environment, then it's expensive and costs a lot, a lot of money. I am getting impatient.
... and if you keep walking on the paths of imagination, imagine that you have turned to your government about your dilemma, and to other governments as well, as well as to the economy, and nobody answers you ...
So I have two options.
A.) Either my aborted mathematics studies and 34 years of experience as a programmer led me to a dead end. I need a prove for myself.
B.) I'm right and nobody believes me. Either because I've never published anything or because they think it's technically impossible. Or they consider the risk to be insignificant, or the profit to be too small. I need a prove for them.
For (A) the solution is simple. Publish proceedings and someone will be found to prove that I am wrong. So I have the proof and I can do something else.
For (B) the answer is also very simple. DO NOT publish procedure. Apparently there are people who classify the risk as harmless. If there are people who don't care about risk, there will also be programmers among them who simply reprogram it out of curiosity. So what?
- Should such information be in the public hand? Which involves risks.
- Or should I look for someone who will provide the money for it? Then the AGI would be developed in a secure environment. However, it would be at least partially under the control of the funder. Which is also associated with risks.
- Or should I just write it myself and see what happens? What would be nice if it worked, or what would be terrible if it worked.
So I'm sitting at home in front of an imaginary, flashing red button.
Should I press it or not?
As I said, it's all just hypothetical, and what if not? I'm confused.
I am curious about your opinion and your suggestions what I should do.
Thank you for your attention.
#shallwebitetheapple
Answers
Most likely your design doesn't work to produce an AGI that effectively self-improves. In most scenario where it actually produces a design that self-improves it will become really smart and you will lose control over it and a bit later that's the end of human life.
↑ comment by MrLight · 2020-11-19T20:31:40.363Z · LW(p) · GW(p)
Well, I know all the possible problems and obstacles in development. I solved them in my calculations and also solve each problem individually. It seems reasonable to assume that it will work. I am not alone in this assessment. But only a real experiment would prove it. But since I do not bear the risk alone, all of you with me. I wanted to ask for your opinion.
Replies from: ChristianKl↑ comment by ChristianKl · 2020-11-19T20:45:09.298Z · LW(p) · GW(p)
Well, I know all the possible problems and obstacles in development.
I wouldn't trust someone to do anything safety critical if they claim that they know all possible problems and obstacles. Unknown unknown problems are always part of the concern when doing something new.
If you actually do make a decision to run this, I recommend doing it on an airgapped computer and commit to if it actually manages to self-improve in any way show the thing to someone well-versed in AI safety before removing airgapping.
Replies from: MrLight↑ comment by MrLight · 2020-11-19T21:58:06.744Z · LW(p) · GW(p)
You're absolutely right. I only claim to have solved the known manufacturing problems. I know how to build/code one. Not the unknown problems that are sure to come.
I didn't mean security issues. That question is still open.
A development on a laptop or a separate system is not possible with the required computing capacity. And yes, I am talking to universities and specialists. It's all in the works. But none of them can answer the moral question for me.
Folks, I'm not far from completion, only a few months. And then what? I wanted to think about a few things beforehand.
Because You can't trap an AGI in a box. It will always find a way out. I see the code here in front of me. I see the potential. Believe me. I don't believe it can't be hold in a 'Black Box'. The question is also should we?
Do you want to be born in a maximum security prison?
What opinion should the AI have of us when we put it in jail?
Replies from: ChristianKl↑ comment by ChristianKl · 2020-11-21T21:38:16.186Z · LW(p) · GW(p)
I didn't mean security issues. That question is still open.
It seems like if your AGI actually works there's a good chance that it kills humanity.
Replies from: MrLight↑ comment by MrLight · 2020-11-21T23:07:48.869Z · LW(p) · GW(p)
It seems like if your AGI actually works there's a good chance that it kills humanity.
But isn't humanity already killing itself? Maybe is an AI our last chance to survive?
Replies from: ChristianKl↑ comment by ChristianKl · 2020-11-22T17:05:56.467Z · LW(p) · GW(p)
But isn't humanity already killing itself?
No, population is growing. Spending a few additional decades on AI safety research is likely improving our chances of survival. Of course listening to AI safety researchers and not just AI researchers from a random university matters as well.
Replies from: MrLight↑ comment by MrLight · 2020-11-25T04:18:31.286Z · LW(p) · GW(p)
Of course listening to AI safety researchers ...
I've read quite a bit about this area of research. I haven't found a clear solution anywhere. There is only one point that everyone agrees on. With increasing intelligence, the possibilities of control is declining in the same way as the rise of the possibilities and the risk.
Replies from: ChristianKl↑ comment by ChristianKl · 2020-11-25T09:25:34.810Z · LW(p) · GW(p)
I haven't found a clear solution anywhere. There is only one point that everyone agrees on.
Yes, according to current knowledge most AGI designs are dangerous. Speaking to researchers could help one of them to explain to you why your particular design is dangerous.
7 comments
Comments sorted by top scores.
comment by Mitchell_Porter · 2020-11-19T10:14:29.498Z · LW(p) · GW(p)
What will your AGI do, once it exists?
Replies from: MrLightcomment by Pattern · 2020-11-20T18:37:52.917Z · LW(p) · GW(p)
My dilemma is that I cannot ensure that the process can be stopped once it has started. I'm curious.
What is the setup where you can't switch it off? That it might find a way to disable that capability, or are you worried about something else?
Replies from: MrLight↑ comment by MrLight · 2020-11-20T20:10:44.296Z · LW(p) · GW(p)
So much bad karma....
That's a good question.
A fractal, self-organizing digital organism that learns at geometric speed.
What can go wrong when it is calculated in a cloud on several computers around the world?
The software needs a higher computing capacity, so maybe I'll book some in the cloud. If everything works the way I expect, everything is OK. If it doesn't work, that's OK too. But what if things go better than expected?
There are enough papers that deal with exactly this risk. I don't have to quote them here. You can find enough information on the net. With a "stop" button or switch you won't get far ..
Google's AI has found glitches in games. And this AI can only play Atari games.
In addition, if it runs on a computer that can be reached in the network, the algorithm can be stolen and easily reprogrammed into a virus (actually a chained worm, but that doesn't matter now). The algorithm is designed so that you can define any (formally definable) goal. In the wrong hands, this can cause considerable damage.
Maybe I shouldn't worry so much and just try it out?
(I know, my English isn't perfect. I apologize for that.)
↑ comment by Pattern · 2020-11-24T04:04:22.531Z · LW(p) · GW(p)
Can you run it for a while and then stop it?
Replies from: MrLight↑ comment by MrLight · 2020-11-25T03:52:49.503Z · LW(p) · GW(p)
Sure. At the beginning, as it develops, sure (99%).
At the beginning everything will be nice, I guess. It will only consume computing power and slowly develop. Little baby steps.
I can watch how memory cells switch from 0 to 1 and maybe back. I can maybe play some little games as test, to see how it develops.
But the question when to stop. Will I be able to realize the moment when and how it starts to go bad? Before it's to late? When will it be? When will it be to late?
The developing speed depends on how many cores and computers works on the development. But to see if it develops in the "right" way (whatever the "right" way is), I need to let it to develop.
But what if it develops curiosity? ...and it will. What if it needs more calculating power? What If I say no and try prevent it? Should I?
About all this questions I was already thinking, and it was always a remaining risk. The probability is small, but not zero. The possible risk is immensely.
And, as far as it develops, more an more people will be involved.
More people are more opinions, beliefs, needs, fears and desires. Corporations will show interest and governments will express their concerns and desire to participate. Contracts will be made and laws will be passed. Interests will be served. At what point should I pull the plug? How long will I be able to do it? Won't it be too late then?
Shouldn't we talk now about it?
Replies from: Pattern↑ comment by Pattern · 2020-11-26T15:43:25.955Z · LW(p) · GW(p)
I don't know when you should stop. All I'm suggesting is that you not turn it on, without a time on which it is supposed to (automatically) switch off. In other words, you should stop it regularly, over and over again. This has the benefit of letting you consider the new information you have received, and decide how to respond to it. Perhaps your design will be "flawed" - and won't have the risk of going 'foom' that you think it will (without further work to revise and change it - by you, before it is capable of 'improving'). If you decide that it is risky, then the 'intervention' isn't turning it off - it's just not deciding to turn it back on (which maybe shouldn't be automatic).