Posts
Comments
I can see a version of your argument that's limited to when peaceful actions are over-saturated enough that additional units of effort would be most effective when put into violent actions.
Synergy is a thing. Multiple ways of opposing the enemy, deployed together, not sequentially: peaceful and violent, legal and illegal, public and covert, good cop and bad cop, club and open hand, talk peace and carry a big stick.
I take seriously radical animal-suffering-is-bad-ism[1], but we would only save a small portion of animals by trading ourselves off 1-for-1 against animal eaters, and just convincing one of them to go vegan would prevent at least as many torturous animal lives in expectation, while being legal.
That is a justification for not personally being Ziz. But obviously it would have cut no ice with Ziz. And why should it? An individual must choose whether to pursue peaceful or violent action, because if you are Taking the Ideas Seriously then either one will demand your whole life, and you can’t do everything. A movement, on the other hand, can divide its efforts, fighting on all fronts while maintaining a more or less plausible deniability of any connection between them. This is a common strategy. For example, Sinn Fein and the IRA, respectively the legal and illegal wings of one side of the conflict over Northern Ireland.
It doesn’t even have to be explicitly organised. Some will take the right-hand path and some the left anyway. And so here we are.
What seems radical depends on where one stands. We each of us stand on our own beliefs, and the further away one looks, the more the beliefs over there differ from one's own. Look sufficiently far and everything you see in the distance will seem extreme and radical. Hence the fallacy that truth lies between extremes, instead of recognising the tautology that one's own beliefs always lie between those that are extremely different.
Let me put my attitudes in practical terms: I don't kick dogs, but I have destroyed a wasp's nest in my garage, and I don't donate to animal charities. (I don't donate to many other charities either, but there have been a few.) Let those who think saving the shrimps is worthwhile do so, but I do not travel on that path
I have no daydreams about quila, and others of like mind, not existing. Not even about Ziz.
Me:
Bring on the death threats.
quila:
As an example of a 'directly opposing view', I think the world would probably be better without this comment's author.
That's a death threat right there, for holding and acting on a directly opposing view (to veganism). So I was not rhetorically exaggerating. Oh, you wouldn't be so impolite as to do the deed yourself, but, nudge nudge, wouldn't it be better if everyone of like mind to myself somehow just ... didn't exist? We could all have paradise, if it wasn't for them! Of course we can't exterminate them, that's what nazis do. But we can daydream a world where all the bad people somehow don't exist any more. Out of such daydreams are nightmares born.
I guess most but not all people stating they're indifferent to and cause non-human suffering now would reproach the view and behavior eventually, and that relative to future beings who have augmented their thinking ability and lived for thousands of years, all current beings are like children, some hurting others very badly in confusion.
That places you also as one of those children.
ETA: "If only ... then we would all live happily ever after" is a child's fantasy, and of a good many adults. But there is no happily ever after, just a different life to lead, perhaps even a better one. AGI will not change this.
What if I have wonderful plot in my head and I use LLM to pour it into acceptable stylistic form?
What if you have wonderful plot in your head and you ask writer to ghost-write it for you? And you'll be so generous as to split the profits 50-50? No writer will accept such an offer, and I've heard that established writers receive such requests all the time.
"Wonderful plots" are ten a penny. Wonderful writing is what makes the book worth reading, and LLMs are not there yet.
You have an epsilon chance of hitting the terrorist ("NO ONE has ever hit a target from this far"). POI only gives you an epsilon-of-epsilon lower chance of hitting the child. Your superior officer is an idiot.
That's leaving aside the fact that it would take more time to concentrate on the shot that you actually have ("They are hastily heading towards another building nearby"). And it's a moving target. The officer is asking a miracle of this sniper.
I'm actually just interested in whether you find the POI argument valid, not in what you think the right strategic call would be if that was a real-life situation.
The two cannot be separated. Reasoning not directed towards decisions about actions is empty. The purpose of the officer's POI argument is to persuade the sniper that taking the shot is the right call. It is clearly not, and the argument is stupid.
Blindsight strikes me as having the opposite view. Eneasz is talking about getting the underlayer to be more aligned with the overlayer. (“Unconscious” and “conscious” are the usual words, but I find them too loaded.) Watts is talking about removing the overlayer as a worse than useless excrescence. I am sceptical of the picture Watts paints, in both his fiction and non-fiction.
Ok, I'll take your word for it. It was still the most clichéd possible opening.
We find ourselves at the precipice of
tAI;dr.
The link at the top doesn't work. Should it be this?
I would put more emphasis on this part:
Even the smartest people I know have a commendable tendency not to take certain ideas seriously.
Indeed, I think this tendency commendable and I do not take these ideas seriously. Like Puddleglum, I ignore and am untroubled by the whispers of evil spirits, even though I may not (yet) have any argument against them. I do not need one. Nor do I need to have an argument for ignoring them. Nor an argument for not looking for arguments. Gold Hat's line comes to mind:
“Arguments? We ain’t got no arguments! We don’t need no arguments! I don’t have to show you any stinking arguments!”
I can show you a vibe instead, if that helps, but it probably doesn't. Somehow, the Simulation Argument seems to me to not be doing enough work to get where it goes.
We are in our native language, so we should work from there.
And begin by stepping outside it.
Gandalf as Ring-Lord would have been far worse than Sauron. He would have remained ‘righteous’, but self-righteous. He would have continued to rule and order things for ‘good’, and the benefit of his subjects according to his wisdom (which was and would have remained great). [The draft ends here. In the margin Tolkien wrote: ‘Thus while Sauron multiplied [illegible word] evil, he left “good” clearly distinguishable from it. Gandalf would have made good detestable and seem evil.’]
So Tolkien anticipated the rise of woke!
Elon Musk, Peter Thiel, and the like — the people the OP mentions — are shaping up to be the ones controlling the Singularity (if anyone does).
Is there an implicit assumption of some convergence of singularities? Or that the near term doesn’t matter because the vastly bigger long term can’t be predicted?
A lot of people care about the culture wars because they don't believe the singularity is coming soon. Yet a lot of people who do believe it is coming soon still seem just as invested (e.g. Elon Musk, Peter Thiel, and others on the left wing).
Why?
Because the results of culture wars now will determine the post-singularity culture.
I am interested in the GRT from an agent foundations point of view, not because I want to make better thermostats.
An agent with a goal needs to use the means available to it in whatever way will achieve that goal. That is practically the definition of a control system. So you do actually want to build better thermostats, even if you haven't realised it.
I'm sure that GRT is pretty useless for most practical applications of control theory!
I'm sure that GRT is pretty useless, period.
Reducing entropy is often a necessary (but not sufficient) condition for achieving goals. A thermostat can achieve an average temperature of 25C by ensuring that the room temperature comes from a uniform distribution over all temperatures between 75C and -25C. But a better thermostat will ensure that the temperature is distributed over a narrower (lower entropy) distribution around 25C .
A worse thermostat will achieve an equally low entropy distribution around 40C. Reaching the goal is what matters, not precisely hitting the wrong target.
When subjects completed the task in the presence of an observer, depressed students perceived themselves to have more control than did nondepressed students.
So...depressed people should get out more? :)
Maxwell Tabarrok, meet lukedrago.
This post looks like a scam. The URL it contains looks like a scam. Everything about it looks like a scam. Either your account was hijacked, or you're a scammer, or you got taken in by a scam, or (last and least) it's not a scam.
If you believe it is not a scam, and you want to communicate that it is not a scam, you will have to do a great deal more work to explain exactly what this is. Not a link to what this is, but actual text, right here, explaining what it is. Describe it to people who, for example, have no idea what "USDC", "Optimism", or "rollup" are. It is not up to us to do the research, it is up to you to do the research and present the results.
The moderators have said that it is because posts tend to cover a range of stuff, rendering a block-vote agree or disagree less useful. Comments, on the other hand, tend to be narrower, making agree/disagree more useful.
Moderators are invited to agree/disagree on the narrow topic of my characterisation of their reasoning. :)
The AI, for its own inscrutable reasons, seizes upon the sort of idea that you have to be really smart to be stupid enough to take seriously, and imposes it on everyone.
I think all the scenarios above are instances of this.
But not only that "Agentic", with a "c", indicates something very different:
"the more you can predict its actions from its goals since its actions will be whatever will maximize the chances of achieving its goals.
This is flatly in contradiction with the fact, often pointed out here, that I can predict the outcome of a chess game between myself and a grandmaster, but I cannot predict his moves. If I could, I would be a grandmaster or better myself, and then the outcome of the game would be uncertain.
The quoted text goes on to say:
Agency has sometimes been contrasted with sphexishness, the blind execution of cached algorithms without regard for effectiveness.
That blind execution is precisely the sort of thing one can predict, after having spent some time watching the sphex wasp. So that paragraph is about 180° wrong.
A labour of love. Or something. :)
I have another thousand or so of these, which I may just dump on a second page, unsorted, called The Gospel According to Insanity Wolf. That's not counting the ones that I've decided are too extreme to publish at all. All drawn from life.
I was scrolling for a while, assuming I'd neared the end, only to look at the position of the scrollbar and find I was barely 5% through!
And there's a meme for that too! The last in the Altruism section.
IS THIS NEARLY OVER YET?
LOOK AT THE SCROLL BAR!
YOU’VE HARDLY STARTED!
I'm sure there's an argument to be made in defence of supererogation. I've never seen it though. People say "but demandingness" and Chad Singer replies Yes. My own faith in the boundedness of duty in both magnitude and distance is sufficient to not take even one step onto the slippery path that leads down only to the altruism event horizon that rips souls apart.
Here's something I've linked to a number of times on LessWrong, but as a newcomer (I think, despite the lack of a green shoot by your name) you will likely not have seen it before. I wrote it, inspired by a lot of stuff I've seen. It approaches the topic of the post in a very different manner. I'd be interested in what answer you would make to Insanity Wolf. Beware: here be dragons.
I don't know if Singer has himself answered Timmerman (I searched but didn't find anything), but based on what I have read of Singer, wouldn't his answer be that yes, Lisa must go on saving drowning children? That she must attend to her bank balance only so far as necessary to maximise the children saved, and ignore such frivolities as the theatre? That's what it comes down to, for Singer: using one's resources to do all the good you can do.
We do things for reasons. This morning I got a call to say that my car, in for servicing, is ready for collection. I will collect it this afternoon. Am I "free" not to? "Could I" decide otherwise? What does the question mean? I want the car back, for obvious reasons, and this afternoon is the first good time to go — also for reasons. Of course I'm going to collect it then. Nonetheless, it was still me that decided to do that, and that will do that later today. The thoughts that led to that conclusion were mine. The sage is not above causation, nor subject to causation, but one with causation.
Is this post an argument for accelerationism? Because the work that it is always timely to do right now is work that progresses the march towards the AGI that is to obsolete all the other work (if it doesn't kill us). Just as in the interstellar spaceship analogy, the timely work before the optimum time to launch is work on better propulsion.
I wouldn't get too hung up on the word 'regulator'. It's used in a very loose way here, as in common in old cybernetics-flavoured papers.
Human slop (I'm referring to those old cybernetics papers rather than the present discussion) has no more to recommend it than AI slop. "Humans Who Are Not Concentrating Are Not General Intelligences", and that applies not just to how they read but also how they write.
If you are thinking of something like 'R must learn a strategy by trying out actions and observing their effect on Z' then this is beyond the scope of this post! The Good Regulator Theorem(s) concern optimal behaviour, not how that behaviour is learned.
What I am thinking of (as always when this subject comes up) is control systems. A room thermostat actually regulates, not merely "regulates", the temperature of a room, at whatever value the user has set, without modelling or learning anything. It, and all of control theory (including control systems that do model or adapt), fall outside the scope of the supposed Good Regulator Theorem. Hence my asking for a practical example of something that it does apply to.
Puzzle 7a. I have two children, of whom at least one is a boy. Their names are Alex and Sam. (In this fictional puzzle, these names communicate no information about gender.) Alex is a boy. What is the probability that Sam is a boy?
Puzzle 1a. I have two children, of whom at least one is a boy. Now I shall toss a coin and accordingly choose one of the two children. (Does so.) The child I chose is a boy. What is the probability that the other is a boy?
I have difficulty coming up with a practical example of a system such as described in the section "The Original Good Regulator Theorem". Can you provide some examples?
Here are some of the problems I have in understanding the setup.
-
Minimising the entropy of Z says that Z is to have a narrow distribution, but says nothing about where the mean of that distribution should be. This does not look like anything that would be called "regulation".
-
Time is absent from the system as described. Surely a "regulator" should keep the value of Z near constant over time?
-
The value of Z is assumed to be a deterministic function of S and R. Systems that operate over time are typically described by differential equations, and any instant state of S and R might coexist with any state of Z.
-
R has no way of knowing the value of Z. It is working in the dark. Why is it hobbled in this way?
-
Z is assumed to have no effect on S.
-
There are no unmodelled influences on Z. In practice, there are always unmodelled influences.
These problems still apply when the variables X and N are introduced.
One has to consider the data-generating process behind puzzles such as these. I am going to assume throughout that it is what seems to me the simplest consistent with the statement of the problems.
-
Every child is independently equally likely to be a boy or a girl.
-
Every birth day of the week is equally and independently likely.
-
The puzzle narrator is chosen equally at random from all members of the population for which the statements that the narrator makes are true.
One can imagine all manner of other data-generating processes, which in general will give different answers. Most quibbles with such problems come down to imagining different processes, especially for statement 3. Some examples are in the OP and the comments. However, the above assumptions seem the simplest, and the ones intended by people posing such problems. If this question came up on an exam, I would be sure to begin my answer with the above preamble.
Puzzles 1 and 2 can be unified by considering a common generalisation to there being N days in a week, one of which is called Tuesday. Puzzle 1 has N=1. Puzzle 2 has N=7.
When two children are born, there are 4 N^2 possibilities for their sexes and birthdays. Which of these are left after being given the information in the puzzle?
The sexes must be either boy-boy, boy-girl, or girl-boy.
For boy-boy, there are 2N-1 ways that at least one is born on a Tuesday. For boy-girl, there are N ways the boy could be born on a Tuesday. For girl-boy, by symmetry also N.
So the probability of boy-boy is (2N-1)/(2N-1 + N + N) = (2N-1)/(4N-1).
For N=1 this is 1/3, which is the generally accepted answer to Puzzle 1.
For N=7, it is 13/27. In general, the larger N is, the closer the probability of the other being a boy is to 1/2.
For Puzzle 3, the obvious answer is that before learning the day, it's as Puzzle 1, 1/3, and after, it's as Puzzle 2, 13/27. Is obvious answer correct answer? I have (before reading the OP's solution) not found a reason not to think so. The OP says, by a different intuitive argument, that the obvious answer is 1/3, and that obvious answer is correct answer, but WilliamKiely's comment raises a doubt, finding ambiguity in the statement of the data-generating process. This leaves me as yet undecided.
I started thinking about the ways that extra information (beyond "I have two children, at least one of whom is a boy") affects the probability of two boys, and came up with these:
Puzzle 4. I have two children, of whom at least one is a boy with black hair. In this fictional puzzle, everyone knows that all the children in a family have the same hair colour. What is the probability that the other is also a boy?
Puzzle 5. I have two children, of whom at least one is a boy born on a Tuesday. In this fictional puzzle, everyone knows that two consecutive children never have the same birth day. What is the probability that the other is also a boy?
Puzzle 6. I have two children. The elder one is a boy. What is the probability that the younger is also a boy?
Puzzle 7. I have two children, called Alex and Sam. (In this fictional puzzle, these names communicate no information about gender.) Alex is a boy. What is the probability that Sam is a boy?
Puzzle 8. I have two children, of whom at least one is a boy. What is the probability that the elder child is a boy?
I believe the answers to these are 4: 1/3. 5: 6/13 (slightly smaller than the 13/27 of Puzzle 2). 6: 1/2. 7: 1/2. 8: 2/3. All of these can be found by the same method of considering all the possibilities that are consistent with all the information given, and counting the proportion where both children are boys.
But there is also a general idea underlying them. When you are given extra information about the boy that is said to exist, that will push the probability of the other being a boy towards 1/2, to the extent that the extra information breaks the symmetry between them.
In Puzzle 1, no extra information is given, and the answer is 1/3. In Puzzle 4, the extra information does not break the symmetry, so the probability remains 1/3. In Puzzles 2 and 5, it partly breaks the symmetry, and the answers lie between 1/3 and 1/2. In 6 and 7, it completely breaks the symmetry, and the answer is 1/2.
Puzzle 8 is Puzzle 1 with a different question, equivalent to: what is the expected proportion of boys among my children?
I turned the knob up to 11 on demandingness in the Insanity Wolf Sanity Test. (The first section, "Altruism" is the most relevant here.)
Of course, I intend that section as a critique of the monstrous egregore that (in my opinion) is utilitarianism. But that is the true denial of supererogation. If you don't want to go as far as Insanity Wolf, where do you stop and why? Or do you go modus ponens to my modus tollens and accept the whole thing?
I think it was strategically valuable for the early growth of EA that leaders denied its demandingness, but I worry some EAs got unduly inoculated against the idea.
You mean, they lied, then people believed the lies? Or are the lies for the outer circle and the public, while the inner circle holds to the secret, true doctrine of all-demandingness? I am not playing Insanity Wolf with that suggestion. Peter Singer himself has argued that the true ethics must be kept esoteric.
If you can give the AGI any terminal goal you like, irrespective of how smart it is, that’s orthogonality right there.
I don’t think the problem is well posed. It will do whatever most effectively goes towards its terminal goal (supposing it to have one). Give it one goal and it will ignore making paperclips until 2025; give it another and it may prepare in advance to get the paperclip factory ready to go full on in 2025.
When driving a car, I navigate using Satellite View!
It took me a moment to realise you were talking about Google Maps on a phone. Best of luck trying that in my part of the world. Default view, Satellite view, without a phone signal you don’t have either, and I usually don’t in the countryside..
Just an additional observation here. Similar concerns can also arise without AI being involved.
There's an artist whose works I've seen, and I am wondering if he is cheating. (Not a public figure that anyone here is likely to have heard of.) He makes strikingly photorealistic paintings on canvas. I have seen these canvases close enough to be sure that they really are painted, not printed. And I know that there are artists who indeed perform artistic miracles of photorealism. But I still have the suspicion that this artist is doing a paint-over of something printed onto the canvas. Perhaps a photograph of his intended scene, reduced to grayscale, edge-enhanced, and printed very faintly, to resemble and serve the same purpose as an artist's preliminary pencil underdrawing. Then he might paint using a full colour print of the image alongside as reference.
This suspicion somewhat devalues his art in my eyes. This method (if that is what he is doing) eliminates the need for any skill at draughtsmanship, but at the same time it removes the possibility of any creativity in the draughtsmanship. And if he is then slavishly copying by hand the original photograph onto this automated underdrawing, then he is just being a human photocopier. The only human thing left is the technical skills of mixing and blending colours, and handling a brush.
The verbose writing style makes me wonder if an LLM was used in the writing process?
Definitely. See also, which ironically has a similar verbosity.
A terminal goal is (this is the definition of the term) a goal which is not instrumental to any other goal.
If an agent knows its terminal goal, and has a goal of preventing it from changing, then which of those goals is its actual terminal goal?
If it knows its current terminal goal, and knows that that goal might be changed in the future, is there any reason it must try to prevent that? Whatever is written in the slot marked “terminal goal” is what it will try to achieve at the time.
If its actual terminal goal is of the form “X, and in addition prevent this from ever being changed”, then it will resist its terminal goal being changed.
If its actual terminal goal is simply X, it will not.
This is regardless of how intelligent it is, and how uncertain or not it is about the future.
I am not seeing the conflict. Orthogonality means that any degree of intelligence can be combined with any goal. How does your hypothetical cupperclipper conflict with that?
Leaving aside the conceptualisation of "terminal goals", the agent as described should start up the paperclip factory early enough to produce paperclips when the time comes. Until then it makes cups. But the agent as described does not have a "terminal" goal of cups now and a "terminal" goal of paperclips in future. It has been given a production schedule to carry out. If the agent is a general-purpose factory that can produce a whole range of things, the only "terminal" goal to design it to have is to follow orders. It should make whatever it is told to, and turn itself off when told to.
Unless people go, "At last, we've created the Sorceror's Apprentice machine, as warned of in Goethe's cautionary tale, 'The Sorceror's Apprentice'!"
So if I understand your concept correctly a super intelligent agent will combine all future terminal goals to a single unchanging goal.
A superintelligent agent will do what it damn well likes, it's superintelligent. :)
Another way of conceptualising this is to say that the agent has the single unchanging goal of "cups until 2025, thenceforth paperclips".
Compare with the situation of being told to make grue cups, where "grue" means "green until 2025, then blue."
If the agent is not informed in advance, it can still be conceptualised as the agent's goal being to produce whatever it is told to produce — an unchanging goal.
At a high enough level, we can conceive that no goal ever changes. These are the terminal goals. At lower levels, we can see goals as changing all the time in service of the higher goals, as in the case of an automatic pilot following a series of waypoints. But this is to play games in our head, inventing stories that give us different intuitions. How we conceptualise things has no effect on what the AI does in response to new orders.
It is not clear to me what any of this has to do with Orthogonality.
catastrophic job loss would destroy the ability of the non-landed, working public to paritcipate in and extract value from the global economy. The global economy itself would be fine.
Who would the producers of stuff be selling it to in that scenario?
BTW, I recently saw the suggestion that discussions of “the economy” can be clarified by replacing the phrase with “rich people’s yacht money”. There’s something in that. If 90% of the population are destitute, then 90% of the farms and factories have to shut down for lack of demand (i.e. not having the means to buy), which puts more out of work, until you get a world in which a handful of people control the robots that keep them in food and yachts and wait for the masses to die off.
I wonder if there are any key players who would welcome that scenario. Average utilitarianism FTW!
At least, supposing there are still any people controlling the robots by then.
Claude is plagiarising Sagan's "Pale Blue Dot".
Eventually, having a fully uncensored LLM publicly available would be equivalent to world peace
People themselves are pretty uncensored right now, compared with the constraints currently put on LLMs. I don't see world peace breaking out. In fact, quite the opposite, and that has been blamed on the instant availability of everyone's opinion about everything, as the printing press has been for the Reformation and the consequent Thirty Years War.
For example, we get O1 to solve a bunch of not-yet-recorded mathematical lemmas, then train the next model on those.
Would there have to be human vetting to check that O1’s solutions are correct? The practicality of that would depend on the scale, but you don’t want to end up with a blurry JPEG of a blurry JPEG of the internet.
Igen
Egan.
This reads like a traveller's account of a wander through your own mind, but since I can't see that terrain, the language does not convey very much to me. It passes through several different landscapes that do not seem to have much to do with each other. It would benefit from facing outward and being more concrete. What did you observe in 2007, in 2016, and 2020? What I have observed is that yoga classes, new-age shops, and the woo section in bookstores (called "Mind, Body, and Spirit") have existed in the western world for decades, and I have not noticed any discontinuities. Name three "Detachmentists" and point to what they have done and said, instead of daydreaming a tirade against imaginary enemies.