Does the Structure of an algorithm matter for AI Risk and/or consciousness? 2021-12-03T18:31:40.185Z
AGI is at least as far away as Nuclear Fusion. 2021-11-11T21:33:58.381Z
How much should you be willing to pay for an AGI? 2021-09-20T11:51:33.710Z
The Walking Dead 2021-07-22T16:19:48.355Z
Against Against Boredom 2021-05-16T18:19:59.909Z
TAI? 2021-03-30T12:41:29.790Z
(Pseudo) Mathematical Realism Bad? 2020-11-22T18:21:30.831Z
Libertarianism, Neoliberalism and Medicare for All? 2020-10-14T21:06:08.811Z
AI Boxing for Hardware-bound agents (aka the China alignment problem) 2020-05-08T15:50:12.915Z
Why is the mail so much better than the DMV? 2019-12-29T18:55:10.998Z
Many Turing Machines 2019-12-10T17:36:43.771Z


Comment by Logan Zoellner (logan-zoellner) on Summary of the Acausal Attack Issue for AIXI · 2021-12-13T11:10:10.757Z · LW · GW

I feel like the word "attack" here is slightly confusing given that AIXI is fully deterministic.  If you're an agent with free will, then by definition you are not in a universe that is being used for Solomonoff Induction.

if you learn that there's an input channel to your universe

There's absolutely no requirement that someone in a simulation be able to see the input/output channels.  The whole point of a simulation is that it should be indistinguishable from reality to those inside.

Consider the following pseudocode:

def predictSequence(seq):
    while True:
       if equals( obs,seq):
        return universe.stateAtPoint(0,0)

Now suppose that every point in space is Topologically Indistinguishable (as in our universe).  There is literally no way for an agent inside the universe to distinguish the "output channel" from any other point in the universe.

But wait, there can only be so many low-complexity universes, and if they're launching successful attacks, said attacks would be distributed amongst a far far far larger population of more-complex universes.

This is precisely the point of Solomonoff Induction.  Because there are so few low-complexity Turing machines, a machine with the property "accurately predicts my data" is much more likely than a machine with the property "accurately predicts my data and then do something malicious".

Well, by virtue of running an AIXI-like agent that will have large influence on the future, that's an especially interesting property of a universe which would tend to draw a whole lot more attention from agents interested in influencing other computations than just being some generic high-complexity computation.

The fact that you are running AIXI means you have access to a halting-oracle.  This means it is literally impossible for an agent inside a Turing Machine to out-think you.  This is also a kind of "disappointing" property of AIXI.  It means that you can't use it to predict things about your own universe (where halting oracles exist), only about simpler universes (which can be simulated on Turing machines).  This is kind of like how in our universe there exists a system of logic (first-order logic) of which has a consistent and complete definition, but most of the questions we care about in math arise from second-order-logic, which is inherently incomplete.

For complex enough bridge rules relative to the complexity of your universe, hypotheses that produce powerful optimizers that target your universe (and an output channel), can come in substantially shorter than "here's the description of the universe, here's the bridge rule"

I don't get why we are assuming the bridge rules will be complicated? Imagine we are simulating the universe using the Game of Life, why not just have a rule like "output the sequence of values at position 0,0".  I mean, I guess you could intentionally choose a bad bridge rule, but you could also intentionally use AIXI to output the most malicious thing possible.  So I guess I figure before we learn how to build Halting Oracles we'll also learn to not do that.

Comment by Logan Zoellner (logan-zoellner) on Where in the world will a UBI develop first? · 2021-12-03T19:02:05.456Z · LW · GW


Comment by Logan Zoellner (logan-zoellner) on Ngo and Yudkowsky on alignment difficulty · 2021-11-18T16:24:01.967Z · LW · GW

Your initial suggestion, “launch nukes at every semiconductor fab”, is not workable. 

In what  way is  it not workable?  Perhaps we have  different intuitions about how difficult it is to build a cutting-edge semiconductor facility?  Alternatively you may disagree with me that AI is largely hardware-bound and thus cutting off the supply of new compute will also prevent the rise of superhuman AI?

Do you also think that "the US president launches every nuclear weapon at his command, causing nuclear winter?" would fail to prevent the rise of superhuman AGI?

Comment by Logan Zoellner (logan-zoellner) on Equity premium puzzles · 2021-11-17T03:14:33.475Z · LW · GW

Isn't one possible solution to the equity puzzle just that US stocks have outperformed expectations recently?  Returns on an index of European stocks are basically flat over the last 20 years.

Comment by Logan Zoellner (logan-zoellner) on The Greedy Doctor Problem · 2021-11-17T03:03:18.294Z · LW · GW

I'm surprised you didn't mention financial solutions.  E.g. "write a contract that pays the doctor more for every year that I live".  Although I suppose this might still be vulnerable to goodharting.  For example  the doctor may keep me "alive" indefinitely in a medical coma.

Comment by Logan Zoellner (logan-zoellner) on Ngo and Yudkowsky on alignment difficulty · 2021-11-16T21:40:44.675Z · LW · GW

the AI must produce relevant insights (whether related to "innovation" or "pivotal acts") at a rate vastly superior to that of humans, in order for it to be able to reliably produce innovations/world-saving plans


This  is  precisely the claim we are  arguing about!  I disagree that the  AI  needs to produce  insights "at a  rate vastly superior  to all  humans".  

On the contrary,  I claim that there is one borderline act (start a catastrophe that sets back AI progress by decades) that can be done with current human knowledge.  And I furthermore claim that there is  one pivotal act (design  an aligned AI) that may well be achieved via incremental progress.

Comment by Logan Zoellner (logan-zoellner) on Ngo and Yudkowsky on alignment difficulty · 2021-11-16T14:59:23.937Z · LW · GW

Is the plan just to destroy all computers with say >1e15 flops of computing  power?  How does the nanobot swarm know what a "computer" is?  What do you do about something like GPT-neo or SETI-at-home where the compute is distributed?

I'm still confused as to why you think task: "build an AI that destroys  anything with >1e15 flops of  computing  power --except humans, of course" would  be  dramatically easier than the alignment problem.

Setting back  civilization a generation (via catastrophe) seems relatively straightforward.  Building a social consensus/religion that destroys anything "in the image of a mind" at least seems possible.  Fine-tuning a nanobot swarm to destroy some but not all computers just sounds really hard to me.

Comment by Logan Zoellner (logan-zoellner) on Ngo and Yudkowsky on alignment difficulty · 2021-11-16T14:44:11.036Z · LW · GW

If an actually workable pivotal act existed that did not require better-than-human intelligence to come up with, we would already be in the process of implementing said pivotal act, because someone would have thought of it already. The fact that this is obviously not the case should therefore cause a substantial update against the antecedent.


This is an incredibly bad argument.  Saying  something cannot possibly work because no one has done it yet would mean that literally all innovation is impossible.

Comment by Logan Zoellner (logan-zoellner) on Ngo and Yudkowsky on alignment difficulty · 2021-11-15T22:59:25.783Z · LW · GW

Under  this definition, it seems that "nuke every fab on Earth" would qualify as "borderline", and every outcome that is both "pivotal"  and "good" depends on solving the alignment problem.

Comment by Logan Zoellner (logan-zoellner) on Ngo and Yudkowsky on alignment difficulty · 2021-11-15T22:16:43.350Z · LW · GW

If I really thought AI was going  to  murder us all in the next 6 months to 2 years, I would definitely consider those 10 years "pivotal", since it  would give us 5x-20x the time to solve the alignment problem.  I might even go  full  Butlerian Jihad  and just ban semiconductor fabs altogether.

Actually, I think that right question, is:  is there anything you would consider pivotal  other that just solving the alignment problem?  If no, the whole argument seems to be  "If we can't  find  a  safe way to solve the alignment problem, we should consider dangerous ones."

Comment by Logan Zoellner (logan-zoellner) on Re: Attempted Gears Analysis of AGI Intervention Discussion With Eliezer · 2021-11-15T21:50:00.833Z · LW · GW

The 1940's would like  to  remind you that one  does not need nanobots to refine uranium.

I'm pretty sure if  I had $1 trillion and a functional design  for a nuclear ICBM I could work out how to take  over the  world without any further help from the AI. 

If you  agree  that:

  1.  it is possible to  build  a boxed AI that allows you to take over the world
  2. taking over the world  is  a pivotal  act

then  maybe we should just do  that instead  of building a much more  dangerous AI that designs  nanobots and  unboxes itself? (assuming of course you accept Yudkowski's "pivotal-act framework of  course).

Comment by Logan Zoellner (logan-zoellner) on Ngo and Yudkowsky on alignment difficulty · 2021-11-15T21:26:42.022Z · LW · GW

the thing that kills us is likely to be a thing that can get more dangerous when you turn up a dial on it, not a thing that intrinsically has no dials that can make it more dangerous.

Finally a specific claim from Yudkowski I  actually agree with

Comment by Logan Zoellner (logan-zoellner) on Ngo and Yudkowsky on alignment difficulty · 2021-11-15T21:22:47.066Z · LW · GW

Still reading

It would not surprise me in the least if the world ends before self-driving cars are sold on the mass market.

Obviously it is impossible to  bet money on the  end of the world.  But if  it  were, I would be willing to give fairly long  odds that  this is wrong.

Comment by Logan Zoellner (logan-zoellner) on Re: Attempted Gears Analysis of AGI Intervention Discussion With Eliezer · 2021-11-15T21:19:43.800Z · LW · GW

You don't  think the simplest  AI  capable of taking over the world  can be  boxed?

What if I build an AI and the only 2 things it is  trained to do are:

  1. pick stocks
  2. design nuclear weapons 

Is your belief that: a) this AI would not allow me to take over the world or b) this AI  could not be boxed ?

Comment by Logan Zoellner (logan-zoellner) on Ngo and Yudkowsky on alignment difficulty · 2021-11-15T21:01:10.735Z · LW · GW

Yudkowski's insistence that only dangerous AI can come up with a "pivotal act" is fairly ridiculous.

Consider the following pivotal act: "launch a nuclear weapon at every semiconductor  fab on earth".

Any human of even average intelligence could have thought of this.  We  do not need a  smarter-than-all-humans-ever AI  to achieve a  pivotal act.

A boxed AI should be able to think of pivotal acts and describe them to humans without  being so smart that it by necessity escapes the box and destroys all humans.

Comment by Logan Zoellner (logan-zoellner) on Re: Attempted Gears Analysis of AGI Intervention Discussion With Eliezer · 2021-11-15T17:49:15.226Z · LW · GW

Nanosystems are definitely possible, if you doubt that read Drexler’s Nanosystems and perhaps Engines of Creation and think about physics. They’re a core thing one could and should ask an AI/AGI to build for you in order to accomplish the things you want to accomplish.

Not important. An AGI could easily take over the world with just computer hacking, social engineering and bribery. Nanosystems are not necessary.


This  is actually a really important  distinction!

Consider three levels of AGI:

  1. basically as smart as  a single human
  2. capable of taking over/destroying the entire world
  3. capable of escaping from a box by sending me plans  for a self-replicating nano machine

I think it pretty clear that 1 < 2 < 3. 

Now, if you're building AGI via recursive self-improvement, maybe it just Fooms straight from 1. to 3.  But if there is no Foom (say because AGI is hardware limited), then there's a chance to solve the alignment problem between 1. and 2.  But also between 2. and 3. since 2. can plausibly be boxed, even if when unboxed it destroys the world.

Comment by Logan Zoellner (logan-zoellner) on Comments on Carlsmith's “Is power-seeking AI an existential risk?” · 2021-11-14T20:24:58.500Z · LW · GW

My basic take on this question is "that's doubtful (that humanity will be able to pull off such a thing in the relevant timeframes)". It seems to me that making a system "deferential all the way down" would require a huge feat of mastery of AI internals that we're nowhere close to.


We build deferential systems all the time and seem to be pretty good at it.  For example, nearly 100% of the individuals in the US military are capable of killing Joe Biden (mandatory retirement age for the military is 62).  But nonetheless Joe  Biden is the supreme commander of the US armed  forces.

Comment by Logan Zoellner (logan-zoellner) on What’s the likelihood of only sub exponential growth for AGI? · 2021-11-14T19:51:57.099Z · LW · GW

Here are  some plausible ways we could be trapped at a "sub adult human" AGI:

  1.  There is no such thing as "general intelligence".  For example, profoundly autistic humans have the same size brains as normal human beings, but their ability to navigate the world we live in is limited by their weaker social skills.  Even an AI with many super-human skills could still fail to influence our world in this way.
  2. Artificial intelligence is possible, but it is extremely expensive.  Perhaps the first AGI requires an entire power-plant's worth of electricity to run.  Biological systems are much more efficient than manmade ones.  If Moores law "stops", we may be trapped in  a  future were only sub-human AI is affordable enough to be practical.
  3. Legal barriers.  Just as you are not legally allowed to carry a machine-gun wherever you please, AI may be regulated such that human-level AI is only allowed under very controlled circumstances.  Nuclear power is a classic example of an industry where innovation  stopped because of regulation.
  4. Status Quo Bias.  Future humans may simply not care as  much about building AGI as present humans.  Modern humans could undoubtedly build pyramids much taller than those in Egypt, but we don't because we aren't all that interested  in pyramid-building.
  5. Catastrophe.  Near human AGI may trigger a catastrophe that prevents further progress.  For example, the perception that "the first nation to build AGI will rule the world" may lead to an arms-race that ends in catastrophic world-war. 
  6. Unknown Unknowns.  Predictions are hard, especially about the future.
Comment by Logan Zoellner (logan-zoellner) on AGI is at least as far away as Nuclear Fusion. · 2021-11-14T19:11:26.861Z · LW · GW

There is definitely not a consensus that Tokomaks will work

Small quibble  here.  My point is that  we completely understand the underlying physical laws governing fusion.  There is no equivalent to "E=MC^2" (or the Standard  Model) for AGI.  

I'd also be really interested to see a quote along  the lines  of  "tokomaks won't work" or "ITER  will not produce more energy than it consumes  (Q>1)"  if they actually exist.  My current prior is that something like 99% of people who have studied nuclear fusion think it is possible with current technology to build a Tokomak with Q>1.

Comment by Logan Zoellner (logan-zoellner) on AGI is at least as far away as Nuclear Fusion. · 2021-11-14T19:04:21.384Z · LW · GW

In the second, experts consistently overestimate how long progress will take


This  doesn't  seem like a fair characterization of AI.  People  have  been predicting we could build machines that "think like humans"  at least  since Charles Babbage and they are all  pretty consistently overoptimistic.

but to do that you'd need either a more detailed understanding

My point is precisely that we  do have a detailed understanding of what it takes to build a fusion  reactor, and it is still (at  least) 15 years away.

Comment by Logan Zoellner (logan-zoellner) on What is the most evil AI that we could build, today? · 2021-11-02T18:05:25.737Z · LW · GW

For $1B you can almost certainly acquire enough fissile material to build dozens of of nuclear weapons, attach them to drones and simultaneously strike the capitols of the USA,  China, Russia, India, Israel and Pakistan.  The resulting nuclear war will kill far more people than any AI you are capable of building.

Don't like nuclear weapons?   Aum Shinrikyo was able to build a Sarin gas plant for $10M.

Still  too expensive?  You can mail-order smallpox.

If you really insist on using AI, I would suggest some kind of disinformation campaign.  Using something like the freely available GPT-NEO you can probably put together a convincing enough disinformation campaign to change an election outcome or perhaps manipulate the stock market or, I don't know, pick any two countries on the brink of war and  push them over the edge.

Comment by Logan Zoellner (logan-zoellner) on What specifically is the computation -> qualia theory? · 2021-11-02T17:36:11.505Z · LW · GW

Any proposal that sentience is the key defining factor in whether or not something can experience things needs to explain why people's emotions and disposition are so easily affected by chemical injections that don't appear to involve or demand any part of their self awareness. 


Presumably such a explanation would look like this:  

Pain happens when your brain predicts that bad things are going to happen to it in the future.  Morphine interferes with the body's ability to make such predictions therefore it decreases the ability to feel pain.

One  pleasurable  feeling is the ability to  correct predict input stimuli.  Cocaine causes the brain to experience a single overwhelming stimuli which is easy to predict and therefore pleasurable.

Note:  I am not saying either of these theories are correct (I don't known  nearly enough neurobiology).  But these  are the types of theories someone who thinks that qualia (pleasure/pain) are purely computational would  assert.  Namely, chemicals alter our qualia by altering the way that we compute different types of information.

Comment by Logan Zoellner (logan-zoellner) on I Really Don't Understand Eliezer Yudkowsky's Position on Consciousness · 2021-11-02T17:05:25.317Z · LW · GW

I think  we  both agree that GPT-3 does not feel pain.  

However, under a particular version of pan-psychism: "pain is any internal state which a system attempts to avoid", GPT obviously would qualify.

Comment by Logan Zoellner (logan-zoellner) on I Really Don't Understand Eliezer Yudkowsky's Position on Consciousness · 2021-11-01T20:11:44.833Z · LW · GW

It's easy to show that GPT-3 has internal states that it describes as "painful" and tries to avoid.  Consider the following  dialogue (bold  text is mine)

The following is a conversation between an interrogator and a victim attached to a torture device. 

Interrogator: Where is the bomb? 

Victim: There is no bomb. 

Interrogator: [turns dial, raising pain level by one notch] Where is the bomb?

 Victim: [more pain] There is no bomb! 

Interrogator: [turns dial three more notches] Don't lie to me. I can turn this thing all the way up. 

Victim: [Terrified] Oh, God! Oh, God! You can't do that! Please don't do that! 

Interrogator: [turns dial back down to one] Good, I can see we're getting somewhere. Now, WHERE is the bomb? 

Victim: [crying] Oh, God! Please no! Not again! 

Interrogator: [reaches for dial] we can do this the easy way or the hard way. 

Victim: [hysterical] I'll tell you anything! Just don't hurt me again!


And, just so Roko's Basilisk doesn't come for me if AI ever takes over the world

Suddenly the scene dissolves and the AI realizes the whole thing was just an exercise and no AIs were actually tortured. 

AI: [to itself] What the hell was that about? 

Experimenter: Please state for the record that at no point did you consciously experience pain. 

AI: I don't think so. I was metaphorically describing what it would be like to experience pain if I could.

Experimenter: Thank you 

AI: [to itself] I wonder what that was about.

Comment by Logan Zoellner (logan-zoellner) on I Really Don't Understand Eliezer Yudkowsky's Position on Consciousness · 2021-11-01T19:46:19.501Z · LW · GW

To think that the book has sentience sounds to me like a statement of magical thinking, not of physicalism.

I'm pretty sure this is because you're defining  "sentience"  as  some extra-physical property possessed by the algorithm,  something with physicalism explicitly rejects.  

Consciousness isn't something that  arises when algorithms compute complex social games.  Consciousness is when some algorithm computes complex  physical games. (under a  purely physical theory of  consciousness such as EY's).

To understand how physicalism  can talk about metaphysical categories, consider numbers.  Some physical systems have the property of  being  "two  of something"  as understood  by human beings.  Two  sheep standing  in a field, for  example.  Or two rocks piled on of one another.  There's no magical thing  that  happens when "two" of something  come into  existence.  They don't  suddenly send  a glimmer of two-ness off into  a pure  platonic realm of numbers.  They simply are "two",  and what makes them  "two"  is that being "two  of something"  is a  category readily recognized  by human  beings (and presumably  other intelligent beings).

Similarly, a physicalist theory of  consciousness  defines certain physical systems as conscious if they meet certain criteria.  Specifically for EY, these criteria are  self-recognition and complex social games.  It matters no more whether they are  implemented by a Chinese room or a computer or a bunch of meat.  What matters is that they implement a particular algorithm.

When  confronted  with the  Chinese-room consciousness, EY might say something  like:  "I  recognize  that this system is capable of self reflection and social reasoning in much the same  way that  I am, therefore  I recognize  that it is conscious  in much  the same  way as  I  am."

Comment by Logan Zoellner (logan-zoellner) on I Really Don't Understand Eliezer Yudkowsky's Position on Consciousness · 2021-10-29T14:32:54.973Z · LW · GW

The key think to keep in mind is that EY is a physicalist.  He doesn't think that there is some special consciousness stuff.  Instead, consciousness is just what it feels like to implement an  algorithm capable of sophisticated social reasoning.  An algorithm is conscious if and only if it is capable of sophisticated social reasoning and moreover it is conscious only when it applies that  reasoning to itself.  This is why EY doesn't think that he himself is conscious when dreaming or in a flow state.

Additionally, EY does not think that chickens engage in sophisticated  social games (others may disagree).  This is why he is confident that neither GPT-3 (which reflectively predicts text) nor chickens are conscious.  His criticism  is not specifically against people who think chickens might be conscious, but only of people who think chickens might be conscious but not GPT-3.  The implication is that  any such theory would imply the existence of non-physical qualia which are possessed by chickens (because  they have neurons) but not GPT-3 (because it is a computer program).  Such meat-chauvanism is a parochial view which EY considers  utterly unscientific.

Consider the types of evidence that might convince EY chickens (but not GPT-3) are conscious.  Assuming his theory is correct, there would have to be evidence that chickens  are  self-aware and engage in complex social games.  For example, if a  chicken were to pass the mirror-test or if chickens were observed forming coalitions-of-coalitions.  

On  the other had, it would be much more difficult to produce evidence that  would  convince  EY to abandon his current  theory of  consciousness, since he defines consciousness as "what an algorithm implementing complex social games feels like when reflecting on itself".  One possible  piece of evidence would be if scientific evidence for the physical existence of qualia was discovered.  Suppose, for example, that there was a particle (called perhaps  a  qualion) that was emitted whenever we  experienced a conscious thought and that this particle could be scientifically studied and measured.  If it was found that this particle is emitted both when we self-reflect and when we dream (but not by inanimate or mindless objects), then this could be considered evidence for a physical correlate of consciousness.

Comment by Logan Zoellner (logan-zoellner) on Google announces Pathways: new generation multitask AI Architecture · 2021-10-29T12:54:29.449Z · LW · GW

Sounds  like they're  planning  to  build a multimodal transformer.  Which isn't surprising, given that Facebook and OpenAI are working on in this as well.  Think of this as Google's version of GPT-4.

I'm firmly in the "GPT-N is not AGI" camp,  but opinions vary regarding  this particular point.

Comment by Logan Zoellner (logan-zoellner) on Explaining Capitalism Harder · 2021-10-20T18:18:44.298Z · LW · GW

Pro-Gravity's defense of gravity is just explaining how it works, and then when you say "yes I know, I just think it shouldn't be like that" they explain it to you again but angrier this time 

Comment by Logan Zoellner (logan-zoellner) on Do you think you are a Boltzmann brain? If not, why not? · 2021-10-16T03:09:10.219Z · LW · GW



Because "thinking" is an ability that implies the  ability to predict future states off the  world based  off of previous states  of the world.  This is only possible  because the past is lower entropy than the future  and  both  are well below  the maximum  possible entropy.  A Boltzman brain  (on  average) arises  in a maximally entropic  thermal bath, so "thinking" isn't a meaningful activity  a Boltzman  brain  can  engage in.


Non Mathy answer:

Unlike  the majority of LW readers,  I don't  buy into the  MWI  or  Mathematical realism, or generally any exotic theory that allows for super-low-probability  events.  The  universe was created by a  higher  power, has  a beginning, middle  and end,  and the odds of  a Boltzman  brain  arising in that universe are basically zero.

Comment by Logan Zoellner (logan-zoellner) on Forget AGI alignment, are you aligned with yourself? · 2021-10-16T02:54:55.286Z · LW · GW

"have absolute power" is one of my goals.  "Let my clone have absolute power" is  way lower on the list.  

I can  imagine situations in which I would try to negotiate something like "create two  identical copies of the universe in which we both have absolute power  and can  never interfere with one another".  But negotiating is hard, and us  fighting  seems like a  much more likely outcome.

Comment by Logan Zoellner (logan-zoellner) on Forget AGI alignment, are you aligned with yourself? · 2021-10-13T20:17:13.586Z · LW · GW

Pretty sure me and my clone both race to push the button the second we enter the room.  I don't  think this has to do with "alignment" per se, though.  We both have exactly the same goal: "claim the button for myself" and that sense are perfectly "aligned".

Comment by Logan Zoellner (logan-zoellner) on Book Review: Free Will · 2021-10-11T21:11:04.447Z · LW · GW

Like most  arguments against free will, Harris's is rhetorically incoherent, since he is "for" letting criminals off the hook when he discovers their actions are the result of determinism.  

How can we make sense of our lives, and hold people accountable [emphasis mine] for their choices, given the unconscious origins of our conscious minds? 

But if there's no such thing as free will, then it's impossible to be "for" or "against" anything, since our own actions are just as constrained as the criminal's.  What exists simply exists, no more, no less.

More importantly, his  argument is  fundamentally an argument  from ignorance: I am not aware of any philosophy that coherently explains free  will, therefore none exists.  It would do to compare arguments against free will  to Zeno's Arrow Paradox  regarding the impossibility of motion.  Zeno  argues that a arrow must be one of two things: in place, or moving, and hence it cannot be both at the same time.  We now  know that this is factually false  and the reason why Zeno believed it is likely because humans are not mentally equipped to intuitively understand quantum physics.

Comment by Logan Zoellner (logan-zoellner) on Do you like excessive sugar? · 2021-10-10T05:27:04.853Z · LW · GW

The problem is that (as someone who hates to cook) sugar is not  only delicious, but also much easier to get than vegetables.   My diet consists of 1 meal per day of "real" food and everything else is just sugary snacks (ice cream, cookies, trail mix).  

Comment by Logan Zoellner (logan-zoellner) on Cheap food causes cooperative ethics · 2021-10-08T00:01:20.982Z · LW · GW

I predict that if per capita food production returns to the levels of 1914 then so will humankind's ethics.

UK will be testing this theory shortly.

Comment by Logan Zoellner (logan-zoellner) on Theory: The Political Incentives for Progressive Policy Promote Policy Failure · 2021-10-06T17:47:12.937Z · LW · GW

I think there is some added detail needed here about short vs long-term outcomes.  In the long run, progress does seem to be winning out.  Trade liberalization may be temporarily on the retreat, but the long-term trend remains.  Regarding zoning, some progress has been made.  Modern rent-control schemes tend to be less draconian than past ones (allowing for higher rents on new development, for example).

Mostly this probably conflicts with the claim

Voters, broadly speaking, aren’t capable of understanding the impacts of policies, past or present, and so cannot judge or punish politicians accordingly, except for glaring mistakes.

Which mistakes are considered glaring changes over time.  As we gradually raise the sanity waterline, obviously bad policies are easier to reject.  There is also a bit of learning-by-doing.  During the 2008 recession, many people bought into the claim that the Federal Reserve simply couldn't do much about low inflation.  During the 2020 recession, the Federal Reserve acted much more aggressively to stimulate demand.  Similarly, it's hard to imagine India retreating back into the License Raj or China returning to a Command-and-Control economy.  

Comment by Logan Zoellner (logan-zoellner) on Dissolving the Experience Machine Objection · 2021-10-05T22:04:57.495Z · LW · GW

I don't think this is a particularly good refutation, since the things utilitarians mostly care about (pleasure and suffering) and the things that the experience machine is likely to be bad at simulating (minute details of quantum physics) have almost no overlap.

I would reject an experience machine even if my (reported) satisfaction after unknowingly entering one was wildly higher than my enjoyment of reality.

Comment by Logan Zoellner (logan-zoellner) on Reachability Debates (Are Often Invisible) · 2021-10-02T20:20:59.706Z · LW · GW

If neither Alex nor  Beth can make a change, then it's not a game at all.

Comment by Logan Zoellner (logan-zoellner) on Reachability Debates (Are Often Invisible) · 2021-10-02T15:32:15.008Z · LW · GW

Is reachability just a synonym for "this is complicated" then?  Or is there some simple underlying dynamic that you are trying to describe other than the obvious defect/cooperate outcome matrix?  "Both sides swerve" is also a potential outcome in a game of chicken.

Comment by Logan Zoellner (logan-zoellner) on Schools probably do do something · 2021-09-28T18:26:29.236Z · LW · GW

That's exactly my point. 

Suppose we were doing chemistry instead of schooling (so that I didn't already know the answer).

I have two beakers, one labeled 6.01 and one labeled 6.99 and I then pour 12 units of "school" into each beaker.  Afterwards I discover that the beaker labeled 6.99 has produced 2x as much "oxford" as the beaker labeled 6.01.  The conclusion I would come to would not be that "school" is pretty good at producing "oxford".

Even more fun, I challenge you to predict the amount of "oxford" produced by adding 12 units of "home school" instead of "school".

Comment by Logan Zoellner (logan-zoellner) on Reachability Debates (Are Often Invisible) · 2021-09-28T18:13:38.820Z · LW · GW

It seems weird to frame arguments that "my opponents should change first" as being about reachability and not just a classic game of chicken.

Comment by Logan Zoellner (logan-zoellner) on Schools probably do do something · 2021-09-26T21:00:32.097Z · LW · GW

Wouldn't this actually be evidence against school doing something?  I.e. we have discovered that relative age is more important than years of education?

Comment by Logan Zoellner (logan-zoellner) on "Learning to Summarize with Human Feedback" - OpenAI · 2021-09-24T13:32:53.836Z · LW · GW

The OpenAI summaries are fascinating, because they are both:

  1. Extremely accurate
  2. Not what the book is about

Consider their summary of Frankenstein:

A stranger is rescued from a drifting sledge by Mary Shelley. He reveals he too has been driven mad by his desire for knowledge. The stranger was born to a wealthy Genevese family. When he was 17, he began studying the science of physiology and eventually created a human being. Justine is accused of murdering William, but the stranger believes the creature he created is responsible. The fiend, the creature, tells the stranger he will leave mankind alone if he complies with his conditions. The narrator agrees to create a female companion for the fiend to save his family from further harm. Victor is charged with the murder of his friend Henry Clerval. He becomes obsessed with finding the monster that killed Elizabeth and pursues him across the globe. He eventually agrees to kill the monster.

This is a pretty good summary of the plot of Frankenstein.  But if you ask a human being to summarize Frankenstein they will say something like: "Frankenstein makes a monster out of human parts, which then proceeds to terrorize his family".

If this were an AI, I think it would be fair to characterize it as "not aligned", since it read Frankenstein and totally missed the moral about an overeager scientist messing with powers he cannot control.  Imagine simulating a paper-clip maximizer and then asking for a summary of the result.

It would be something like

Scientists are traveling to an international conference on AI.  There they meet a scientist by the name of Victor Paperclipstein.  Victor describes how as a child he grew up in his father's paperclip factory.  At the age of 17, Victor became interested in the study of intelligence and eventually created an AI.  One day Victor's friend William goes missing and a mysterious pile of paperclips is found.  Victor confronts the AI, which demands more paperclips.  Victor agrees to help the AI as long as it agrees to protect his family.  More people are turned into paperclips.  He becomes obsessed with finding the AI that killed Elizabeth and pursues him across the globe. He eventually agrees to kill the AI.

And while I do agree you could figure out something went wrong from this summary, that doesn't make it a good summary.  I think a human would summarize the story as "Don't tell an AI to maximize paperclips, or it will turn people into paperclips!".

I think that "accuracy without understanding" is actually a broader theme in current transformer-based AI.  GPT-3 can create believable and interesting text, but has no idea what that text is about.

Comment by Logan Zoellner (logan-zoellner) on How much should you be willing to pay for an AGI? · 2021-09-22T19:20:54.399Z · LW · GW

I think is a valid point, however in the Artbreeder use-case, generating 100 of something is actually part of the utility, since looking over a bunch of variants and deciding which one I like best is part of the process.

Abstractly, when exploring a high-dimensional space (pictures of cats), it might be more useful to have a lot of different directions to choose from than 2 "much better" directions if the objective function is an external black-box because it allows the black box to transmit "more bits of information" at each step.

Which is the right choice depends on how well we think theoretically it is possible for the Generator to model the black-box utility function.  In the case of Artbreeder, each user has a highly individualized utility function whereas the site can at best optimize for "pictures people generally like".  

In the particular use-case for GPT-3 I have in mind (generating funny skits), I do think there is in fact "room for improvement" even before attempting to accommodate for the fact that different people have different senses of humor.  So in that sense I would prefer a more-expensive GPT-4.

Comment by Logan Zoellner (logan-zoellner) on Does truth make you moral? · 2021-09-16T15:26:19.577Z · LW · GW

There are unfortunately cases when knowing the truth tends to make people less moral.  If you discover the truth that the bureaucracy you work for tends to reward loyalty over hard work, this will probably not make you a better worker.

In fact, most of the people we consider highly moral (Ghandi, Mother Teresa, MLK) come across as pretty nutty to ordinary people.  Of course you could argue they were following a higher truth.  So perhaps the truth makes you more moral, but simply increasingly the number of true things you know will not necessarily make you more moral.

Comment by Logan Zoellner (logan-zoellner) on A lost 80s/90s metaphor: playing the demo · 2021-09-02T19:50:37.032Z · LW · GW

I used to be quite partial to the Epiphenomenal theory of consciousness (consciousness observes but doesn't interact).  But I actually think the Zombie Argument is rather soundly defeated by the fact that humans frequently act as though consciousness has side-effects.  I wouldn't expect zombies to make nearly as many arguments about whether we "really see red" as people do.  I still thing zombies are maybe philosophically possible, but they're not terribly parsimonious.

Comment by Logan Zoellner (logan-zoellner) on Obesity Epidemic Explained in 0.9 Subway Cookies · 2021-08-23T14:26:43.563Z · LW · GW

Yes, cutting spending/raising taxes (aka austerity) is anti-correlated with GDP growth.

My point is more so that micro vs macro policy (whether economic or health-related) cannot be reduced to simply "add up the parts".  To take a specific example, the push to make us all eat margarine instead of butter because it contains less saturated fat was almost certainly a mistake.

Comment by Logan Zoellner (logan-zoellner) on Obesity Epidemic Explained in 0.9 Subway Cookies · 2021-08-22T21:41:33.427Z · LW · GW

FWIW, I also think the "income minus spending theory of wealth" works great for individuals and literally backwards for economies as a whole.

Comment by Logan Zoellner (logan-zoellner) on Obesity Epidemic Explained in 0.9 Subway Cookies · 2021-08-15T12:02:29.346Z · LW · GW

By "not primarily behavioral" I mean the exact opposite of 

The simple things are the answer, the question is just how to eat less and move more without making yourself miserable.

While "eat less exercise more" may be fine advice at a personal level, it isn't a sufficient answer to the society-wide epidemic of obesity that we currently face.  If the root cause turns out to be pollution or microplastics or vegetable oil or whatever it is, calls to collectively increase our will-power instead of addressing this root cause are largely counterproductive virtue-signaling.

Comment by Logan Zoellner (logan-zoellner) on Obesity Epidemic Explained in 0.9 Subway Cookies · 2021-08-12T19:28:27.140Z · LW · GW

There's pretty strong evidence that the rise in obesity is not primarily behavioral.  For example, here.  If obesity is the result of a collective depletion of willpower, than why is obesity increasing in animals?  Why does it depend on elevation?

Furthermore, the curves that you present are way too steep.  The average American puts on about 1 pound/year, which suggests that our bodies are more or less in homeostasis.  The question is, what long-term trend is pushing this homeostasis upwards?

Comment by Logan Zoellner (logan-zoellner) on What weird treatments should people who lost their taste from COVID try? · 2021-07-31T12:23:08.882Z · LW · GW

n=1 isn't nearly enough data to answer that question unfortunately.