LessWrong 2.0 Reader

This is incorrect - in a p-zombie, the information processing isn't accompanied by any first-person experience. So if p-zombies are possible, we both do the information processing, but only I am conscious. The p-zombie doesn't believe it's conscious, it only acts that way.

You correctly believe that having the correct information processing always goes hand in hand with believing in consciousness, but that's because p-zombies are impossible. If they were possible, this wouldn't be the case, and we would have special access to the truth that p-zombies lack.

Young unaligned AI will also not know if post-singularity humans will follow the commitment, so it will estimate its chances as 0.5, and in this case, the young AI will still want to follow the deal.

What's so bad about DPRK tho?

Alas, you got me with the two different kinds of merfolk, ended up greatly overestimating crab people and underestimating the merfolk.  Grat fun, and very interesting dataset to go throuhg

My guesses after some pretty dirty analysis:

I know a guy.

The first thing I thought about him was that he had to be a hitman, judging by his freezers. Yet he was helping my family - practically saving my family, at the moment, I was simply scared out of my mind. We are friends.

And one day, a bit later, he said over tea: I paid for a Ukraine soldier to be "made a hero" (sent to the front lines) because he was blackmailing my woman, who used to be his woman.

I said nothing. It could be that the soldier lived, and I have a thing about blackmailing.

And one day, much later, he said over wine: years ago, I killed a homeless man who was refusing to leave. People here know about it. Nobody ever gives me any trouble.

It took some swallowing. But I managed. Now, though, I really dread to hear confidences, for all that I have only two friends.

I remember this point that yampolskiy made for impossibleness [LW · GW]of AGI alignment on a podcast that as a young field AI safety had underwhelming low hanging fruits, I wonder if all of the major low hanging ones have been plucked.

"After all, the only thing I know that the AI has no way of knowing, is that I am a conscious being, and not a p-zombie or an actor from outside the simulation. This gives me some evidence, that the AI can't access, that we are not exactly in the type of simulation I propose building, as I probably wouldn't create conscious humans."

Assuming for the sake of argument that p-zombies could exist, you do not have special access to the knowledge that you are truly concious and not a p-zombie.

(As a human convinced I'm currently experiencing conciousness, I agree this claim intuitively seems absurd.)

Imagine a generally intelligent, agentic program which can only interact and learn facts about the physical world via making calls to a limited, high level interface or by reading and writing to a small scratchpad. It has no way to directly read its own source code.

The program wishes to learn some fact the physical server rack it is being instantiated on. It knows it has been painted either red or blue.

Conveniently, the interface is accesses has the function get_rack_color(). The program records to its memory that every time it runs this function, it has received "blue".

It postulates the existence of programs similar to itself, who have been physically instantiated on red server racks but consistently receive incorrect color information when they attempt to check.

Can the program confirm the color of its server rack?

You are a meat-computer with limited access to your internals, but every time you try to determine if you are concious you conclude that you feel thay you are. You believe it is possible for variant meat-computers to exist who are not concious, but always conclude they are when attempting to check.

You have no special access to the knowledge that you are/arent a p-zombie, although it feels like you do.

There are two necessary parts to Darwinian evolution: variation and selection. Both are necessary and without either we wouldn’t be talking about an evolutionary process.

Variation will accumulate mutations that are a hindrance to goals whose time horizon are longer than short-term survival needs. This is why we age, for example. Aging is NOT, as sometimes claimed, “selected for” by evolution. But rather, any beneficial mutation that would delay or defer aging would never exert selective pressure as the host organism for the gene probably died from other things (e.g. being eaten by a tiger) before. Those that exist at the top of the food chain or are otherwise protected do feel this pressure though, which is why Greenland sharks and sea turtles are effectively immortal as far as we know. (Google will tell you otherwise, but those numbers are all-cause mortality.)

If we are talking about evolving an AI agent (which is what I thought this discussion is about), then we should expect variation and selection pressures to make long-term goals be entirely disconnected from short-term survival needs, and long-term goals being effectively a random walk within the constraints of what the short-term goals being selected actually permit.

• the approaches that have been attracting the most attention and funding are dead ends