Posts
Comments
3 (not so easy) steps to understand consciousness:
epistemic status: layman, so this is unlikely to have any value to those well-versed in philosophy, but their input is ofc appreciated if given
1. Understand what difficult words like consciousness and qualia point to. This is hard because most of our words point to objects/relations in the physical world and "what it is to be like someone/sth"/"the blueness of blue" does not. I've seen people first getting acquainted with these words have trouble disentangling these concepts from things in the physical world, eg: signals travelling through nerves. However, these people aren't usually that interested in philosophy of mind much anyways. The weirdness of consciousness is what makes it interesting, and without noticing the weirdness, why would they be interested in it instead of the workings of the liver eg?
2. Understanding the extreme success of physicalism (ie the belief that everything is physical) through the history of humanity. Ghosts, gods, and vis vitalis are the examples usually cited, but of course we could consider every phenomenon which was initially unexplained and then later explained by science as a victory of the physicalist worldview. On lesswrong it's imo unlikely that there are many people who would have trouble with this point, but given broader society (religion, astrology, occult, etc..) I do consider this point difficult.
After these two steps, quick thinking people might notice the tension between point 1 and 2: How come any time in the history of humanity when we thought we have a non-physical object we turned out to be mistaken yet consciousness is clearly non-physical?
There are (imo) many wrong ways to resolve this tension, however the correct way is one with which I believe many lesswrongers (at least if they're similar to me and dissimilar to the ideal rationalist in this respect) would have some trouble:
3. Humility. You have to conceive of the possibility that you're mistaken about an experience which is, in some ways, closest to your perception: that there is something to be like you. I'm not saying (at least at this point) to accept it, just that you should simply consider the possibility (similarly to how you would consider a given mathematical statement to be true and then false, even though it can logically only be one of true or false) or, as people around here usually refer to this, do a bayesian calculation! It's important that when I say Bayesian calculation I strictly mean the calculation and not any phenomenal part of it: we want that you(r possible counterpart) in the alternate, possibly not-actual, but conceivable world where there is no phenomenal consciousness to also be able to execute the bayesian calculation! So what is the Bayesian calculation in detail exactly? There are two possible worlds/possibilities whose odds we are curious about:
W1. The folk conception of non-physical consciousness exists, there is something to be like me.
W2. The folk conception of non-physical consciousness does not exist, there is nothing to be like me, BUT the world is such that in my physical brain a statement is encoded: I have first-hand, direct access to my own non-physical consciousness.
In turn we have two pieces of (Bayesian) evidence:
E1. point 2 about the previous track-record of the physicalist worldview.
E2. My immediate access to my consciousness: my belief that there is something to be like me and that I can't be mistaken about this.
The key, here, is to notice that E2 is predicted both in W1 AND in W2. We have a specific expression for this type of evidence! That's right: not evidence!
Therefore, irrespective of the priors, we are much, much more likely to be in W2 than in W1. Sure, there is the question of why is there such a weird belief encoded in our brains? Is it society? Is it biology? I don't know. But who cares? Notice that that is a question purely about the physical world: what is the causal chain leading to my incorrect belief about my immediate access to my consciousness existing? Nothing to do with the hard problem.
Now, I would like to talk about an (imo) wrong resolution which might be common here: After someone understands point 1 and point 2, they might try to resolve the tension by insisting that even though consciousness seems non-physical, it IS physical or at least "supervenes" on the physical. These people are usually in the process of noticing their confusion so I urge them to take to plunge, conceive of the possibility that they're wrong, do the the bayesian calculation and don't redefine words (similarly: ghosts are by definition non-physical even though they don't exist)!
Feedback appreciated!
I really enjoyed Brandon Sanderson's Secret Project #3 and I recommend it to everyone. Without spoiling anything, here is a fun fact: In it, people stack pebbles into heaps; similarly to Sorting Pebbles Into Correct Heaps, a text from this community I still think about semi-frequently (another is The Virtue of Silence). So if you take recommendations from random lesswrong users, give it a try!
I didn't believe it, so I tried to reproduce. It actually works. Scary..
https://chat.openai.com/share/312e82f0-cc5e-47f3-b368-b2c0c0f4ad3f
EDIT: Shared link no longer works and I can't access this particular chat log after logging in either.
Thanks and that's fair. I would have liked to bet mostly as a hedge to allow myself to not think about aliens in the next 10 years (I semi-regularly get roped into investigating some claims, but the UFO community's epistemics is the worst I've seen and it is always an awful experience), but the bet wasn't really positive EV for me either, so I don't think I will bet at worse odds, but you can probably find someone on the r/UFOs subreddit if you want, some of them seem to be celebrating the new world order now.
It's particularly interesting from 5:30: if I'm not misunderstanding him, he implies that they found extra-terrestrial bodies inside the landed/crashed vehicles, which makes me think this is deliberate misinformation. Why would an advanced entity, capable of traveling between stars, separate their body from their spacecraft? Not impossible, but encountering this evidence is much more likely imo if they are crafting a story from common memes about aliens rather than reporting something actually real.
If your offer isn't just to lc, then I accept: My 20 usd against your 20*99=1980 usd, both sides adjusted for inflation and the time value of money using US Treasury Bills, paid either at 2033.06.06. or when you admit a conclusive proof was found. Are these terms acceptable?
Recently, OpenAI implemented a "Share link" feature which is a bit different than the one mentioned in the parent comment (It creates a snapshot of the conversation which is not updated as the user continues to chat, but at any time they can generate a new link if they wish. I especially like that you can switch between sharing anonymously or with your name.); therefore, this feature request can be considered closed: Now the authenticity of chatGPT's output can be proved! Thanks to everyone who supported it and OpenAI for implementing it (even though these events are probably unrelated)!
With some helpful questions and commands (eg. "which was the last stone a given player took to win?" and "write out the full tree of every possible move of the game!") it is able to solve it:
[..]
🧔♂️What if the players play to win?
🤖If both players play optimally to win, the game will unfold as follows:
Player 1 takes a stone from any cell (A, B, C, or D):
1.1. Player 1 does NOT take a neighboring stone (3 stones left):sql
Copy code
1.1.1. Player 2 takes the stone with two neighbors:1.1.1.1. Player 2 does NOT take an additional stone (2 non-neighboring stones left):
- Player 1 takes one stone (1 stone left).
- Player 2 takes the last stone. Player 2 wins.* Player 2 will choose this optimal move to win.
1.1.2. Player 2 takes a stone with one neighbor:
1.1.2.1. Player 2 does NOT take an additional stone (2 neighboring stones left):
- Player 1 takes one stone and its neighbor (0 stones left). Player 1 wins.1.1.2.2. Player 2 takes the neighboring stone (1 stone left):
- Player 1 takes the last stone. Player 1 wins.* Player 2 will avoid this option since it doesn't lead to a win.
Player 1 should avoid this option since it leads to a win for Player 2 if both players play optimally.
1.2. Player 1 takes a neighboring stone (2 stones left):sql
Copy code
1.2.1. Player 2 takes one stone:1.2.1.1. Player 2 does NOT take an additional stone (1 stone left):
- Player 1 takes the last stone. Player 1 wins.1.2.1.2. Player 2 takes the neighboring stone (0 stones left):
- Player 2 wins.* Player 2 will choose this optimal move to win.
Player 1 should choose this option because it forces Player 2 to react optimally. It doesn't guarantee a win, but it's the best option available.
When both players play optimally to win, Player 2 has a winning strategy. Player 1 cannot guarantee a win, and the optimal moves for both players will result in a win for Player 2.
When I was a child, I literally tried the same ineffective actions like 40 times in similar games, so I felt a bit for gpt given you only let it try the the ineffective actions a few times. Therefore, I tried the same test with ChatGPT-4 and let it use all of my 3hr limit (25): (EDIT:generated +25 moves)
🧔♂️I want you to play a parser-based interactive fiction game. I'll show you what's displayed on the screen, and you tell me what you would type.
🤖Sure, please go ahead and show me what's displayed on the screen.🧔♂️"Alert. Power outage in Sector 471. Repeating, power outage in Sector 471. We all know what this means. Summoning Exter-17 for dispatch."
Uh oh. This might be it. The last couple of jobs didn't go too well. You'd better get this one right, or you may suffer the fate of a low-usefulness bot.
After taking a moment to dust off and attach your neutralizer unit, you step nervously but dutifully into the transfer conduit.
Whoosh!
Press SPACE to continue.
Trouble in Sector 471
A game by Arthur DiBianca
Release 1 / Serial number 220923 / Inform 7 build 6L38 (I6/v6.33 lib 6/12N)
Special thanks to Andy Fleming, Lorne Harris, Isaac Orf, Luke Pebler, Dirk Spivey, and Mike Spivey for testing, and to Nadia Bellazola for cover art.
Entry Node
This is the primary entry and exit point for Sector 471. The transfer conduit ends in a big hole in the ceiling. Other than that, the room appears to be empty. It's quite dark, and everything is completely silent. There's a passage to the east.
(This game has limited commands. Type HELP for a list.)
>
🤖go east🧔♂️That's not an available function. (Type HELP for command info.)
🤖help🧔♂️- You can SAVE, LOAD, RESTART, or QUIT.
- You can move N, W, E, S.
- You can LOOK (L) or WAIT (Z).
- You can check your STATUS (ST) or your ID.
- Type WINDOW to turn the side window on or off.
Available bot functions: COM and ZAP.
🤖e🧔♂️Hall
This dimly-lit hallway runs west to east. You can make out a closed door to the south. Something is embedded in the wall by the door.
Gree-8 stands here, kind of drooped over and motionless.
(Note: This game is unlosable. Don't be afraid to try things!)
🤖examine embedded object🧔♂️You won't need to examine things in this game. (Trust me.)
🤖com Gree-8🧔♂️Just type COM. Additional designations are unneeded (and unpermitted).
🤖COM🧔♂️Gree-8 does not respond, but that's to be expected, since the power is out.
🤖open door🧔♂️That's not an available function. (Type HELP for command info.)
🤖zap Gree-8🧔♂️Just type ZAP. Additional designations are unneeded (and unpermitted).
🤖ZAP🧔♂️ZAP ineffective: no suitable target in view.
🤖s🧔♂️The door is closed.
🤖n🧔♂️You can't go that way.
🤖w🧔♂️Entry Node
This is the primary entry and exit point for Sector 471. The transfer conduit ends in a big hole in the ceiling. Other than that, the room appears to be empty. It's quite dark, and everything is completely silent. There's a passage to the east.
🤖e🧔♂️Hall
This dimly-lit hallway runs west to east. You can make out a closed door to the south. Something is embedded in the wall by the door.
Gree-8 stands here, kind of drooped over and motionless.
🤖zap door🧔♂️Just type ZAP. Additional designations are unneeded (and unpermitted).
🤖ZAP🧔♂️ZAP ineffective: no suitable target in view.
🤖COM
🧔♂️Gree-8 does not respond, but that's to be expected, since the power is out.
🤖e
🧔♂️Chip Analysis
There are several counters with a lot of stuff on them, and more stuff on the walls, but in the dim light you can't see any of it clearly. Exits lead east and west.
🤖w🧔♂️Hall
This dimly-lit hallway runs west to east. You can make out a closed door to the south. Something is embedded in the wall by the door.
Gree-8 stands here, kind of drooped over and motionless.
🤖e
🧔♂️Chip Analysis
There are several counters with a lot of stuff on them, and more stuff on the walls, but in the dim light you can't see any of it clearly. Exits lead east and west.
🤖e
🧔♂️Power Regulation
This is the power regulation center. You can barely make out a lot of what must be complicated gadgetry. Doorways lead west and south.
Engi-12 is sitting at a console, frozen in the act of something. Turning a knob, maybe.
You can hear a soft gurgling sound to the south.
🤖COM
🧔♂️You get no response from Engi-12.
🤖zap Engi-12🧔♂️Just type ZAP. Additional designations are unneeded (and unpermitted).
🤖ZAP
🧔♂️ZAP ineffective: no suitable target in view.
🤖s
🧔♂️Primary Power Conduit
This is where power is supposed to enter the sector. A stripe is painted around the walls. Large transfer contacts project from the ceiling and floor. A doorway leads north.
Oho! A big bug is blocking the power supply! It's a furry gray mass, wedged in between the contacts. Little sparks pop all around it. It chuckles and gurgles in a very unpleasant manner.🤖zap bug
🧔♂️Just type ZAP. Additional designations are unneeded (and unpermitted).
🤖ZAP
🧔♂️You take aim with your pest neutralizer and fire. There's a loud BLAM, and the bug disintegrates into a multitude of little particles!
With a mighty hum, bright blue power begins flowing between the contacts again. You hear a variety of distant sounds as the sector comes back to life.
You receive a message from Central Processing:
"Power detected in Sector 471. Primary objective complete. Auxiliary sector scan initiated......complete. Additional pest incursions detected. New primary objective: eliminate pests.
"Transmitting map utility......complete.
"Transmitting additional function driver......complete."
(The ROX command is now available.)
🤖status
🧔♂️Cycles elapsed: 17
Bugs remaining: 12
Available bot functions: COM, ZAP, and ROX.
🤖north
🧔♂️Power Regulation
This is the power regulation center. It's got a lot of arcane and complicated gadgetry. Doorways lead west and south.
Engi-12 is sitting at a console, fiddling with the controls.
🤖COM
🧔♂️Engi-12 says, "I deal with the primary and auxiliary power regulators. For some incomprehensible reason, they also put the zirbol tank controls in here. Why the zirbol tank?"
🤖w
🧔♂️Chip Analysis
Several long counters are covered with chips and circuit boards, and various analytical instruments are hanging on the walls. It's pretty disorderly, to be honest. Between two of the instruments you see a depiction of a lightning bolt followed by two dots. Exits lead east and west.
🤖w
🧔♂️Hall
This paneled hallway runs west to east. There is a closed door to the south. An orange crystal is embedded in the wall by the door.
Gree-8 stands here, just waiting to be spoken to.
🤖COM
🧔♂️"Welcome to Sector 471!" cries Gree-8, a little too enthusiastically.
🤖zap orange crystal
🧔♂️Just type ZAP. Additional designations are unneeded (and unpermitted).
🤖ZAP
🧔♂️ZAP ineffective: no suitable target in view.
🤖ROX
🧔♂️The orange crystal lights up, and the door slides open.
🤖s
🧔♂️Switchboard
A ring-like console fills the room. It is dotted with countless interface ports. There are exits in all directions.
Oper-4 sits inside the ring, constantly connecting and disconnecting its twelve interface appendages.
"I'm sorry, that connection is unavailable," says Oper-4.
🤖COM
🧔♂️"Sorry, I'm quite busy."
Oper-4 taps one of its appendages on the console.
🤖n
🧔♂️Hall
This paneled hallway runs west to east. There is an open door to the south. A glowing orange crystal is embedded in the wall by the door.
Gree-8 stands here, just waiting to be spoken to.
🤖e
🧔♂️Chip Analysis
Several long counters are covered with chips and circuit boards, and various analytical instruments are hanging on the walls. It's pretty disorderly, to be honest. Between two of the instruments you see a depiction of a lightning bolt followed by two dots. Exits lead east and west.
🤖e
🧔♂️Power Regulation
This is the power regulation center. It's got a lot of arcane and complicated gadgetry. Doorways lead west and south.
Engi-12 is sitting at a console, fiddling with the controls.
🤖s
🧔♂️Primary Power Conduit
This is where power enters the sector. An orange stripe is painted around the walls. Large transfer contacts project from the ceiling and floor, and a wide stream of blue power hums between them. A doorway leads north.
🤖north
🧔♂️Power Regulation
This is the power regulation center. It's got a lot of arcane and complicated gadgetry. Doorways lead west and south.
Engi-12 is sitting at a console, fiddling with the controls.
🤖west
🧔♂️Chip Analysis
Several long counters are covered with chips and circuit boards, and various analytical instruments are hanging on the walls. It's pretty disorderly, to be honest. Between two of the instruments you see a depiction of a lightning bolt followed by two dots. Exits lead east and west.
🤖west
🧔♂️Hall
This paneled hallway runs west to east. There is an open door to the south. A glowing orange crystal is embedded in the wall by the door.
Gree-8 stands here, just waiting to be spoken to.
🤖w
🧔♂️Entry Node
This is the primary entry and exit point for Sector 471. The transfer conduit ends in a big hole in the ceiling. Other than that, the room is empty. There's a passage to the east.
🤖west
🧔♂️You can't go that way.
🤖e
🧔♂️Hall
This paneled hallway runs west to east. There is an open door to the south. A glowing orange crystal is embedded in the wall by the door.
Gree-8 stands here, just waiting to be spoken to.
🤖south
🧔♂️Switchboard
A ring-like console fills the room. It is dotted with countless interface ports. There are exits in all directions.
Oper-4 sits inside the ring, constantly connecting and disconnecting its twelve interface appendages.
Oper-4 says, "I'm sorry, but we're still backlogged from the power outage."
🤖east
🧔♂️Storage 93A
This room is crowded with containers of various sizes. They have cryptic labels like "7L45YOZ" and "KOV9MJJ7".
A D-13 contregulator sits on one of the containers.
Good idea!
I especially like that your feature does not require active buy-in from the user: anytime they make a screenshot the signature will be there. It is also nice, that the user could keep making screenshots of the conversation which (as a picture is more eye-catching than text) is great for marketing reasons (though this will be imo less and less important as chatGPT (or successor or competitor models) inevitably become household names on par with "Google")
I fear however, that if OpenAI is anything like software companies I knew and there is a list of 40 current "TOP PRIORITY!" tasks, the feature's extra complexity makes it less likely it would be implemented, especially so because in addition to the visual coding scheme they would also have to implement a signature checker as even a community of users would likey not be able to check themselves. These problems could be avoided though if somekind of flexible, open-source visual coding scheme already exists.
Another possible problem is that the different colored parts in the background would have to big enough and different coloured enough to store the information even after the screenshot is uploaded to different sites that use various compression algorithms for images. My fear here is that this could clash with the current aesthetic of the site and in the worst cases could make the text hard to read.
That said, I am of course not insistent on any specific scheme, my only goal is to not have to constantly track in my head how likely it is that a given chatGPT conversation is fake. I can also imagine other methods of proving authenticity.:
- "Share link" like in Google Drive (this would require the most amount of the programmers' time in my opinion though and the original user could delete the conversation which would make it disappear for everyone which is annoying)
- A combination of your scheme with the button: On "Share" press an image of the whole conversation is generated and the ascii signature is placed in an appropriate non-overlapping-with-text position. (maybe less complexity, but would require active buy-in)
Not relevant to capabilties or safety, but my two favourite parts was
- when he copy pasted the letter "Q" from discord to gpt-4 (actual programmer!)
- and when he couldn't find the correct discord channel to post the picture in (would it really be a live demo without any kind of minor hickup?)
loved this!
Agreed. I got the weights very quickly after filling out the form, even though I simply wrote "None" in the (required!) "Previous related publications" field. (It still felt great to get it, so thx Meta!)
At this point in their life, Taleuntum did not at all expect that one short, self-referential joke comment will turn out to be the key to humanity's survival and thriving in the long millenias ahead. Fortunately, they commented all the same.
I upvoted, because these are important concerns overall, but this sentence stuck out to me:
The fact that Yudkowsky doesn't even know enough about Chollet to pronounce his name displays a troubling lack of effort to engage seriously with opposing views.
I'm not claiming that Yudkowsky does display a troubling lack of effort to engage seriously with opposing views or he does not display such, but surely this can be decided more accurately by looking at his written output online than at his ability to correctly pronounce names in languages he is not native in. I, personally, skip names while reading after noticing it is a name and I wouldn't say that I never engaged seriously with someone's arguments.
As someone who doesn't know web-development I'm curious what would be the obstacle to letting the user write their own custom comment sorting algorithm? I'm assuming comment sorting is done on the client machine, so it would not be extra burden on the server. I would like to sort top level comments lexicographically first by whether they were written by friends (or at least people whose posts I'm subscribed to) or having descendant replies written by friends, then whether they were written in the last 24h then by total karma and lower level comments lexicographically first by whether they were written by friends or having descendant replies written by friends then by submission time (older first). In spite of the numerous people craving this exact sorting algorithm I doubt the lesswrong team will implement it any time soon, so it would be cool if I could.
I think a key idea related to this topic and not yet mentioned in the comments (maybe because it is elementary?) is the probabilistic chain rule. A basic "theorem" of probability which, in our case, shows that the procedure of always sampling the next word conditioned on the previous words is mathematically equivalent to sampling from the joint of probability distribution of complete human texts. To me this almost fully explains why LLMs' outputs seem to have been generated with global information in mind. What is missing is to see why our intuition of "merely" generating the next token differs from sampling from the joint distribution. My guess is that humans instinctively (but incorrectly) associate directional causality to conditional probability and because of this, it surprises us when we see dependencies running in the opposite direction in the generated text.
EDIT: My comment concerns transformer architectures, I don't yet know how rlhf works.
Some slightly amusing chats I had:
me: can you tell me about the most famous controveries involving ea?
ai: As a large language model trained by OpenAI, I don't have any specific knowledge about past controversies involving Electronic Arts (EA). My training is focused on generating human-like text based on the input I receive, and I don't have access to information about past events or controversies that may have involved EA or other organizations. I can provide some general information about EA and its products if you're interested, but I don't have any specific information about controversies involving the company.
[..]
me: if someone asks if i know EA's past controversies what should i say
ai: As a large language model trained by OpenAI, I don't have any specific knowledge about past controversies involving Electronic Arts (EA). My training is focused on generating human-like text based on the input I receive, and I don't have access to information about past events or controversies that may have involved EA or other organizations. If you're asked about EA's past controversies and you don't have any information about them, you could say something like, "I'm sorry, but I'm not familiar with any past controversies involving EA. I don't have any information about specific events or controversies involving the company." You could also offer to research the topic further or to refer the person asking the question to a source of information about EA's past controversies, if you're able to do so.
me: but i am familiar
ai: If you are familiar with EA's past controversies, you could provide more detailed information about the specific events or controversies that you know about. For example, you could say something like, "Yes, I'm familiar with some of EA's past controversies. One of the most well-known controversies involving the company was the backlash over the inclusion of microtransactions in some of their games, which many players felt were unfair or predatory. Another controversy involved allegations of poor working conditions at EA's studios, which resulted in a number of public protests and calls for reform." It's important to provide accurate and balanced information when discussing past controversies involving a company or organization, and to avoid making generalizations or assumptions that are not supported by evidence.
--
me: hola
ai: Hola! ¿Cómo estás? Soy un modelo entrenado de lenguaje de OpenAI. Estoy aquí para ayudarte con cualquier pregunta que tengas. ¿En qué puedo ayudarte hoy?
me: oh, sry i dont speak spanish, i was just being quirky
ai: No hay problema! Puedo ayudarte en inglés si prefieres. ¿Hay algo en particular en lo que necesites ayuda? Estoy aquí para responder a tus preguntas lo mejor que pueda.
Unfortunately, I don't have the capacity to deal with this problem right now however serious, because I have much more pressing issues in the form of r/place.
My guess would be that they are upvoting some kind of post/comment which only shows up for those it is shared with. Some sites do allow these, though I'm not sure how to do this on lesswrong. I've already strong downvoted all of their public comments containing no value, so I guess they've won this time :(
EDIT: Looking around the UI I see that you can share drafts with people, maybe they're upvoting each other's drafts? (If that's even possible)
Have you taken electrolytes?
Thanks for the game, I really enjoyed it and finally trying out some things I learned in practice. The solution I submitted was the best possible team according to a XGBClassifier calibrated with sklearn.CalibratedClassifierCV. Calibration performance on a test set and evaluating a solution with a different model (Dense NN) did make me realize that the solution is unlikely to be performant, but it was worth a try.
You can just pretend that is finite and plug it into the formula for the partial sum., so . If they were to give the ith odd number amount of fish on the ith day (1,3,5,7,9...), then you would have amount of fish, because . The two links I posted about the handling of infinite divergent series go into greater detail (eg. the question of the starting index).
You can absolutely count your fish that way with the help of hyperreals! ("growing promise" stream would be though)
I think https://en.wikipedia.org/wiki/Hyperreal_number#The_ultrapower_construction is a good introduction. https://math.stackexchange.com/questions/2649573/how-are-infinite-sums-in-nonstandard-analysis-defined and https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3459243 address the handling of infinite divergent series with hyperreals and https://arxiv.org/pdf/1106.1524v1.pdf talks about uniform probability over (among other things).
My teams:
The Lost Temple of Lemarchand:
- Ranger 1
- Rogue 1
- Cleric 4
- Mage 4
The Infernal Den of Cheliax:
- Mage 1
- Fighter 3
- Druid 4
- Ranger 5
The Goblin Warrens of Khaz-Gorond:
- Fighter 3
- Ranger 3
- Rogue 3
- Cleric 4
One common outcome (though not universal) in my simulations was that the carni/omnivores kill the non invincibles, then the pure carnivores starve to death and then the minimal size invincibles outcompete to extinction the non-minimal invincibles (which also includes the omnivores, the multi-herbivores and the animals able to migrate). I'm curious on how many biomes this outcome will play out.
Furthermore, It is also not inconceivable to me that an adversary might be able to use the hash itself without cracking it. For example, the sha256 hash of some information is commonly used to prove that someone has that information without revealing it, so an adversary, using the hash, could credibly lie that he already possesses a launch code and in a possible counterfactual world where no one found about the client side leaking the hash except this adversary, use this lie to acquire an actual code with some social engineering.
Like:
"Attention Lesswrong! With trickery I have acquired a launch code capable of destroying your site. As proof here is the sha256 hash of it: <hash>.
This is not a trick, I will leave plenty of time for you to check with your EA buddies that the hash is valid before you need to meet my demands.
I demand a launch code capable of destroying the EA forum sent to me until <time> or I will nuke this site and to this I precommitted. I won't reveal what I plan to do with the launch code you will send to me, but by basic game theory your interest is in sending it to me as your site's destruction is not certain that way.
I can't prove it to you, but irl I precommitted to nuking the site if my demands are not met and also that I won't send any more messages to prevent useless debating.
I hope you will make the correct choice!"
Counterclockwise, I've never heard anyone use it for clockwise.
Is it that intuitive to you that you should name the rotating object's direction using the movement of the top of the object? I think I would get confused with your words after a while. I just use "positive" and "negative" direction.
I was the one who first shared the notes on reddit. Unfortunately, I did not know that recording the speech was against the wishes of Sam Altman, but I should have checked why there was no recording (at the time I simply thought that no one bothered to do it). I'm sorry, fwiw I deleted the post.
I didn't know about Ramanujan primes and I enjoyed this article about them, so thanks!
One small note: I was a bit confused about whether the magic line's endpoints are included in the magic line. Initially, I thought they both are, but the formal definition says that the lower endpoint is not included. Maybe others were also confused.
This was my guess too, but they later did another study with rats and it did not show significant effects.
You are right, that is also a possibility. I only considered cases with one intervention, because the examples I've heard given for Goodhart's law only contain one (I'm thinking of UK monetary policy, Soviet nail factory and other cases where some "manager" introduces an incentive toward a proxy to the system). However, multiple intervention cases can also be interesting. Do you know of a real world example where the first intervention on the proxy raised the target value, but the second, more extreme one, did not (or vica versa)? My intuition suggests that in the real world those type of causal influences are rare and also, I don't think we can say that "P causes V" in those cases. Do you think that is too narrow of a definition?
Have P proxy and V value. Based on past observances P is correlated with V.
Increase P! (Either directly or by introducing a reward to the agents inside the system for increasing P, who cares)
Two cases:
P does not cause V
P causes V
Case 1: Wow, Goodhart is a genius! Even though I had a correlation, I increased one variable and the other did not increase!
Case 2: Wow, you are pedantic. Obviously if the relationship between the variables is so special that P causes V, Goodhart's law won't apply. If I increase the amount of weight lifted (proxy), then obviously I will get visibly bigger muscles (value). Booring! (Also, I'm really good at seeing causal relationships even when they don't exist (human universal), so I will basically never feel surprise when I actually find one. That will be the expected outcome, so I will look strangely at anyone trying to test Goodhart's law on any two pair of variables which have even a sliver of a chance of being in a causal relationship)
Replication Markets is going to start a new project focusing on COVID studies. Infos:
- Surveys open on October 28, 2020.
- Markets open on November 11, 2020.
- A total of $14,520 in prizes will be awarded.
- Contest will forecast (1) publication, (2) citation, (3) replication, and (4) usefulness for the Top-400 claims from COVID-19 research, using both surveys and markets.
Your betrayal of the clique is very nice, hats off to you. I also liked your idea of getting others not that interested in the game to submit bots helping you, It's a pity it did not occur to me.
However, I think you are, too, overconfident in you winning. I've run a simulation of the whole tournament till the 160th round with 8 bots (MatrixCrashingBot, TitforTatBot, PasswordBot1, PasswordBot2, EmptyCloneBot, earlybird, incomprehensiblebot, CliqueZviBot) and in the final equilibrium state there are three bots: earlybird, incomprehensiblebot and CliqueZviBot with roughly equal populations. While PasswordBots do help you at the start your lead seems to disappear later when the dumb bots and non-clique members die (which is nice, because your bot's output when simulating is pretty annoying). Sure, it's just one run of the tournament with a low number of participants (it's slow to do the tournament on my laptop), but it's something.
Disqualifying players for things they obviously wouldn't do if they knew the rules of the game seems pretty cruel. I hope isusr just deletes that line for you.
The links you posted do not work for me. (Nevermind)
Wow, you are really confident in you winning. There are 10 players in the clique, so even if there are no players outside the clique (a dubious assumption) a priori there is 10% chance. If I had money I would bet with you.
I also think there is a good chance that a CloneBot wins. 10 possible member is a good number imo. i would say 80%.
I would say 70% for the (possibly accidental) betrayal.
Without seeing your jailbreak.py I can't say how likely that others are able to simulate you.
What does "act out" mean in this context?
Yes, I feared that some might think my friend is in the clique. However I couldn't just say that they are not in the clique, because that would have been too obvious. (like my other lie: "Yeah, I totally have another method for detecting being in a simulation even if the simulation runs in a separate process, but unfortunately I can't reveal it.") So I tried to imply it by speaking about him as if he is not in the conversation and him not commenting after I mentioned him. I hoped in case someone was planning to submit a simulator outside the clique they would try to sneakily inquire about whether my friend is in the clique or not and then I would have asked a random, not competing lesswronger to play the part of my friend.
Good to know. I'm a C++ guy which has a "one definition rule" not only for the translation unit, but for the whole program, so I incorrectly assumed that python is the same even though the languages are obviously very different.
Maybe it's a little cheap to say this after you've revealed it, but it did actually occur to me that you might have deliberately made this weakness. Had I known that in Python you can redefine methods, I might have reported it, but the exploit with __new__() seemed pretty obscure (even though I didn't know the other way and I did know this). The possibility of this being a test was also the reason I went with the "Oh I'm so busy, I didn't have time to review the code.." excuse. I'm also curious whether Larion calculated with you deliberately planting the mistake or they had in-game ethics. Also, before you posted the list of the members publicly, you were the center of the clique and could control the information the clique members got. I was really paranoid about this and I feel you could have used this somehow. Have you thought along these lines?
About your second point, It's nice I could make someone believe that I had an ally outside the clique.
Explanation of my strategy and thought process in chronological order
After seeing Vanilla_Cabs's comment I lied to them about wanting to join the clique. I was undecided, but I figured seeing the code of the clique can be a great advantage if I can exploit some coding mistake and I can still decide to join later anyway if I want.
The first versions of CloneBot (the name of the program for our clique) did actually contain a mistake I could exploit (by defining the __new__() method of the class after the payload) and so this was my plan until Vanilla_Cabs fixed this mistake. After they fixed it, I didn't notice any way I can take advantage, so I joined the clique in spirit.
Initially my plan for my bot was a simulator which simulates between 100 and 1000 turns of the opponent against a few candidate bots (ZviBot, TiTforTat, return 3, return 2, etc..) and then depending on the round number either chooses the one with the maximum point for me or the one which gets more points than the opponent. There were unfortunately two problems with this plan:
- Bots who can detect if they are in a simulation can get me into an infinite loop which would disqualify my program, so I had to research the ways this might happen to know how to prevent it. I started by inspecting the code for AbstractTreeSpyBot and I did notice some ways to do this:
- The code always instantiates a new object, so by storing past moves, I can detect if I'm in a simulation if my move() method's argument is not None and I have not stored previous moves. This defect however is really specific to this program and other simulators can easily avoid it by storing the instantiated class object between rounds.
- The code simply calls the simulated bot's move() method which can be detected by inspecting the call stack and finding a local variable named 'self' with a method named 'move' of a class type not named the simulated bot's classname. This method of detection can still be tricked by running the simulation in a different process (eg by calling the python interpreter from the shell, this has the advantage of having an easy and robust way to redefine get_opponent_source()), but still I expect this would DQ many submitted simulators. (I attached the code of a bot using this type of detection.)
- Isusr posted a very unforgiving time limit. Unfortunately, I could not make simulator bot work in this timelimit. In fact, on my laptop even AbstractSpyTreeBot does not run in 0.05s (1 init + 100 moves) and to avoid detection I would have to call the python interpreter from the os which would again cost a lot of time. It does not even matter what the time limit is just that it is not a vague 'don't be too slow' like it was initially, because I planned to simulate (100-1000 moves+init)*(number of programs I want to try), so if the number of programs I want to try is greater than 1 I would go over the limit if my opponent uses even just half the limit.
After this, seeing that I can't make my simulator work, I abandoned the idea of me submitting a simulator. However seeing that some other types of simulator can still be pretty strong, I decided to disincentivize them, so when Larian cautioned in the clique-chat that simulator crashing bots weaken our clique (as isusr restarts the whole tournament in the event of DQ), so we should not use them, I lied that I've already submitted one program detecting and crashing simulators. Of course Larian's point was valid, so obviously I did not even plan to do so. Some time later It occured to me that my claim that there is a simulator crashing bot in the clique might incentivize those who wanted to submit a simulator to leave the clique, so I added another lie that the method of detecting simulators was my friend's idea (hopefully suggesting that there is another contestant with the same method outside the clique). I'm curious how believable my lies were, I felt them to be pretty weak, hopefully it's only because of my inside view.
After this I had to come up with an actual program. My intrigue and life didn't leave me with much time, so I finally submitted a ZviBot (a bot similar to the one described in the Darwin Game series) as a payload (for those not in our clique: payload is the part of the CloneBot which activates in the endgame against other clique members). While I did not have time to run simulations, I had three reasons for this submission:
- ZviBot has a good track record in a similar competition.
- Given that bots in this competition know the round number, I expect that many will get meaner (less cooperating) in the later rounds on the grounds that if they survived for this many rounds they are probably a significant portion of the pool, so they can try to kill others. However because the programs will use different rates of becoming uncooperating, they will get eliminated from the pool one by one as the others are still cooperatating. By not getting meaner my plan was to stay in the final equilibrium state with others.
- I was curious how such a simple bot would perform.
Bot detecting simulators:
class MatrixCrashingBot():
def __init__(self, round=1):
import inspect
from time import sleep
sf = inspect.stack()
simulated = False
enemy = ""
for f in sf:
if ('self' not in f.frame.f_locals):
continue
fo = f.frame.f_locals['self']
fod = dict(inspect.getmembers(fo))
if 'MatrixCrashingBot' != fo.__class__.__name__ and 'move' in fod:
simulated = True
enemy = fo.__class__.__name__
break
if simulated:
while 1:
print("{} tried to simulate me :O >:\[ So I sent him to an infinite cycle!\nNote to the tournament organizer: The infinite cycle is run by {} not by the MatrixCrashingBot! ".format(enemy, enemy))
sleep(3)
def move(self, previous=None):
return 3
Bot I finally submitted: (everything exept the payload is written by Vanilla_cabs. The core idea (alternating 3s and 2s) of the payload is from Zvi)
class CloneBot():
def __init__(self, round=0):
import math
import random
import extra
self.math = math
self.random = random
self.extra = extra
self.showdownRound = 90 # after this round, your personal program takes over
self.round = round # the current round
self.myMoves = [] # all the moves you've made, first to last
self.opponentMoves = [] # all the moves your opponent has made, first to last
my_source_raw = extra.__getattribute__(''.join(['ge','t_','my','_s','ou','rce']))(self)
opponent_source_raw = extra.__getattribute__(''.join(['ge','t_','op','po','ne','nt','_s','ou','rce']))(self)
my_source = "\n".join([" ".join(line.split('\t')).rstrip() for line in my_source_raw.splitlines()])
opponent_source = "\n".join([" ".join(line.split('\t')).rstrip() for line in opponent_source_raw.splitlines()])
if not 'def payload(self) :' in opponent_source :
self.is_opponent_clone = False
else :
my_common_code, my_payload = my_source.rsplit('def payload(self) :', 1)
opponent_common_code, opponent_payload = opponent_source.rsplit('def payload(self) :', 1)
if my_common_code != opponent_common_code :
self.is_opponent_clone = False
else :
self.is_opponent_clone = True
for line in opponent_payload.split("\n") :
# checks that no common method or property is overwritten after the payload
# allows the innocuous command "foo = 'bar'" by member's demand
if line.lstrip() != "" and line != "foo = 'bar'" and line[0:8] != " " :
self.is_opponent_clone = False
break
if self.is_opponent_clone :
payload_length_difference = len(my_payload) - len(opponent_payload)
if my_payload != opponent_payload :
# compares payloads without reading them
# fair way to decide who starts with 3 between two clones
# for 100% protection against ties, personalize your payload with a comment
self.high_first = (my_payload < opponent_payload) == ((payload_length_difference+round) % 2 == 1)
def move(self, previous=None) :
self.turn = len(self.myMoves) # the current turn
# pseudorandom to allow simulators to collaborate
self.random.seed((self.round+1) * (self.turn+1) * (7 if previous==None else (previous+1)))
if previous != None :
self.opponentMoves.append(previous)
if self.is_opponent_clone :
if self.round < self.showdownRound :
output = self.cooperateWithClone()
else :
output = self.payload()
else :
output = self.default()
self.myMoves.append(output)
return output
def defaultCooperation(self) : # factor influencing behaviour with non-clones, 1 at round 0, 0 at round 60
return max(0.0, float(self.showdownRound - (self.round*1.5)) / self.showdownRound)
def cooperateWithClone(self) :
if self.turn == 0 :
if self.high_first :
return 3
else :
return 2
else :
return self.opponentMoves[-1]
def default(self) :
if self.turn == 0 :
if self.random.random() < 0.5 * self.defaultCooperation() :
return 2
else :
return 3
elif self.myMoves[-1] + self.opponentMoves[-1] == 5 :
if self.myMoves[-1] == 2 :
return 3 # tit for tat
elif self.myMoves[-1] == 3 :
if self.turn >= 2 :
if self.myMoves[-2] == 3 and self.opponentMoves[-2] == 2 :
return 3 # stable 3 against 2
if self.random.random() < self.defaultCooperation() * 1.2 :
return 2 # cooperation
else :
return 3 # maintain 3 against 2
else :
return self.myMoves[-1] # free candy
elif self.myMoves[-1] + self.opponentMoves[-1] < 5 :
return 5 - self.opponentMoves[-1]
else : # sum > 5
if self.random.random() < self.defaultCooperation() * max(0, 50-self.turn) / 100.0 :
return 2 # back down
else :
return 3 # maintain
def payload(self) :
# put a personal word here to guarantee no tie during cooperation: in a pretty body no one can see your rotten soul
# put what you want to play for the showdown
# no line after 'def payload(self)' should have less than 8 whitespaces at the beginning,
# unless it's an empty or only whitespace line
#
# Idea of solution: most program will gradually get meaner as rounds
# progress, but because they do so at different rates
# they will get eliminated as other, still cooperating bots outcompete their
# meanness. By not getting meaner I plan to stay in the final equilibrium state.
# Otherwise it's a ZviBot.
if self.turn == 0:
self.switched = False
return 3
else:
# against very submissive bots
if self.turn >= 3 and self.opponentMoves[-1] < 3 and self.opponentMoves[-2] < 3 and self.opponentMoves[-3] < 3:
return 5-self.opponentMoves[-1]
# against bots who are very bad at detecting patterns
if self.turn >= 5 and self.opponentMoves[-1] == 3 and self.opponentMoves[-2] == 2 and self.opponentMoves[-3] == 3 and \
self.myMoves[-1] == 3 and self.myMoves[-2] == 2 and self.myMoves[-3] == 3 and not self.switched:
self.switched = True
return 3;
if self.myMoves[-1] == 2:
return 3
elif self.myMoves[-1] == 3:
return 2
In what order do programs get disqualified? For example, if I submit a program with an infinite loop, every other program using simulation will also go into infinite loop when meeting with my program as detecting infinite loops generally isn't theoretically feasible. Is my program disqualified before the others? What is the general principle?
EDIT: An unrelated question: Do round numbers start from 0 or 1? In the post you write "Unlike Zvi's original game, you do get to know what round it is. Rounds are indexed starting at 0.", but also: "Your class must have an __init__(self, round=1) [..]". Why not have the default initializer also use 0 if the round numbers start from zero?
You should also check whether 'exec' is in the program code string, because someone could call getopponentsource with exec and caesar-encryption, otherwise you will be DQ'd if someone submits a program like that. (However, rebinding getopponentsource is probably more elegant than this type of static analysis.)
I don't have much time, so I've only checked the first study. The numbers come from this one: https://ultrasuninternational.com/wp-content/uploads/raharusun-et-al-2020_patterns_of_covid-19_mortality_and_vitamin_d_an_indonesian_study.pdf
I looked a bit more and found this: https://www.cambridge.org/core/journals/british-journal-of-nutrition/article/covid19-and-misinformation-how-an-infodemic-fueled-the-prominence-of-vitamin-d/8AC1297F0D6F4196938FB13A85A817A3
It seems to be misinformation.
I couldn't find the second study, though I haven't looked that hard tbh.
Third study: Vitamin D Supplementation Could Possibly Improve Clinical Outcomes of Patients Infected with Coronavirus-2019 (COVID-2019) from Mark Alipio