Posts
Comments
Cool puzzle. (I've wrote like 4 versions of this comment each time changing explanation and conclusions and each time realizing I am still confused).
Now, I think the problem is that we don't pay much attention to:
What should one do when one has drawn a red ball?
(Yeah, I strategically use word "one" instead of "I" to sneak assumption that everyone should do the same thing)
I know, it sounds like an odd question, because, the way the puzzle is talked about, I have no agency when I got a red ball, and I can only wait in despair as the owners of green balls make their moves.
And if you imagine a big 2-dimensional array where each of 100 columns is an iteration of a game, and each of 20 rows is a player, and look at an individual row (a player) then, we'd expect, say 50 columns to be "mostly green", of them roughly 45 have the player "has drawn green" cell, and 50 columns to be "mostly red", with 5 of them having "has drawn green". If you focus just on those 45+5 columns, and note that 45:5 is 0.9:0.1, then yeah, indeed the chance that the column is "mostly green" given "I have drawn green" is 0.9.
AND coincidentally, if you only focus on those 45+5 columns, it looks like to optimize the collective total score limited to those 45+5 columns, the winning move is to take the bet, because then you'll get 0.9*12-0.1*52 dollars.
But what about the other 50 columns??
What about the rounds in which that player has chosen "red"?
Turns out they are mostly negative. So negative, that it overwhelms the gains of the 45+5 columns.
So, the problem is that when thinking about the move in the game, we should not think about
1. "What is the chance one is in mostly green column if one has a green ball?" (to which the answer is 90%)
but rather:
2. "What move should one take to maximize overall payout when one has a green ball?" (to which the answer is: pass)
and that second question is very different from:
3. "What move should one take to maximize payout limited just to the columns in which they drew a green ball when seeing a green ball?" (to which the answer is: take the bet!)
The 3. question even though it sounds very verbose (and thus weird) is actually the one which was mentally substituted (by me, and I think most people who see the paradox?) naturally when thinking about the puzzle, and this is what leads to paradox.
The (iterated) game has 45+5+50 columns, not just 45+5, and your strategy affects all of them, not just the 45+5 where you are active.
How can that be? Well, I am not good at arguing this part, but to me it feels natural, that if rational people are facing same optimization problem, they should end up with same strategy, so whatever I end up doing I should expect that others will end up doing it too, so I should take that into account when thinking what to do.
It still feels feel a bit strange to me mathematically, that a solution which seems to be optimal for 20 various different subsets (each having 45+5 columns) of 100 columns individually, is somehow not optimal for the whole 100 columns.
The intuition for why it is possible is that a column which has 18 green fields in it, will be included in 18 sums, and a column which has just 2 green fields in it will be counted in just 2 of them, so this optimization process, focuses too much on the "mostly green" columns, and neglects those "mostly red".
Is it inconsistent to at the same time think:
"The urn is mostly green with ppb 90%" and
"People who think urn is mostly green with ppb 90% should still refuse the bet which pays $12 vs $-52"?
It certainly sounds inconsistent, but what about this pair of statements in which I've only changed the first one:
"The urn is mostly green with ppb 10%" and
"People who think urn is mostly green with ppb 90% should still refuse the bet which pays $12 vs $-52?"
Hm, now it doesn't sound so crazy, at least to me.
And this is something a person who has drawn a red ball could think.
So, I think the mental monologue of someone who drew a green ball should be:
"Yes, I think that the urn is mostly green with ppb 90%, by which I mean, that if I had to pay -lg(p) Bayes points when it turns out to be mostly green, and -lg(1-p) if it isn't, then I'd choose p=0.9. Like, really, if there is a parallel game with such a rules, I should play p=0.9 in it. But still, in this original puzzle game, I should pass, because whatever I'll do now, is whatever people will tend to do in cases like this, and I strongly believe that "People who think urn is mostly green with ppb 90% should still refuse the bet which pays $12 vs $-52", because I can see how this strategy optimizes the payoff in all 100 columns, as opposed to just those 5+45 I am active in. The game in the puzzle doesn't ask me what I think the urn contained, nor for a move which optimizes the payoff limited to the rounds in which I am active. The game asks me: what should be the output of this decisions process so that the sum over all 100 columns is the largest. To which the answer is: pass".
How to combine this with the fact that "the nudge" apparently doesn't work https://phys.org/news/2022-08-nudge-theory-doesnt-evidence-future.html ?
≈ 41,000
Why not ?
The way I understood the story, to define a function on two numbers from I need to fill-in a table with 59*59 cells, by picking for each cell a number from . If 20% of it is still to be filled, then there are 0.2*59*59 decisions to be made, each with 59 possibilities.
Right?
Thank you for heads up!
Could you please clarify for parents like me, who don't fully understand Minecraft's ecosystem and just want their kids to stay safe:
1. If my kids only use Minecraft downloaded from the Microsoft Store, and only ever downloaded content from the in-game marketplace - what's the chance they are affected?
2. Am I right in thinking that "mods" = "something which modifies/extends the executable", while "add-ons"="more declarative content which just interacts with existing APIs, like maps, skins, and configs"?
3. Am I right that "Minecraft from Micosoft Store" + "content from in-game marketplace" would translate to "Bedrock Edition" + "add-ons"?
4. Am I right that the fractureiser affects "Java Edition" + "mods" only?
Upon seeing the title (but before reading the article) I thought it might be about a different hypothetical phenomenon: one in which an agent which is capable of generating very precise models of reality might completely lose any interest in optimizing reality whatsover - after all it never (except "in training" which was before "it was born") cared about optimizing the world - it just executes some policy which was adaptive during training to optimize the world, but now, these are just some instincts/learned motions, and if it can execute them on a fake world in his head, it might be easier to feel good for it.
For consider: porn. Or creating neat arrangements of buildings when playing SimCity. Or trying to be polite to characters in Witcher. We, humans, have some learned intuitions on how we want the world to be, and then try to arrange even fake worlds in this way, even if this disconnected from real world outside. And we take joy from it.
Can it be, that a sufficiently advanced AGI will wire-head in this particular way: by seeing no relevant difference between atomic-level model of reality in its head and atomic-level world outside?
Thanks for clarifying! I agree the twitter thread doesn't look convincing.
IIUC your hypothesis, then translating it to AI Governance issue, it's important to first get general public on your side, so that politicians find it in their interest to do something about it.
If so, then perhaps meanwhile we should provide those politicians with a set of experts they could outsource the problem of defining the right policy to? I suspect politicians do not write rules themselves in situations like that, they rather seek people considered experts by the public opinion? I worry, that politicians may want to use this occasion to win something more than public support, say money/favor from companies, and hence pick not the right experts/laws - hence perhaps it is important to not only work on public perception of the threat but also on who the public considers experts?
Why? (I see several interpretations of your comment)
What did it take to ban slavery in Britain:
TL;DR: Become the PM and propose laws which put foot in the door, by banning bad things in the new areas at least, and work from there. Also, be willing to die before seeing the effects
Source: https://twitter.com/garius/status/1656679712775880705
I agree that my phrasing was still problematic, mostly because it seems to matter if she said something spontaneously or as a response to a specific question. In the first case, one has to consider how often people feel compelled to say some utterance in various life scenarios. So for example in case one has two boys the utterance "i have to pick up Johny from kindergarten" might have to compete with "i have to pick up Robert from kindergarten" and might be strange/rare if both are in similar age and thus both should be picked up etc. Still, I think that without knowing much about how people organize their daily routines, my best bet for the question "does she have two boys?" would be 33%.
It's get funnier with "i have to pick up my younger one, John from kindergarten" :)
I guess what confuses some people is the phrase "the other one" which sounds like denoting a specific (in terms of SSN) child while it's not at all clear what that could even mean in case of two boys. I think step one when being confused is to keep rephrasing the puzzle until everything is well defined/clear. For me it would be something like:
My friend has two kids, and I don't initially know anything about their sex beyond nation level stats which are fifty-fifty. She says something which makes it clear she has at least one boy, but in such a way that it just prohibits having two girls without hinting at all if these are two boys or one, perhaps something like "i have to pick up Johny from kindergarten". How much should I bet she actually has two boys vs a boy and a girl?
I'd expect that IF there is a shoggoth behind the mask THEN it realises the difference between text interaction (which is what the mask is doing) and actually influencing the world (which the shoggoth might be aiming at). That is I expect it's perfectly possible that an LLM will behave perfectly ethical when playing choose your own adventure at the same time thinking how to hack the VM it's running on.
Thanks, fixed. I guess this is not why it got -18 votes, though. I would like to hear what exactly people didn't like in this post
ChatGPT's answer:
Your two assumptions and intuitions are plausible, but they may not hold true in every case. It is important to consider the specific context and motivations of individual rulers when making predictions about their behavior.
Regarding your first intuition, it is possible that some rulers may support the development of powerful AGI if they see it as a means to achieve their goals more efficiently. However, they may also take precautions to ensure that the AGI is under their control and cannot threaten their power.
Regarding your second intuition, rulers may not necessarily get rid of humans as soon as AI can achieve goals more efficiently. Human labor can still be cheaper or more reliable in certain contexts, and rulers may also want to maintain the appearance of caring about human welfare for political reasons.
To make your intuitions more testable, you could formulate specific hypotheses and predictions that can be observed or measured. For example, you could predict that rulers of authoritarian regimes will be more likely to suppress the development of powerful AGI than rulers of democracies, or that companies with highly centralized decision-making structures will be more resistant to AI-driven automation than companies with more decentralized structures. You could then collect data to test these hypotheses, such as by analyzing government policies or conducting surveys of corporate leaders.
(I am a bit worried by this given that China seems to restrict AIs more than US...)
Me: Thanks, the idea with checking what authoritarian regimes are doing today, was very helpful. Can you suggest more operationalisations of my theses?
GPT:
Sure, here are some possible operationalizations of your intuitions:
- Operationalizing your first intuition:
- Hypothesis: Rulers will be more likely to suppress the development of powerful AGI if they perceive it as a threat to their power.
- Prediction: Countries with more authoritarian governments will have fewer AI research projects and investments than countries with more democratic governments.
- Operationalization: Use publicly available data on AI research funding and publications by country, as well as measures of political freedom and democracy, to test the relationship between government type and AI development.
- Operationalizing your second intuition:
- Hypothesis: Rulers will be more likely to replace human workers with AI when it is cost-effective and does not threaten their power.
- Prediction: Companies with highly centralized decision-making structures will be slower to adopt AI-driven automation than companies with more decentralized structures.
- Operationalization: Survey companies to collect data on their decision-making structures and level of AI adoption, and test the relationship between these variables using statistical analysis.
- Operationalizing your conclusion:
- Hypothesis: Rulers will prioritize their own interests over the well-being of their constituents.
- Prediction: Countries with high levels of corruption will be less likely to invest in social welfare programs and more likely to invest in projects that benefit the ruling elite.
- Operationalization: Use measures of corruption and social welfare spending, as well as data on government contracts and investments, to test the relationship between corruption and government priorities.
I like how ChatGPT can help in operatinalizing fuzzy intuitions. I feel an eerie risk that it makes me think even less, and less carefully, and defer to AIs wisdom more and more... it's very tempting ... as if finding an adult who you can cede control to.
I've just finished reading it, and wanted to thank you very much for recommending this great experience :)
Thanks to whoever upvoted my comment recently bringing it again to my attention via notification system - rereading my comment after 2 years, I feel really sorry for myself that despite writing the sentence
And your post made me realize, that the technique from the book you describe is somewhat like this, if you look through "subagents model of the brain" perspective: there is a part of you which is having emotional crisis, and it's terrified by some problem it needs to solve, but this part is not ready to listen for solution/change, as long as it's in the busy loop waiting for an ACK packet confirming someone got the S.O.S. signal.
I did not really understand what it means and how to implement it and how huge impact on my life it will have once finally executed. Only recently I took part in a Lowen's therapy, in which by performing some body movements typical for aggression I've finally established connection between the part which was angry and the part which could listen about it.
Who's the intended audience of this post?
If it's for "internal" consumption, summary of things we already knew in the form of list of sazens, but perhaps need a refresher, then it's great.
But if it's meant to actually educate anyone, or worse, become some kind of manifesto cited by New Your Times to show what's going on in this community, then I predict this is not going to end well.
The problem, as I see it, is that in the current way this website is setup, it's not up to author to decide who's the audience.
ML models, like all software, and like the NAH would predict, must consist of several specialized "modules".
After reading source code of MySQL InnoDB for 5 years, I doubt it. I think it is perfectly possible - and actually, what I would expect to happen by default - to have a huge working software, with no clear module boundaries.
Take a look at this case in point: the row_search_mvcc() function https://github.com/mysql/mysql-server/blob/8.0/storage/innobase/row/row0sel.cc#L4377-L6019 which has 1500+ lines of code and references hundreds of variables. This function is in called in the inner loop of almost every SELECT query you run, so on the one hand it probably works quite correctly, on the other was subject to "optimization pressure" over 20 years, and this is what you get. I think this is because Evolution is not Intelligent Design and it simply uses the cheapest locally available hack to get things done, and that is usually to reuse the existing variable#135, or more realistically combination of variables#135 and #167 to do the trick - see how many of the if statements have conditions which use more than a single atom, for example:
if (set_also_gap_locks && !trx->skip_gap_locks() &&
prebuilt->select_lock_type != LOCK_NONE &&
!dict_index_is_spatial(index)) {(Speculation: I suspect that unless you chisel your neural network architecture to explicitly disallow connecting a neuron in question directly to neuron#145 and #167, it will, as soon as it discovers they provide useful bits of information. I suspect this is why figuring out what layers and connectivity between them you need is difficult. Also, I suspect this is why simply ensuring right high-level wiring between parts of the brain and how to wire them to input/output channels might the most important part to encode in DNA, as the inner connections and weights can be later figured out relatively easily)
I've made a visualization tool for that:
https://codepen.io/qbolec/pen/qBybXQe
It generates an elliptical cloud of white points where X is distributed normally, and Y=normal + X*0.3, so the two are correlated. Then you can define a green range on X and Y axis, and the tool computes the correlation in a sample (red points) restricted to that (green) range.
So, the correlation in the general population (white points) should be positive (~0.29). But if I restrict attention to upper right corner, then it is much lower, and often negative.
The extremely-minimalist description would be: “Stop believing in the orthodox model, stop worrying, feel and act as if you’re healthy, and then the pain goes away”.
IDK if this will be important to you, but I'd like to thank you for this comment, as it relieved my back pain after 8 years! Thank you @p.b. for asking for clarification and not giving up after first response. Thank you @Steven Byrens for writing the article and taking time to respond.
8 fucking years..
I've read this article and comments a month ago. Immediately after reading it the pain was gone. (I never had mystical experiences, like enlightenment, so the closest thing I can compare it to personally, was the "perspectival shift" I've felt ten years ago when "the map is not the territory" finally clicked)
I know - it could've been "just a placebo effect" - but as the author argues, who cares, and that's kinda the main point of the claim. Still, I was afraid of giving myself a false hope - there were several few days long remissions of pain scattered along these 8 years, but the pain always returned - this is why I gave myself and this method a month before writing this comment. So far it works!
I know - "Post hoc ergo propter hoc" is not the best heuristic - there could be other explanations of my pain relief. For example a week or two before reading this article I've started following this exercise routine daily. However, I've paused the routine for three days before reading your article, and the pain relief happened exactly when I've finished reading your comment, so IMO timing and rarity (8 years...) of the event really suggests this comment is what helped. I still do the exercise routine, and it surely contributes and helps, too. Yet, I do the routine just once in the morning, yet I consciously feel how whenever throughout the day the pain starts to raise its head again, I can do a mental move inspired by this article to restore calm and dissolve the pain.
Also this is definitely how it felt from the inside! In the hope that it will help somebody else alleviate their pain here are some specific patterns of thoughts induced by this article I found helpful:
- "oh, so my pain-center is simply confused about the signals, it is screaming like a child who can't express well what's wrong, and I was overreacting. I should show it love, not anger, I should calm it down, I must be the adult in the room and figure out what's the real problem here."
- "I should ignore the pain by gently putting the pain to the side (like you do to the thoughts during meditation) as opposed to fighting through it. Like hitting snooze, vs clenching my jaw and fist to overcome it."
- "yeah, I've heard you pain-center, but I think you are mistaken about the magnitude and source of the problem, and I am actively working on the solution to the real problem, so please do not distract me while I am helping you"
- "the pain-center is presenting me a whole crayon-drawn image of a tiger, but it was just connecting-the-dots creatively, and there really was no tiger, just the dots". I think this one is most helpful metaphor for me. I can feel how I dissolve a full certainty of "the pain of the whole upper back" into individual, small, shaky dots of unsure signals from small patches of the back.
- "looks like it was just one small place around this shoulder blade which started the alarm, maybe I should just change the position of right arm, oh, yes, this brought relief, good"
- "ok, so this part near neck is so tense it started complaining, and this was probably because I was trying too hard to finish answering this email before visiting the restroom - let's just give myself a pause and treat the body more gently".
- "ok, I need to be more precise: which patch of my back is in pain right now? If I can't tell, then perhaps it's something in the environment that is causing stress, or some thought, or some anticipation, or maybe some physiological need? Let's look around and find out what this alarm is about"
- The Bohr's horseshoe: "I was told that it works even if you don't believe in it"
- I just imagine a volume knob on the pain and just turn it down
I am really excited about all this positive change in my mind, because as one can imagine (and if you can't, recall main character of House M.D.) a constant pain corrupts other parts of your mind and life. It's like a prior to interpret every sentence of family-members and every event in life. It's a crony belief, a self-sustaining "bitch eating cracker syndrome". It took 8 years to build this thought-cancer, and it will probably take some time to disband it, but I see the progress already.
Also, I am "counter-factually frightened" by how close I was to completely missing this solution to my problem. I was actively seeking, you see, fruitlessly, though, for years! I had so much luck: to start reading LW long ago; to subscribe Scott Alexander's blog (I even read his original review of "unlearn your pain" from 2016 yet it sounded negative and (I) concentrated too much on discrediting the underlying model of action, so perhaps I could fix my pain 6 years earlier); to develop a habit of reading LW and searching for interesting things and reading comments, not just the article.. Thank you again for this article and this comment thread. When I imagine how sad would be the future if on that afternoon I didn't read it I want to cry...
I have similar experience with it today (before reading your article) https://www.lesswrong.com/editPost?postId=28XBkxauWQAMZeXiF&key=22b1b42041523ea8d1a1f6d33423ac
I agree that this over-confidence is disturbing :(
We already live in a world in which any kid can start a difficult to stop and contain chain reaction: fire. We responded by:
- making a social norm of not allowing kids to buy or use
- separating houses 1.5 of their height
- adding sprinklers, requiring them by law
- having a state founded agency to stop fires
Honestly I still don't understand very well what exactly stops evil/crazy people from starting fires in forests whenever they want to. Norms to punish violators? Small gain to risk factor?
Also, I wonder to what extent our own "thinking" is based on concepts we ourselves understand. I'd bet I don't really understand what concepts most of my own thinking processes use.
Like: what are the exact concepts I use when I throw a ball? Is there a term for velocity, gravity constant or air friction, or is it just some completely "alien" computation which is "inlined" and "tree-shaked" of any unneeded abstractions, which just sends motor outputs given the target position?
Or: what concepts do I use to know what word to place at this place in this sentence? Do I use concepts like "subject", or "verb" or "sentiment", or rather just go with the flow subconsciously, having just a vague idea of the direction I am going with this argument?
Or: what concepts do I really use when deciding to rotate the steering wheel 2 degrees to the right when driving a car through a forest road's gentle turn? Do I think about "angles", "asphalt", "trees", "centrifugal force", "tire friction", or rather just try to push the future into the direction where the road ahead looks more straight to me and somehow I just know that this steering wheel is "straightening" the image I see?
Or: how exactly do I solve (not: verify an already written proof) a math problem? How does the solution pop into my mind? Is there some systematic search over all possible terms and derivations, or rather some giant hash-map-like interconnected "related tricks and transformations I seen before" which get proposed?
I think my point is that we should not conflate the way we actually solve problems (subconsciously?), with the way we talk (consciously) about solutions we've already found when trying to verify them ourselves (the inner monologue) or convey them to another person. First of all, the Release binary and Debug binaries can differ (it's completely different experience to ride a bike for a first time, than an on 2000th attempt). Second, the on-the-wire format and the data structure before serialization can be very different (the way I explain how to solve an equation to my kid is not exactly how I solve it).
I think, that training a separate AI to interpret for us the inner workings of another AI is risky, the same way a Public Relations department or a lawyer doesn't necessarily give you the honest picture of what the client is really up to.
Also, I there's much talk about distinction between system 1 and 2, or subconsciousness and consciousness, etc.
But, do we really treat seriously the implication of all that: the concepts our conscious part of mind uses to "explain" the subconscious actions have almost nothing to do with how it actually happened. If we force the AI to use these concepts it will either lie to us ("Your honor, as we shall soon see the defendant wanted to..") , or be crippled (have you tried to drive a car using just the concepts from physics text book?). But even in the later case it looks like a lie to me, because even if the AI is really using the concepts it claims/seems/reported to be using, there's still the mismatch in myself: I think I now understand that the AI works just like me, while in the reality I work completely differently than I thought. How bad that is depends on problem domain, IMHO. This might be pretty good if the AI is trying to solve a problem like "how to throw a ball" and a program using physic equations is actually also a good way of doing it. But once we get to more complicated stuff like operating a autonomous drone on the battlefield or governing country's budget I think there's a risk because we don't really know how we ourselves make these kind of decisions.
Based on the title alone I was expecting a completely different article: about how our human brains had originally evolved to be so big and great just to outsmart other humans in the political games ever increasing in complexity over millennia and
==thus==>
our value system already steers us to manipulate and deceive others but also ourselves so that we don't even realize that that's what our goal system is really about so that we can be more effective at performing those manipulations with straight face
==so==>
any successful attempt at aligning a super-intelligence to our values, will actually result in a super-manipulator which can perfectly hide it from everyone including self diagnostic
It's already happening https://githubcopilotinvestigation.com/ (which I've learned yesterday from is-github-copilot-in-legal-trouble post)
I think it would be interesting plot twist: humanity saved from AI FOOM by the big IT companies having to obey intellectual property rights they themselves defended for so many years :)
One concrete advice on cracking eggs with two hands: try to pull your thumbs in opposite directions as if you wanted to tear the egg in halves (as opposed to pushing them in).
Sorry for "XY Problem"-ing this, but I felt strong sad emotion when reading your post and couldn't resist trying to help - you wrote:
Unless I'm eating with other people, food for me is fuel.
Have you tried to rearrange your life so that you can eat the breakfast together with people you care much more often, to the point where you no longer care to make it as quick as possible?
There's only so many ways our hardware can be stimulated to feel happy, don't give up on "eating together with close people"!
Thank you! I've read up to and including section 4. Previously I did know a bit about neural networks, but had no experience with RL and in particular didn't know how RL can actually bridge the gap between multiple actions leading to a sparse reward (as in: hour of Starcraft gameplay just to learn you've failed or won). Your article helped me realize how it is achieved - IIUC by:
0. focusing on trying to predict what the reward will be more than on maximizing it
1. using a recursive approach to infinite sum: sum(everything)=e+sum(verything).
2. by using a different neural network on the LHS than on the RHS of this equality, so that one of them can be considered "fixed" and thus not taking part in backpropagation. Cool!
Honestly, I am still a bit fuzzy on how exactly we solve the problem that most "e"s in e+sum(verything) will be zero - before reading the article I thought the secret sauce will be to have some imaginary "self-rewarding" which eventually sums up to the same total reward but in smaller increments over time (like maybe we should be happy about each vanishing pixel in Arkanoid, or each produced unit in Starcraft a little, even if there's no direct reward for that from the environment)?
After reading your explanation I have some vague intuition that even if "e" is zero, e+sum(verything) has better chance of figuring out what the future rewards will be, because we are one step closer to these rewards, and that somehow this small dent is enough to get things going. I imagine this works somewhat like this: if there are 5 keys I have to press in succession to score one point, then sooner or later I will sample the state after 4 presses and fix my predictor output for this state as the reward will be immediate result of one keypress, then I'll be ready to learn what to do after 3 presses as I will already know what happens after 4th keypress so I can better predict this two-actions chain, and so on, and so on learning to predict consequences of a longer and longer combos. Is that right?
Things which could improve the article:
1. Could you please go over each occurrence of word "reward" in this article and make it clear if it should be "reward at step t" or "sum of rewards from step t onwards"? I felt it is really difficult for me to understand some parts of explanation because of this ambiguity.
2. In the pseudo-code box you used "r(j) + Q(s’, a’, q)" formula. I was expecting "r(j)+gamma*q(s',a')" instead. This is because I thought this should be similar to what we saw earlier in the math formula (the one with red,green and blue parts) and I interpreted that Q(s',a',theta^-) in the math formula was meant to mean something like "the network output for state s', proposed action a' and weights set according to old frozen settings", which I believe should be q(s',a') in the pseudocode. If I am wrong, then please clarify why we don't need to multiplication by gamma and why Q can take three arguments, and why q is one of them, as none of this is clear to me.
3. Some explanation of how this sparsity of rewards is really addressed in games which give you zero reward until the very end.
So what's the end state Putin wants to achieve through invading Ukraine? If Ukraine becomes part of Russia, then Russia will be bordering with NATO states.
Thank you for sharing!
Hello glich! Thanks for writing this whole series. When I've first read it a year ago, I thought to myself, that instead of impulsively going to implement it right ahead, I'll wait one year to hear from you about how your strategy worked for you, first.
So.. How are you doing?
Wouldn't the same argumentation lead to conclusion, that world should've already end soon after we've figured out how to make atomic bomb?
I don't know how to write a novel with world which survives in equilibrium longer than a week (and this is one reason I've asked this question - I'd like to read ideas of others) but I suspect that the same way atomic bomb releases insane amounts of energy, yet we have reasons not to do that repeatedly, mages in would have good reasons to avoid destroying the world. Perhaps there's not much to gain from doing so, maybe there's M.A.D., maybe once you are smart enough to do that you are also smart enough not to do that, maybe you have to swear not to do that.
It could also be the case that I am too confident of our nuclear security. What's currently our best reason not to blow up ourselves? Is it that nuclear energy costs a lot?
"translater" -> "translator"?
"An division" -> "A division"
Lots of details could matter, and the spareness of the writing only hints at what could be going on "for really reals".
Thank you, this was enlightening for me - somehow, though I've read a few books and watched a few movies in my life, I hadn't realized what you put here plainly, that these cuts are a device for the author to hide some truth from me (ok, this was obvious in "Memento"). I must've been very naive, as I simply thought it has more to do with MTV-culture/catering to short attention span of the audience. It's funny how this technique becomes immediately obvious to me once I mentally flip the roles with the author and ask a question "how would I hide something from the reader or mislead them to believe some alternative explanation while not outright lying?".
Hm, perhaps a similar, but more visible and annoying technique/plot device is when the author abruptly ends a conversation between two characters by some explosion or arrival of third person, and they never get to finish their sentence or clarify some misunderstanding. On some level this is the same trick, but between two characters, as opposed to between author and reader.
I now wonder what other "manipulation" techniques I was subjected to. Anyone care to list some they become aware of?
Given that Vi is counting seconds from encountering soldiers to their collapse, AND that there are three dots between this scene and the scene where Miriam says "I've been there since Z-Day." (which technically is an inequality in the opposite direction than I need, but Miriam's choosing this particular wording looks suggestive to me) I'd venture a guess, that the Z-Day virus was released by Vi in the facility, and Miriam is trying to blame the rouge AI for this. I read this story as Vi and Miriam already crossing a line of "the end justifies the means" and simply infect and kill the "innocent" soldiers protecting the headquarter of their commander who is an em/AI, which Vi and Miriam perceive as a threat that needs to be eliminated at all cost.
[p.s.: I've wrote above comment before I've realized that I somehow missed to read 8th episode, and now, after ridding 8th episode, I think Vi and Miriam are cleaning up the mess they've created themselves - the rouge AI they fight in 9th episode is the one they've released, it just took over the command of the army by pretending to be their real commander]
Hello, very intriguing story!
- "solder" appears twice in text - it should be "soldier"
- What is "Vi didn't wait for her translator." supposed to mean? I'm a bit confused because of earlier "She left her cornea and other electronics with Miriam on the scout ship.". Is it supposed to hint at Vi having non-electronical 'machines' (such as the translator) in her body, or just a statement about her having to override her natural instinct/reflex (=normally she'd just wait for the translation, but this time she had a plan to lay down in advance which she would executed even if she still had the electronic translator)? Do people in far future really need to wait for translator?
"You will die. No matter what actions you'll take all the possible branches end with your death. Still, you try to pick optimal path, because that's what your brain's architecture know how to do: pick optimal branch. You try to salvage this approach by proposing more and more complicated goal functions: instead of final value, let's look at the sum over time, or avg, or max, or maybe ascribe other value to death, or try to extend summation beyond it, or whatever. You brain is a hammer, and it needs a nail. But it never occurs to you, that life is not something one needs to optimize. This is not an instance of the problem your brain is built to solve, and it looks silly to me you try to fit it by force to your preferred format. This is your inductive bias, so strong you probably don't get what I'm trying to say to you: yes, you'll die, but this doesn't count."
(I'm surprised nobody wrote it for 12 years, or at least my eyes can't see it)
"The appeared" -> "They appeared"
This discussion suggests, that the puzzles presented to the guesser should be associated with a "stake" - a numeric value which says how much you (the asker) care about this particular question to be answered correctly (i.e. how risk averse you are at this particular occassion). Can this be somehow be incorporated into the reward function itself or needs to be a separate input (Is "I want to know if this stock will go up or down, and I care 10 times as much about this question than about will it rain today", the same thing as "Please estimate p for the following two questions where the reward function for the first one is f(x)=10(x-x^2) and the second is f(x)=x-x^2"? Does it somehow require some additional output channel from the guesser ("I am 90% confident that the p is 80%?" or maybe even "Here's my distribution over the values of p \in (0,1)") or does it somehow collapse into one dimension anyway (does "I am 90% confident that the p is 80% and 10% that it's 70%" collaps to "I think p is 79%"? Does a distribution over p collapse to it's expected value?).
I also have difficulties in applying this techniques on adults, of the "Me mad?No shit Sherlock!" kind. I'm not fluent with it yet, but what I've observed is that the more sincere I am, and the more my tone matches the tone of the other person, the better the results. I think this explains big chunk of "don't use that tone of voice on me!" responses I've got in my life, which I used to find strange [as I personally pay much more attention to the content of the text/speech, not the tone/style/form], but recently I've realized that this can be quite a rational response from someone who reads the cues from both content AND form, and seeing a mismatch, decides which of the two is easier to forge, and which one is the "real" message [perhaps based on their experience, in which controlling emotions is more difficult].
Also, I agree that the "paraphrase the emotions" only maps to the "positive reframing" part. In my eyes the analogy extended also beyond this single step into the pattern of using this discharge step as a necessary step to use some other rationally obvious thing, which you really think should work on its own in theory (like the “Classic CBT”-ish self-talk), but in practice you need to prepare the ground for it.
Indeed there seems to be no analog of "Magical dial" in the "How to talk.." approach. There are some fragments of the book though which teach how to extract the goals/needs/fears of the child and then help them construct a solution which achieves those goals/needs, but this is more like a part of the analog of "classic CPT-ish self talk"-step I think. (In particular I don't recall the book saying things like "do the same stuff just less intensively", so yeah, this part is new and interesting). For example today I told my son, that "So you get mad each time we come to pick you up from your friend right in the moment when you've finally figured out some cool way to play with each other, and this is mega-frustrating, I know. Sure, one way to handle this would be to find yet another way to express anger which doesn't hurt mommy, say punch a pillow, stump, or tear paper, BUT I think that your feeling of being mad is actually trying to tell you something important: that you like your friend very much, like spending time with him, like playing, and hate to be surprised by abruptly having to stop. I don't think we should expect you to feel this each time over and over again each day we pick you up, and try to somehow manage this - how about instead we give you a handwatch, teach you how time works, and let you know in advance when we gonna pick you up? That should eliminate the root cause, not just the effect.".
There's a wonderful book "How to talk so kids will listen & listen so kids will talk", which teaches that if you want your crying&shouting child to actually solve some problem/change behavior/listen to your advice at all, you must realize that there are actually two different personas in them (say: the reptile part of the brain and the neocortex) and you have to first address the first one before you can even start talking with the other: so for example when a child is having a tantrum, what you see is perhaps more like a frightened lizard, than a reasonable person you can persuade, so first you have to calm it down. And the trick to disarm this emotional alert is to actually acknowledge the message it tries to send you - usually by paraphrasing the emotional content ("Oh, I see you have a terrible day, you are angry that we have to finish playing...") [which is even more important in case of children which don't know how to verbalize the emotional state as they lack the words/introspection skills]. There are subtleties to it, like for example: you don't have to copy/inhabit the same emotional state (you don't have to be angry that the play is over yourself) but you need to match the tone and choice of words to the emotion you talk about so that the child believes you are really treating it seriously, not just impassionately describe/condescending/mocking it. So "I see you are angry" delivered in too robotic tone and poker face can backfire and restart the tantrum, but said with harsher tone and grim face can work much better - obviously this is much easier if you simply truly mean it and honestly try to convey that you accept the feelings of your child. It's much more difficult when you are yourself struggling to learn empathy/figure out how to talk about emotions/etc. So I guess it's more natural for some people than others.
Crucially, only when you calm down this part of the brain which was overwhelmed by emotions, you can move forward to reason with the child and explain anything/exchange arguments/plan solution etc. And often this second stage is not even needed, as the child will be able to fix the problem themselves once they're calmed down. OTOH if you start to reason with the child too early, while it's still nervous, it will not lead anywhere except escalating the conflict. In some sense: there's no one to answer your call, until you answer theirs.
I'm saying all this, because I've found these techniques surprisingly helpful in other contexts, like talking to my grown up family members. In some sense an analogue of this is useful in the Internet debates too: you first should be able to paraphrase the stance of your opponent and demonstrate you understand where they're coming from, before you say even a word of critique or argumentation - otherwise they'll immediately label you as outsider who doesn't know a shit and is not worth listening to.
And your post made me realize, that the technique from the book you describe is somewhat like this, if you look through "subagents model of the brain" perspective: there is a part of you which is having emotional crisis, and it's terrified by some problem it needs to solve, but this part is not ready to listen for solution/change, as long as it's in the busy loop waiting for an ACK packet confirming someone got the S.O.S. signal. So you first need to emphatically talk with this part, describe what it feels, acknowledge its good intentions etc. and only then you can move on to actually persuade/negotiate/change anything. This also reminds me a bit the Focusing technique - where the analogue to the child which doesn't know how to name their own emotions, so is stuck in demonstrating them until you guess the keyword for them seems even better. This all starts to look to me as communication problems, not unlike communication problems between people, but within various parts of the brain.
I'm a bit confused by people in the comments entertaining the idea that priors should influence how we interpret the magnitude of the evidence, even though when I look at the Bayes' rule it seems to say that the magnitude of the update (how much you have to multiply the odds) is independent of what your prior was. I know it's not that simple because sometimes the evidence itself is noisy and needs interpretation "pre-processing" before plugging it to the equation, but this "pre-processing" step should use a different prior then the one we try to update. I'm not sure how exactly this "multi-stage Bayesian update rule" should work, and I was trying to describe my struggle in my https://www.lesswrong.com/posts/JtEBjbEZidruMBKc3/are-dogs-bad
I'm unable to find the source for
> (which Pfizer already said they wouldn’t enforce)
Instead I found some articles about Moderna doing so. Is it a typo?
Thanks for the feedback :) Let me know if you find better answers.
Indeed I wasn't fair to politicians - indeed there are valid arguments in favor of "caring about safety" and "signaling `care about safety`" like the one about impact on public fear of vaccination. Thanks for pointing it out. Similarly, there might be valid arguments in favor of "withholding data, model and analysis even if one was made", so a politician not sharing them doesn't mean it wasn't made. Still, this suggests that words of politicians serve too much as signalling, to be easily interpreted by me verbatim as statements about reality. It's more like Simularca Level 2: their phrase "There's a lion on the other side of the river" perhaps means that there is tiger on the other side, or that perhaps we might drawn in the river, and perhaps indeed going to the other side of the river is not the best idea for us all to do, however I'm less and less certain that there is an actual lion over there, and that whatever really should affect for me personally (perhaps I know how to swim, and perhaps I like tigers). In my post I accuse the politicians of Simularca Level 3 (perhaps they just want to show that they care) or even Simularca Level 4 (perhaps they just want to appear that they care). After your comment I now more seriously consider it could be just Simularca Level 2, thanks. Still, I'd love to see some proof of that.
Why do you think exercise improves health? Is it just an educated guess (if so, then what is the reasoning behind it), or is there actually some study establishing causality? I found https://bjsm.bmj.com/content/52/14/890 which says:
> As presented by Kujala, RCTs, the gold standard in epidemiology for inferring causality, have failed to provide conclusive evidence in this context (eg, Lifestyle Interventions and Independence for Elders,8 Look Action for Health in Diabetes,9 Heart Failure: A Controlled Trial Investigating Outcomes of Exercise Training10 and other large-scale meta-analyses).
OTOH I've found https://www.fightaging.org/archives/2018/04/finding-a-causal-relationship-between-exercise-and-longevity-in-human-data-is-more-challenging-than-one-might-imagine/ claiming that
> It is straightforward enough to prove that exercise extends healthy (but not average or overall) life span in studies of mice. (...). It is felt that these correlations likely reflect causation because of the extensive animal studies and the essential similarities of biochemistry between the mammalian species involved, but that isn't the same thing as a rigorous determination.
It feels somewhat tribal and irrational to me that this gets downvoted without any comments presenting critique. I think it would be beneficial to everyone if thesis of the book were addressed. My best guess for why there are downvotes but no comments is that this is n-th iteration of the interchange between author and the community and community is tired of responding over and over again to the same claims. If that's the case, then it would be beneficial to people like me of there was at list a link to a summary of discussion, so far. I think the book is written in quite clear way which should make critiquing it directly quite easy. And the subject matter is very important regardless if you agree to thesis (then you fear genocides etc.) agree with "longtermism" as defined in the book (then you fear the book can stand on the way to floorishing of posthumans) or you don't think the definition matches the actual rules you and your community live by. I bet it's the third case, but IMHO downvoting looks like a reaction typical for someone accepting the description but not liking the thesis (so more typical for second case) while comments could help explain why the critique does not apply. The are of course other possibilities beyond the three. For example: accepting that "longtermism" correctly caputres the assumptions, but the conclusions the author is coming to do not logically follow, or perhaps do follow, but the final judgment of the outcome is wrong, etc. I think it would be beneficial to me to learn what is the exact case and counterargument.
My thoughts immediately went to various programming languages, file formats, protocols, DSLs which while created by pressure-changing apes, at least optimized for something different. Here are my thoughts:
Assembly language - used to tell CPU what to do, seems very linear, imperatively telling step by step what to do. Uses very simple vocabulary ("up-goer 5"/"explain me like I'm five"). At least this is how CPU reads it. But if you think about how it is written, then you see it has a higher-order form: smaller concepts are used to build larger like blocks, functions, libraries etc. So, it's less like giving someone a long lecture, and more like giving someone a wikipedia full of hyperlinks, where they can jump between definitions, perhaps recursively, etc.
But the linearity, even if you have the branches of jumps, and trees of function definitions, is still there from the perspective of CPU, and seems to be rooted in the axis of time- the time needed to perform computation, the time which orders the already computed before that which is to be computed later. So, to break this constraint, my thoughts immediately jumped to multithreading. How do people create languages for multithreading?
Erlang - if you don't care that much about order of execution and more about conveying the goal of them, then a declarative language like Prolog or Erlang seems a nice idea. You just define what each concept means in terms of others, but not necessarily explain how to achieve the goal step-by-step, rather focusing on (de)composition, and hoping that coordination and executing will be figured out by the listener. This is even more like "here, this is Wikipedia, just make me a sandwitch"-style of communication.
LZW and similar compression methods - while thinking about the "time ordering" and "directed acyclic graph" and "dereferencing" you've mentioned, I recalled that a common way to compress a stream is to often use phrases like "here copy paste the 33 bytes I've told you 127 bytes ago". We sometimes do that (see paragraph 3 on page 4), but it's not the building block of our language as in LZW.
Variables - Also "dereferencing" is something nicely solved by using named, scoped, variables instead of a fixed set of pronouns like "this" and "it". We do that to some degree, but it's not like our language is build around defining lots and lots of local references for other stuff like in C++ or Javascript.
Ok, but programming languages will always be somewhat constrained to be "linear" because their goal is to communicate description of some action, and actions are performed over time, which suggests that some form of "happens-before" will have to slip into them. So I thought, about data file formats, which are more time-invariant.
PNG - also in compression, and in relation to 2D you've mentioned, it's common to make some kinds of references cheaper to express. Like in a 1D language it's cheap (2 bytes) to use the word "it" as a shorthand for most recent noun, in 2D image compression it's natural to refer to the color of the pixel above you or the one just before you. So, we might see more kinds of pronouns in 2D or 3D language corresponding to other directions.
3DS meshes - AFAIR you specify list of vertices' coordinates first, then you specify how to connect them into triangles. It's somewhat like: here's a list of nouns, and here are the verbs connecting them. Or maybe: Chapter 1. Introduction of Heros. Chapter 2. The Interactions between Heros. But this linearity between Chapter 1 and Chapter 2 is not really forced on you - you could read it in a different order if you want just to render a fragment of the mesh, right?
Progressively compressed images - you first get a low-resolution sketch of the concept, and then it gets refined and refined and refined until you are satisfied. Like in Scrum! The unrealistic constraint here is that there is no back-and-forth between sender and recipient so you'll just get all the details of everything, not just what you care about/don't know already. Well you can stop listenning at some point (or set the video stream bandwidth), but you can't selectively focus on one part of the image. This could be fixed with some interactive protocol, for sure. And again: you don't have to read the file in the same order it was written, right?
Excel - perhaps one of the most interesting pieces of psycho-technology: you have a grid (it actually doesn't seem so important it's 2D) of concepts and you can make them depend on each other so that you show how information flow between them. Actually this concept can be used to describe the world quite nicely. There's https://www.getguesstimate.com/ which lets you do something similar, but with random variables distributions, so you can show your mental model of reality.
Fractally compressed images - something like Progressive Compression coupled with a strong prior that things are really similar at lower scales to things you've already saw at larger scales. How would that look in language? Perhaps like metaphors we already use in our languages. Like the one in previous sentence? So what new could it bring? Perhaps more precision - our metaphors seem to leave a lot wiggle room for interpretation to the listener, it's not like "listen, you are really supposed to model this part of reality as a clone of the structure I've already told you happens in that portion of reality" and more like "light is like waves" - now, go figure, if it's wet, or uses cosine function somewhere, or what.
JPEG - a language in which you describe the world is not necessarily the way you see it, but rather from some different perspective which captures the most important details, which are the only details you could perceive anyway. I mean: in some sense it feels quite natural to talk about RGB values pixels aligned in a grid, or about Y-axis value of wave at moment t. But once you realize your brain prefers to process sound in terms of frequencies then MIDI or MP3 seems more "natural", and same once you realize that convolution neural networks for image processing care about some '2D-wavy' aspects of the picture [perhaps because convolution itself can be implemented using Fourier Transform?], JPEG with it's transforms seem "natural". I think MP3 and JPEG are like brain-to-brain protocol for telepathy, where we care more about representation-of-concept-in-actual-brain more than about "being able to look at the words and analyze the words themselves". MIDI seems to strike nice balance, as it talks about notes (frequencies and duration) without going too far. (I mean: it's a language you can still talk about, while I find it difficult to talk about entries in DFT matrix etc.)
HTML - I've already mentioned Wikipedia, and what I really cared about was hyperlinking, which gives more power to the recipient to navigate the discussion as they see most useful for them. Perhaps it would be interesting to figure out what would a symmetric protocol for that look like: one in which the sender and recipient have more equal power to steer the conversation. I guess this is REST, or Web as we know it.
SQL / GraphQL - here I don't mean just the language to query the data, but more the fact that the owner of the data in some sense wants to communicate the data to you, but instead of flooding you with the copy of all they know, they give you an opportunity to ask a very precise question so you'll get what you need to know. People in some sense try to do that, but first, they don't use very precise questions, second, don't "run the question inside my head" in the sense a server is running the query. I could imagine some alien brains communicate this way, that they send a small creature inside the brain of the other alien, to gather the info, and there are some rules which govern what is allowed for this creature to do while inside the host. This is a quite different way of communicating than the one where we just exchange words, I think, because for one thing, it let's you ask many questions in parallel, and questions can "mutate their internal state" while "being asked".
Rsync - when trying to think about network protocols, all of them seemed "linear" in that you have to order response with question, and patiently read/write characters of the stream to understand their meaning. The nice exception (at some level) is the rsync protocol, where one computer tries to learn what's new from another, by probing recursively smaller and smaller ranges, searching for disagreement. It's like a conversation in which we first try to establish if we agree about everything, and if not, then try to find a crux, and drill down etc. This could be parallelized, and perhaps made more passive for the server, if it just made "the Merkle of my knowlege about the world" publicly available, and then anyone could navigate it top-down to learn something new. In some sense Wikipedia is such a tree, but, first, it's not a tree (not clear what's the starting point, the root) and second, it tries to describe the one and only territory, not one of the many subjective maps of particular alien.
BitTorrent - the strange language in which to learn the answer you must exactly know the right question. Like the hash of what you need is the key, literally, and then you can get it from me. So, I'm willing to tell you anything, but only if you prove you would know it anyway, from someone else, and in fact, yes, you can ask the same question to several aliens at once, and get stitch pieces of their answers, and it just matches. Could be combined with the Merkle tree idea above. Well, actually what you'll get looks like... Git. A distributed repository of knowledge (and random stupid thoughts).
Lisp - a language in which you not only can introduce new nouns, verbs, and adjectives on the fly, as humans sometimes do, but also whole new grammar rules, together with their semantic. I'm not entirely sure what it would even mean/feel like to talk this way, perhaps because I'm a time-oriented ape.
Thanks for very interesting question! :)
Our universe is “local” - things only interact directly with nearby things, and only so many things can be nearby at once.
After reading this sentence, I had a short moment of illumination, that this is actually backwards: perhaps what our brains perceive as locality, is the property of "being influenced by/related to". Perhaps childs brain learns which "pixels" of retina are near each other, by observing they often have correlated colors, and similarly which places in space are nearby because you can move things or itself between them etc. So, whatever high-dimensional structure the real universe would have, we would still evolve to notice which nodes in the graph are connected and declare them "local". This doesn't mean, that the observation from the quoted sentence is a tautology: it wouldn't be true in a universe with much higher connectivity - we're lucky to live in a universe with a low [Treewidth](https://en.wikipedia.org/wiki/Treewidth), and thus can hope to grasp it.
This distinction between outcome- and process-oriented accountability strikes me a similar to System 1 vs System 2, or Plato's "Monster" vs "Man", or near- vs far-thinking, lizard- vs animal-brain, id vs ego, etc.: looks like nature had to solve similar problem when designing humans, so that they do not obsess to much on eating the cake now, but also not too much on figuring out the best way to get the cake in future, and it settled on having both systems in adversarial setting and gave them a meta-goal of figure out the balance between the two (that it is "we" feel bad when the two are in unresolved conflict).
If this analogy makes sense, then perhaps it's worth looking more closely at the "solution" - according to Plato, there is one more ingredient, "The Lion"=the social animal=super ego=trying to fit in/please/satisfy commitments to the other people, right? What would be the analog of that if we were to map it back to management world? Some form of mutual contracts between teams/workers as in https://en.wikipedia.org/wiki/Teal_organisation ?
And another analogy which comes to my mind is Reinforced Learning, which I don't know much about, but IIUC it's about figuring out algorithms which try to achieve long term goal by guessing the right short-term goals to pursue which align well with the long term goal. Supposedly the Evolution as a whole pursues the long term goal of "having as many grand-children as possible" to which end it imbues creatures with short-term goals like "get food, and seek sex", but importantly the way it figured out the mapping between long-term and short-term goal was by trial and error=generate and test=babble and filter=GAN=artist and critique="virtual engines"... and I don't know how AlphaStar did it, but I guess, by playing StarCraft2 for subjective millenia between myriads of mutants and letting the fittest survive. What would this mean if translated back to the world of management? Perhaps dividing the company into competing or at least diverse branches and set up their incentives in outcome-oriented way, but with the outcome being measured over very long periods, and leaving the definition of short-term goals and policies to the teams themselves (same way AlphaStar learns that it has lost a battle only after 2h of playing, but had to figure out how to mine minerals and steer soldiers to succeed)?