Posts

I'm still mystified by the Born rule 2021-03-04T02:35:32.301Z
So8res' Shortform Feed 2021-01-31T02:58:26.812Z
Probability interpretations: Examples 2019-05-11T20:32:14.841Z
Correspondence visualizations for different interpretations of "probability" 2019-05-10T17:10:23.317Z
Interpretations of "probability" 2019-05-09T19:16:21.650Z
MIRI's 2016 Fundraiser 2016-09-25T16:55:43.899Z
Safety engineering, target selection, and alignment theory 2015-12-31T15:43:18.481Z
MIRI's 2015 Winter Fundraiser! 2015-12-09T19:00:32.567Z
MIRI's 2015 Summer Fundraiser! 2015-08-19T00:27:44.535Z
MIRI's Approach 2015-07-30T20:03:51.054Z
MIRI Fundraiser: Why now matters 2015-07-24T22:38:06.131Z
Taking the reins at MIRI 2015-06-03T23:52:28.074Z
The Stamp Collector 2015-05-01T23:11:22.661Z
The path of the rationalist 2015-04-12T01:44:57.123Z
Ephemeral correspondence 2015-04-10T03:46:20.479Z
Blind artifacts 2015-04-07T23:23:17.542Z
Desire is the direction, rationality is the magnitude 2015-04-05T17:27:57.757Z
MIRI's technical agenda: an annotated bibliography, and other updates 2015-02-05T18:11:05.051Z
A different angle on UDT 2015-02-02T02:36:08.000Z
Why conditioning on "the agent takes action a" isn't enough 2015-02-02T02:35:51.000Z
The Value Learning Problem 2015-01-29T18:23:09.445Z
Formalizing Two Problems of Realistic World Models 2015-01-22T23:12:21.975Z
Vingean Reflection: Reliable Reasoning for Self-Improving Agents 2015-01-15T22:47:55.625Z
Questions of Reasoning under Logical Uncertainty 2015-01-09T17:37:54.807Z
MIRI's technical research agenda 2014-12-23T18:45:41.421Z
New paper from MIRI: "Toward idealized decision theory" 2014-12-16T22:27:39.519Z
MIRI Research Guide 2014-11-07T19:11:13.209Z
Introducing Corrigibility (an FAI research subfield) 2014-10-20T21:09:44.595Z
On Caring 2014-10-15T01:59:05.567Z
Newcomblike problems are the norm 2014-09-24T18:41:56.356Z
An introduction to Newcomblike problems 2014-09-20T18:40:25.528Z
Causal decision theory is unsatisfactory 2014-09-13T17:05:52.884Z
Knightian uncertainty: a rejection of the MMEU rule 2014-08-26T03:03:57.026Z
Knightian Uncertainty: Bayesian Agents and the MMEU rule 2014-08-04T14:05:47.758Z
Knightian uncertainty in a Bayesian framework 2014-07-24T14:31:54.420Z
Knightian Uncertainty and Ambiguity Aversion: Motivation 2014-07-21T20:32:43.289Z
Failures of an embodied AIXI 2014-06-15T18:29:23.442Z
Exploring Botworld 2014-04-30T22:29:44.052Z
Botworld: a cellular automaton for studying self-modifying agents embedded in their environment 2014-04-12T00:56:23.138Z
Book Review: Linear Algebra Done Right (MIRI course list) 2014-02-17T20:52:42.699Z
On saving the world 2014-01-30T20:00:05.353Z
Dark Arts of Rationality 2014-01-19T02:47:03.011Z
Deregulating Distraction, Moving Towards the Goal, and Level Hopping 2014-01-12T03:21:21.364Z
Habitual Productivity 2014-01-09T06:44:08.080Z
The mechanics of my recent productivity 2014-01-09T02:30:19.908Z
Walkthrough of the Tiling Agents for Self-Modifying AI paper 2013-12-13T03:23:55.984Z
Walkthrough of "Definability of Truth in Probabilistic Logic" 2013-12-09T03:44:42.065Z
Book Review: Computability and Logic 2013-11-21T13:52:02.094Z
On learning difficult things 2013-11-11T23:35:31.327Z
Very Basic Model Theory 2013-10-31T07:06:56.951Z

Comments

Comment by So8res on Reply to Nate Soares on Dolphins · 2021-06-18T03:51:48.939Z · LW · GW

How would you feel if you sunk forty months of your life into deconfusing a philosophical issue that had huge, life-altering practical stakes for you, and the response to your careful arguments from community authorities was a dismissive "haha yeah"? Would you, perhaps, be somewhat upset?

Perhaps! But also that doesn't seem to me like what happened. The response to your careful arguments was the 1000ish words that engage with what seemed to me to be the heart of your questions, and that attempted to convey some ways it seemed you were misunderstanding my communications. Also, the primary intent of the "haha yeah" was not dismissal, and the 100ish words following it were intended to convey some ways it seemed you were misunderstanding the linked twitter thread. Your apparent reading of my "haha yeah" as dismissive of your careful arguments looks to me like yet another way you're misunderstanding my communications.

(FWIW, as far as I can tell on very brief introspection, the main purpose of the 'haha yeah' was to hold open a space -- however small -- for online correspondence to be actually fun, with a comparatively minor side purpose of dismissing a very limited part of your argument. Furthermore, my conversational ethics do not permit such attempts at dismissal to be purely implicit, and the following paragraph attempts to render explicit the grounds on which I believe a specific narrow part of your argument deserved dismissal. There are forces in play whose names I don't readily have on this brief introspection, and my account might change on a deeper introspection, but I feel like I know where the dismissal is in all this, and don't expect I'd find that the main purpose was dismissal-based if I introspected further.)

My take on this conversation is that you're dramatically mismodelling me, and then running really far -- and getting quite distressed -- based on bad models. For instance, it looks to me like you read my 'haha yeah' as "the response", despite how it follows a thousand-odd words of engagement, and is sandwiched between a quote selecting a narrow portion of your argument and a paragraph explaining why I think that argument is off-kilter. I have a sense this sort of dramatic misread of my intentions has been happening repeatedly since the beginning, without engagement with my attempts at clarification, and I currently despair of communication.

I know you haven't solicited advice, but for future reference, if you were taking smaller steps in your assumptions about my intentions, and asking more and earlier questions (example: "What's the 'haha yeah' doing for you? (I'm reading it as broadly dismissive, and feel hurt by it)."), I both expect that you'd have a better chance of communicating with me, and that I'd be more enthusiastic about trying. (I acknowledge that this style of communication requires a high degree of trust, and that you might not have that trust with me.)

As things stand, I have the sense that you're lashing out wildly at shadow-versions of me you've constructed, and I'm not enthusiastic about engaging further on the object level (and my cheerful price is quite high, alas).

But when I do—and I will—if you have any scrap of human decency in your brain, you will not shitpost at me.

I've heard (perhaps incorrectly) that this is very emotionally salient to you, that you take it seriously, and that you care deeply about how I react. I feel an abstract compassion for you in this; it sounds like a thankless job and plausibly a virtuous one. Only abstract compassion, though, because my current emotional response is one that emits sentences like "well fuck you too; call me back when you can model me farther than you can throw me". It is from that state that I dispute your factual claim quoted above. My model of human mindspace contains variants of me who could retain some decency while shitposting in ways you perceive as being "at you". (Example: the thing I'm perceiving as you mismodelling-and-lashing-out hurts to be on the receiving end of, and I claim that humans can retain their decency while lashing out in response to being hurt. Also, I think the shitposting has purposes you're not modelling, and there are ways your response could be shaped such that shitposting in reply seems reasonable on its face.)

(ETA: the primary connotation of the above is something like "you seem to be attempting to apply pressure to me in a manner I consider invalid; it holds no sway over me" / "I have perceived this as a desperate simultaneous plea for both compassion and deference, and while I have the compassion, I lack the deference". Neither of those are quite right etc. I add this parenthetical out of an anticipation that, without it, you'll wildly misinterpret me here. I feel bitterness about how regularly and wildly you seem to misinterpret me, without apparent awareness or acknowledgement, and the bitterness tempts me to sharper phrasings.)

All that said, I've heard you as making a request that I avoid a certain type of glibness in my replies to you, on the grounds that they cause you great distress. Insofar as this doesn't cause me to stop interacting with you wholesale (eg, for fear that I'll cause you undue great distress given your apparent propensity to misread me), and insofar as I don't, like, forget when I'm banging out a very quick response, I predict I'll honor that request. And for the record, I do not intend to cause you distress, and I continue to respect your what-seems-to-me-like conviction in the pursuit of truth.

Comment by So8res on Reply to Nate Soares on Dolphins · 2021-06-15T23:20:46.521Z · LW · GW

I'm surprised you think this is "absolutely critical". Do you think I'm making a grave error in my newfound distaste for paraphyletic groupings? (My ability to notice their awkwardness felt internally like evidence that my joint-carving skills have improved over the years, ftr.) Is there some other joint-carving skill you believe I am lacking, or have lost? Or perhaps you're decrying a decay in general community epistemics, for which my thread is simply a poster-child? Or perhaps you're lamenting some general community or global decline of candor? I'm uncertain what precisely you're dismayed about, and solicit specific criticisms of my actions.

Comment by So8res on Reply to Nate Soares on Dolphins · 2021-06-11T06:41:06.770Z · LW · GW

Why? What changed?

On the object level question? Like, what changed between younger Nate who read that sequence and didn't object to that example, and older Nate who asserted (in a humorous context) that it takes staggering gymnastics to exclude dolphins from "fish"? Here's what springs to mind:

  • I noticed that there is cluster-nature to the marine-environment adaptations
  • I noticed that humans used to have a word for that cluster
  • I noticed a sense of inconsistency between including whales in "even-toed ungulates" without also including them in "lobe-finned fish".
  • I noticed that paraphyletic groupings seem to me to lack the courage of their convictions.
  • I learned that nature just keeps turning out crabs, which is a chink in the armor of phylogenetic classification schemes
  • I noticed my own discomfort as the lines around "fruit" and "berry" started wavering
  • I noticed specifically the distinction between "would be at home atop a fruit pizza" and "everything anatomically analogous to an apple"
  • I generalized, and noticed that I don't actually believe phylogenetics is generally a good way to carve up the life forms around me
  • I took stock of my phylogenetic concepts, and asked myself which concepts were lost but plausibly still useful
  • I decided that the concept previously known as "fish" (the marine adaptation cluster) had some plausible use. (In particular, the specific use that sprang to mind when I checked this a while back was the ability of kids to point at arbitrary marine creatures and say "look a fish", an influence that showed up in my thread.)

Scott's post is I think the source of the first two in me, the rest is novel to me as far as I recall.

Attempting to force the change into a single sentence, I'd say that the main thing that changed is I noticed the subtle background forces that whisper (at least to blue tribe members in their youth) "phylogenetic classification is the one true way to organize life forms", and rejected its claim.

For the record, I don't see my current state as incongruous with Eliezer's example. These days, if someone says that they have a definition of "fish" that includes salmon, guppies, sharks, dolphins, and trout but not jellyfish or algae, I might ask them how they classify camels before assuming they've made Eliezer's mistake #30. But I continue to think #30 is a real mistake (I'd gloss it as "insisting that your concept denotes an awkward boundary even after the boundary has been revealed to be awkward"), and that Eliezer's given example is still doing its job, despite the fact that I'd now say it gets a clarity boost from the subtle-implicit-insidious background cultural context on the ultimacy of phylogenetics, which it perhaps does not deserve.

None of this feels like a big reversal to me. I'd say that the hypothetical person in the example can be understood from context to be proposing a definition of "fish" useful for predicting features like breathing apparatus and bone structure (thanks to implicit, insidious, familiar, and universal background assumptions), and the "nitwit game" is to deny error and assert that the actual desideratum on the definition is that it be shaped in precisely thus-and-such an awkward way. I haven't changed my stance on that point. I also don't think it's a "nitwit game" to say "oh, this concept is actually intended for predicting existence of vertibrae, sorry for the confusion".

I think the place where I've made a decent-sized shift is that the idea of assigning a whole monosyllable to a notion that includes both the jawed and jawless fish, but excludes the shellfish, jellyfish, starfish, and the subset of jawed fish that routed through terrestrial forms, feels a lot more awkward to me than it used to. I don't really see myself using that concept much in practice -- and, like, if I'm watching a documentary on ocean creatures, I don't think I'm being done any favors by having word-structures that group lungfish and lamprey while excluding sea turtles. This is the sense in which it seems to me to take some gymnastics to arrive at the modern word "fish" (as opposed to, say, shorter words for bony fish and jawless fish, plus a "fish" word that includes bears, plus a "fish" word that includes cuttlefish and crayfish), and, furthermore, some gymnastics to find a variant of "fish" that does not include dolphins.

I don't stand by the claim that these gymnastics are "staggering" -- at least, not in the sense that the gymnastics themselves are all that tricky. (Like, I can kinda see it, and it's a fairly striaghtforward paraphyletic group as far as that goes, and there's at least a bunch of "I know it when I see it" style visual clustering). The thing that I find staggering is how pervasive and unremarked the gymnastics are. (And this sort of sloppiness -- being staggered by the size of the psyop, and tacking the adjective "staggering" onto the gymnastics -- is the sort of thing that I endorse while in a shitposting context.)

That said, notably, everyone can already perform these gymnastics. We're all taught to do these gymnastics implicitly, in youth, before we have the wherewithal to question them. Which means that relying on people to understand these (locally universal!) gymnastics when writing an example is perfectly reasonable.

In other words, I read Eliezer as saying something like "[assuming the standard gymnastics] failing to admit that the inclusion of dolphins was an error, and instead insisting that the concept boundary is supposed to contort, constitutes an additional error", and me as drawing attention to the standard gymnastics.

even if it was a shitpost, your epistemic-status followup thread still contends "but also y'all know i'm right"

haha yeah

For the record, the poor capitalization and informal tone there (and in the preceding tweet) were intended to be tells that those tweets were still being written from the "shitposting" frame (as were all my threads since April). The follow-up starting with "By which I mean", plus the remainder of the thread being well-punctuated, plus the final one ending by saying "And with that I'll end this thread; any more and I might slip back into cogent model-building, and I'm not quite ready to end my shitposting streak yet." were intended demarcate the switch from semitrolling to candor. Some of the things I value in shitposting are lost if I break the frame too readily in the threads themselves, so I'm rather loathe to give clearer signals in-context, but I'm generally happy to answer (publicly or privately) which tweets are spawned from which generators.


Apologies for the length. I'd make this shorter if I had more time.

Comment by So8res on Reply to Nate Soares on Dolphins · 2021-06-10T17:04:18.063Z · LW · GW

My epistemic status on that thread is, explicitly, "i been shitpostin since april". I do not wholeheartedly believe the argument I made; I acknowledge your counterpoints and appreciate you making them; I largely agree.

For the record, the fragment of my argument that I currently endorse is something like "be wary of those who say they are here to refine your concepts, for sometimes they are here to steal them". My current stance is that I'm not particularly entrenched on either side wrt "fish", and might push back about "fruit" and "berry" if I found those words being pulled on in the wild.

Another thing the thread attempts to convey, by dint of the visible exasperation and derision, is... something I don't have words for yet (related to why I shoved it into the connotations of a shitpost), but I'll take a few stabs: A visceral sense that the world is mad? A bid for a particular breed of reflexive intellectual independence? A sense of alarm that something sinister has made it past the defenses, in the quest to objectivify?

There's also a specific way in which I'm hurting about something here, and conveying it is another purpose of the exasperation and derision. (I'm not hurting in a bad, unsustainable, or worrying way, for the record.) And of course there are a variety of other reasons for the impolite conduct, including fun, flexing, and throwing some elbows around to put a check on how seriously people take me all the time around these parts.

(For whatever it's worth, I do endeavor to be careful when I throw my elbows around like this, by eg explicitly marking the context as shitposty, and by choosing broad targets and funny contexts before unleashing my ire.)

Having said all that, I hereby push back a little against this style of response to my shitpost. I value you (and others) pointing out weak points and impoliteness in my arguments. I also think the ability to throw elbows around in this way, and the ability to shove inarticulate feelings into the connotations in the right context, are important. The alternative, at least for me, is that I don't say things until I've done a heck of a lot more noticing and explicating the side-channel stuff, which means I say less and don't get to draw on the aid of friends for the noticing and explication. Also, it seems to me that our community has dramatically too little fun on social media. My pushback here is intended to increase the space we have for that.

(There are also other aspects of why I believe shitposting is healthy and good that I have not yet managed to articulate. I also acknowledge that it has costs, and suspect there are costs I could avoid paying if I understood them better.)

Finally, I'll note that that thread was not a veiled attempt to discuss gender issues. The idea that it could be read as subtweeting gender issues didn't cross my mind. The relevant Scott Alexander post didn't cross my mind (though I read it years and years ago, and I suspect it was an influence on my argument). I suspect that Scott's post played a causal role in me starting the thread out on fish (eg, by causing me to have fish-relevant parts of the argument cached in an articulate format), but the things that felt like the causes were rather: (1) someone using fish vs dolphins as an example classification problem (months ago); (2) someone asserting that pillbugs aren't bugs on account of being crustaceans (months ago); and (3) some entertaining banter with a marine biologist (the day before drafting the thread). Not worrying about whether I'm going to be read as subtweeting the controversial topic du jour is one of many benefits of the shitposting mindset, and I recommend it.

Comment by So8res on So8res' Shortform Feed · 2021-03-21T01:48:50.533Z · LW · GW

Crossposted from Twitter, might not engage much with comments on LW and I may or may not moderate replies.

Thread about a particular way in which jargon is great:

In my experience, conceptual clarity is often attained by a large number of minor viewpoint shifts. (A complement I once got from a research partner went something like "you just keep reframing the problem ever-so-slightly until the solution seems obvious". <3)

Sometimes a bunch of small shifts leave people talking a bit differently, b/c now they're thinking a bit differently. The old phrasings don't feel quite right -- maybe they conflate distinct concepts, or rely implicitly on some bad assumption, etc. (Coarse examples: folks who think in probabilities might become awkward around definite statements of fact; people who get into NVC sometimes shift their language about thoughts and feelings. I claim more subtle linguistic shifts regularly come hand-in-hand w/ good thinking.)

I suspect this phenomenon is one cause of jargon. Eg, when a rationalist says "my model of Alice wouldn't like that" instead of "I don't think Alice would like that", the non-standard phraseology tracks a non-standard way they're thinking about Alice. (Or, at least, I think this is true of me and of many of the folks I interact with daily. I suspect phraseology is contagious and that bystanders may pick up the alt manner of speaking w/out picking up the alt manner of thinking, etc.)

Of course, there are various other causes of jargon -- eg, it can arise from naturally-occurring shorthand in some specific context where that shorthand was useful, and then morph into a tribal signal, etc. etc. As such, I'm ambivalent about jargon. On the one hand, I prefer my communities to be newcomer-friendly and inclusive. On the other hand, I often hear accusations of jargon as a kind of thought-policing.

"Stop using phrases that meticulously track uncommon distinctions you've made; we already have perfectly good phrases that ignore those distinctions, and your audience won't be able to tell the difference!" No. My internal language has a bunch of cool features that English lacks. I like these features, and speaking in a way that reflects them is part of the process of transmitting them.

Example: according to me, "my model of Alice wants chocolate" leaves Alice more space to disagree than "I think Alice wants chocolate", in part b/c the denial is "your model is wrong", rather than the more confrontational "you are wrong". In fact, "you are wrong" is a type error in my internal tongue. My English-to-internal-tongue translator chokes when I try to run it on "you're wrong", and suggests (eg) "I disagree" or perhaps "you're wrong about whether I want chocolate".

"But everyone knows that "you're wrong" has a silent "(about X)" parenthetical!", my straw conversational partner protests. I disagree. English makes it all too easy to represent confused thoughts like "maybe I'm bad". If I were designing a language, I would not render it easy to assign properties like "correct" to a whole person -- as opposed to, say, that person's map of some particular region of the territory.

The "my model of Alice"-style phrasing is part of a more general program of distinguishing people from their maps. I don't claim to do this perfectly, but I'm trying, and I appreciate others who are trying.  And, this is a cool program! If you've tweaked your thoughts so that it's harder to confuse someone's correctness about a specific fact with their overall goodness, that's rad, and I'd love you to leak some of your techniques to me via a niche phraseology.

There are lots of analogous language improvements to be made, and every so often a community has built some into their weird phraseology, and it's *wonderful*. I would love to encounter a lot more jargon, in this sense. (I sometimes marvel at the growth in expressive power of languages over time, and I suspect that that growth is often spurred by jargon in this sense. Ex: the etymology of "category".)

Another part of why I flinch at jargon-policing is a suspicion that if someone regularly renders thoughts that track a distinction into words that don't, it erodes the distinction in their own head. Maintaining distinctions that your spoken language lacks is difficult! (This is a worry that arises in me when I imagine, eg, dropping my rationalist dialect.)

In sum, my internal dialect has drifted away from American English, and that suits me just fine, tyvm. I'll do my best to be newcomer-friendly and inclusive, but I'm unwilling to drop distinctions from my words just to avoid an odd turn of phrase.

Thank you for coming to my TED talk. Maybe one day I'll learn to cram an idea into a tweet, but not today.

Comment by So8res on I'm still mystified by the Born rule · 2021-03-05T05:20:42.352Z · LW · GW

Thanks! I expect I can stare at this and figure something out about why there is no reasonable notion of "triality" in Vect (ie, no 3-way analog of vector space duality -- and, like, obviously that's a little ridiculous, but also there's definitely still something I haven't understood about the special-ness of the dual space).

ETA: Also, I'm curious what you think the connection is between the "L2 is connected to bilinear forms" and "L2 is the only Lp metric invariant under nontrivial change of basis", if it's easy to state.

FWIW, I'm mostly reading these arguments as being variations on "if you put anything else than a 2 there, your life sucks", and I believe that, but I still have a sense that the explanation I'm looking for is more about how putting a 2 there is positively natural, not just the best of a bad lot. That said, I'm loving these arguments, and I expect I can mine them for some of the intuition-corrections I seek :-)

Comment by So8res on I'm still mystified by the Born rule · 2021-03-05T04:50:37.335Z · LW · GW

Thanks! This seems to me like another piece of the puzzle =D

In this case, this is one that I already had (at least, well enough for the hindsight bias to kick in :-p), and it's on my list of trailheads next time I try to ground out the 2 in the Born rule. FWIW, some lingering questions I have when I take this viewpoint include "ok, cool, why are there no corresponding situations where I want to compare 3 vectorish thingies?" / "I see why the argument works for 2, but I have a sneaking suspicion that this 2 is being slipped into the problem statement in a way that I'm not yet quite following". Also, I have a sense that there's some fairly important fact about anointing some linear isomorphism between a vector space and its dual as "canonical" that I have yet to grasp. Like, some part of the answer to "why do I never want to compare 3 vectorish thingies" is b/c the relationship between a vector space and its dual space is somehow pretty special, and there's no correspondingly special... triality of vector spaces. (Hrm. I wonder whether I can ground the 2 in the Born rule out into the 2 in categorical duality. That would be nuts.)

FWIW, one of my litmus tests is question "assuming we are supposed to measure distance in R2 using an L2 norm, why are we not supposed to measure distance in R3 using an L3 norm?". And, like, I have a bunch of explanations for why this is (including "L3 isn't invariant under most any change of basis" (per Alex's argument above) and "b/c the natural notion of distance between two points in R3 factors into two questions of distance in R2, so using L2 in R2 pins down using L2 in Rn"), but I still feel like there's some... surprising fixation on 2 here, that I can't yet explain to the satisfaction of young-Nate who guesses that cuberute(x^3 + y^3 + z^3)=r is the equation for a sphere. Like, I still feel kinda like math said "proof by induction: starting in the case where n=2, ..." and I'm like "wat why aren't we starting at 0" and it's like "don't worry, n < 2 will work out as special cases" and I'm like "ok, sure, this argument is valid, but also wtf are you doing". My wtfs aren't all ironed out yet.

And, maybe I'm chasing shadows (eg, seeking a logical explanation where there is none, which is the sort of thing that can happen to a poor sap who lacks an explicit understanding of logical causality), but my suspicion is that I'm still missing part of the explanation. And both this route (which I'd gloss as "understand why it's so important/natural/??? to have/choose/determine a canonical isomorphism between your vector space and its dual, then appeal to bilinearity", with a prereq of better understanding why we have/care-about duality but not triality in Vect) and Alex's (which I'd gloss as "explain why 2 obviously-should-be the balance point in the Lp norms") both feel like good trailheads to me.

(And, in case this wasn't clear to all readers, none of my assertions of confusion here are allegations that everyone else is similarly confused -- indeed, Alex and Adele have already given demonstrations to the contrary.)

<3 hooray.

Comment by So8res on I'm still mystified by the Born rule · 2021-03-05T03:44:22.958Z · LW · GW

Cool, thanks. Yeah, I don't have >50% on either of those two things holding up to philisophical progress (and thus, eg, I disagree that future theories need to agree with UDASSA on those fronts). Rather, happeningness-as-it-relates-to-multiple-simulations and happeningness-as-it-relates-to-the-simplicity-of-reality are precisely the sort of things where I claim Alice-style confusion, and where it seems to me like UDASSA is alledging answers while being unable to dissolve my confusions, and where I suspect UDASSA is not-even-wrong.

(In fact, you listing those two things causes me to believe that I failed to convey the intended point in my analogy above. I lean towards just calling this 'progress' and dropping the thread here, though I'd be willing to give a round of feedback if you wanna try paraphrasing or otherwise falsifying my model instead. Regardless, hooray for a more precise articulation of a disagreement!)

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T21:44:20.301Z · LW · GW

<3, this is exactly the sort of thought I claim to be missing when I say I still don't know how to trace the 2 in the Born rule back to its maker. This is a step I didn't yet have. It doesn't feel like the last piece I'm missing, but it does feel like a piece -- eg, now I can focus some attention on "why precisely is this crossover point at 2 / where is that 2 coming from?". Thanks!

(And ofc there's still a question about why we use an Lp norm, and indeed why we pass out gaze along the walls in a way that factors through the shadow on that wall, but I am fairly happy rolling most of that into the "what are we to learn from the fact that reality has the quantum nature" bundle.)

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T14:26:41.033Z · LW · GW

When you're using TMs to approximate physics, you have to balance the continuity of physics against the discreteness of the machines somehow. The easy thing to do is to discuss the limiting behavior of a family of machines that perform the simulation at ever-finer fidelity. I was doing this implicitly, for lack of desire to get into details.

And as I've said above, I'm not attempting to suggest that these naive approaches -- such as sampling a single classical state and reporting the positions of some things with arbitrary fidelity in the limit -- are reasonable ideas. Quite the opposite. What I'm trying to point out is that if all you have is a quantum state and the Born rule, you cannot turn it into a hypothesis without making a bunch of other choices, for which I know of no consensus answer (and for which I have not seen proposals that would resolve the problem to my satisfaction, though I have some ideas).

I agree that the correct way of making these choices will almost surely not involve recording any observation with infinite precision (in the limit).

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T06:54:45.109Z · LW · GW

Neat! I'd bet against that if I knew how :-) I expect UDASSA to look more like a red herring from the perspective of the future, with most of its answers revealed as wrong or not-even-wrong or otherwise rendered inapplicable by deep viewpoint shifts. Off the top of my head, a bet I might take is "the question of which UTM meta-reality uses to determine the simplicity of various realities was quite off-base" (as judged by, say, agreement of both EY and PC or their surrogates in 1000 subjective years).

In fact, I'm curious for examples of things that UDASSA seems to get right, that you think better theories must improve upon. (None spring to my own mind. Though, one hypothesis I have is that I've so-deeply-internalized all the aspects of UDASSA that seem obviously-true to me (or that I got from some ancestor-theory), that the only things I can percieve under that label are the controversial things, such that I am not attributing to it some credit that it is due. For instance, perhaps you include various pieces of the updateless perspective under that umbrella while I do not.)

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T05:14:19.334Z · LW · GW

How could an SI compare a deterministic theory to a probablistic one?

The deterministic theory gets probability proportional to 2^-length + (0 if it was correct so far else -infty), the probabilistic theory gets probability proportional to 2^-length + log(probability it assigned to the observations so far).

That said, I was not suggesting a solomonoff inductor in which some machines were outputting bits and others were outputting probabilities.

I suspect that there's a miscommunication somewhere up the line, and my not-terribly-charitable-guess is that it stems from you misunderstanding the formalism of Solomonoff induction and/or the point I was making about it. I do not expect to clarify further, alas. I'd welcome someone else hopping in if they think they see the point I was making & can transmit it.

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T05:09:34.608Z · LW · GW

Absolutely no effect does seem pretty counterintuitive to me, especially given that we know from QM that different levels of happeningness are at least possible.

I also have that counterintuition, fwiw :-p


I have the sense that you missed my point wrt UDASSA, fwiw. Having failed once, I don't expect I can transmit it rapidly via the medium of text, but I'll give it another attempt.

This is not going to be a particularly tight analogy, but:


Alice is confused about metaethics. Alice has questions like "but why are good things good?" and "why should we care about goodness?" and "if goodness is not objective, can I render murder good by deciding it's good?".

Bob is not confused about ethics. Bob can correctly answer many of Alice's questions: "good things are good b/c they result in good things such as, eg, human flourishing", and "because we like good consequences, such as human flourishing", and "no, because murder is not in fact good". (...I'm only subtweeting Sam Harris a little bit, here.)

The problem with these answers is not that they are incorrect. The problem with these answers is that they are not deconfusing, they are not identifying the box that Alice is trapped in and freeing her from it.

Claire is not confused about metaethics. Claire can state correct answers to the questions that Alice did not know she was asking, such as "Alice!goodness is more-or-less a fixed logical funtion; Alice!goodness is perhaps slightly different from Claire!goodness but they are close enough as to make no difference against the space of values; this fixed logical function was etched into your genes by eons of sex and death; it is however good, and other logical functions in its place would not be."

The problem with these answers is not that they are incorrect, as answers to the questions that Alice would have been asking were she freed from her box (although, once she's glimpsed the heretofore hidden degree of freedom, she's unlikely to need to actually ask those questions). The problem with these answers is that they are not meeting Alice at the point of her confusion. To her, they sound sort of odd, and do not yet have a distinguishable ring of truth.

What Alice needs in this hypothetical is a bunch of thought-experiments, observations, and considerations that cause her to percieve the dimension along which her hypotheses aren't yet freed, so that the correct hypothesis can enter her view / so that her mind can undergo a subtle shift-in-how-she-frames-the-question such that the answers Claire gives suddenly become intuitively clear. She's probably going to need to do a lot of the walking herself. She needs questions and nudges, not answers. Or something. (This is hard to articulate.)


I claim that my state wrt various anthropic questions -- such as the ol' trilemma -- is analogous to that of Alice. I expect that becoming deconfused about the trilemma to feel like a bunch of changes to my viewpoint that cause the correct hypothesis to enter my view / that cause my mind to undergo a shift-in-how-I-frame-the-question such that the correct answer to snaps into focus. (This is still hard to articulate. I don't think my words have captured the core. Hopefully they have waved in the right direction.) More generally, I claim to know what deconfusion looks like, and I can confidently assert that UDASSA hasn't done it for me yet.

Like, for all I know, the odd shit UDASSA says to me is like the phrases Claire says to Alice -- correct, topical, but odd-seeming and foreign from my current state of confusion. Perhaps there's a pathway through the valley of my confusion that causes me to shift my understanding of (eg) the trilemma, such that the problem falls away, and I start emitting UDASSA-like sentences on the other side, but if so I have not yet found it.

And, as someone in the Claire-state wrt the problem of metaethics, I claim that I would be able to go back and walk Alice through the valley, to the point where she was happily emitting Claire-statements all her own. (Or, at least, I'd have a pretty good hit-rate among particularly sharp friends.) And I have not been able to cause any UDASSA-er to walk me through the valley. And also a number of the UDASSA-moves smell to me like missteps -- perhaps b/c I'm bad at requesting it, but also perhaps b/c UDASSA doesn't do the thing. All told, my guess is that it's making about as much progress at resolving the core confusions as it looks like it's making -- ie, not much.

(To be clear, I have managed to get UDASSA to tell me why I shouldn't be confused about the trilemma. But this is not the currency I seek, alas.)

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T04:44:33.660Z · LW · GW

Yeah yeah, this is the problem I'm referring to :-)

I disagree that you must simulate collapse to solve this problem, though I agree that that would be one way to do it. (The way you get the right random numbers, fwiw, is from sample complexity -- SI doesn't put all its mass on the single machine that predicts the universe, it allocates mass to all machines that have not yet erred in proportion to their simplicity, so probability mass can end up on the class of machines, each individually quite complex, that describe QM and then hardcode the branch predictions. See also the proof about how the version of SI in which each TM outputs probabilities is equivalent to the version where they don't.)

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T04:35:39.368Z · LW · GW

^_^

Also, thanks for all the resource links!

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T04:20:43.295Z · LW · GW

To be clear, the process that I'm talking about for turning a quantum state into a hypothesis is not intended to be a physical process (such as a measurement), it's intended to be a Turing machine (that produces output suitable for use by Solomonoff induction).

That said, to be clear, I don't think this is a fundamentally hard problem. My point is not "we have absolutely no idea how to do it", it's somehing more like "there's not a consensus answer here" + "it requires additional machinery above and beyond [the state vector + the born rule + your home address]" + "in my experience, many (non-specialist-physicist) people don't even register this as a problem, and talk about the Born rule as if it's supposed to fill this gap".

I agree that there are a bunch of reasonable additional pieces of machinery you can use to get the missing piece (such as choice of a "measurement basis"); my own suspicion is that the right answer looks a bit different from what I've seen others propose (and routes through, eg, machinery that lets you read off the remembered history, as opposed to machinery that picks some subbasis to retain); my guess is that there are experimental differences in theory but they're probably tricky to create in practice, but I haven't worked through the details myself.

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T04:10:46.586Z · LW · GW

I agree that the problem doesn't seem too hard, and that there are a bunch of plausible-seeming theories. (I have my own pet favorites.)

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T04:07:59.515Z · LW · GW

This is precisely the thought that caused me to put the word 'apparent' in that quote :-p. (In particular, I recalled the original UDASSA post asserting that it took that horn, and this seeming both damning-to-me and not-obviously-true-for-the-reason-you-state, and I didn't want to bog my comment down, so I threw in a hedge word and moved on.) FWIW I have decent odds on "a thicker computer (and, indeed, any number of additional copies of exactly the same em) has no effect", and that's more obviously in contradiction with UDASSA.

Although, that isn't the name of my true objection. The name of my true objection is something more like "UDASSA leaves me no less confused, gives me no sense of "aha!", or enlightenment, or a-mystery-unraveled, about the questions at hand". Like, I continue to have the dualing intuitions "obviously more copies = more happening" and "obviously, setting aside how it's nice for friends to have backup copies in case of catastrophe, adding an identical em of my bud doesn't make the world better, nor make their experiences different (never mind stronger)". And, while UDASSA is a simple idea that picks a horse in that race, it doesn't... reveal to each intuition why they were confused, and bring them into unison, or something?

Like, perhaps UDASSA is the answer and I simply have not yet figured out how to operate it in a way that reveals its secrets? But I also haven't seen anyone else operate it in a way that reveals the sort of things that seem-like-deconfusion-to-me, and my guess is that it's a red herring.

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T03:46:47.803Z · LW · GW

I agree that the Born rule is just the poster child for the key remaining confusions (eg, I would have found it similarly natural to use the moniker "Hilbert space confusions").

I disagree about whether UDASSA contains much of the answer here. For instance, I have some probability on "physics is deeper than logic" being more-true-than-the-opposite in a way that ends up tossing UDASSA out the window somehow. For another instance, I weakly suspect that "running an emulation on a computer with 2x-as-thick wires does not make them twice-as-happening" is closer to the truth than the opposite, in apparent contradiction with UDASSA. More generally, I'm suspicious of the whole framework, and the "physics gives us hints about the UTM that meta-reality uses" line of attack feels to me like it has gone astray somewhere. (I have a bunch more model here, but don't want to go into it at the moment.)

I agree that these questions likely go to the heart of population ethics as well as anthropics :-)

Comment by So8res on I'm still mystified by the Born rule · 2021-03-04T03:33:36.216Z · LW · GW

I agree that there's a difference between "put a delta-spike on the single classical state you sampled" and "zero out amplitude on all states not consistent with the observation you got from your sample". I disagree that using the latter to generate a sensory stream from a quantum state yields reasonable predictions -- eg, taken literally I think you're still zeroing out all but a measure-zero subset of the position basis, and I expect the momenta to explode immediately. You can perhaps get this hypothesis (or the vanilla delta spike) hobbling by trying to smooth things out a bit (eg, keep a Gaussian centered on each classical state in which you made the sampled observation), but I still expect this to be experimentally distinguishable from what really happens (eg by way of some quantum-eraser-style hijinks or other sizeable entanglements), though I haven't checked the details myself.

Comment by So8res on Are the Born probabilities really that mysterious? · 2021-03-04T02:56:13.409Z · LW · GW

Warning: the post doesn't attempt to answer your question (ie, "can we reduce the Born rule to conservation of information?"). I don't know the answer to that. Sorry.

My guess is that a line can be drawn between the two; I'm uncertain how strong it can be made.

This may be just reciting things that you already know (or a worse plan than your current one), but in case not, the way I'd attempt to answer this would be:

  1. Solidly understand how to ground out the Born rule in the inner product. (https://arxiv.org/abs/1405.7907 might be a good place to start if you haven't already done this? I didn't personally like the narrative there, but I found some of the framework helpful.)
  2. Recall the details of that one theorem that relates unitary evolution to conservation of information.
  3. Meditate on the connection between unitary operators, orthonormal bases, and the inner product.
  4. See if there's a compelling link to be made that runs from the Born rule, through the inner product operator, through unitarity, to conservation of information.

Also, I find this question interesting and am also curious for an answer :-)

Comment by So8res on Are the Born probabilities really that mysterious? · 2021-03-04T02:39:56.798Z · LW · GW

I tried to commentate, and accidentally a whole post. Short version: I think one or two of the many mysteries people tend to find swirling around the Born rule are washed away by the argument you mention (regardless of how tight the analogy to Liouville's theorem), but some others remain (including the one that I currently consider central).

Comment by So8res on So8res' Shortform Feed · 2021-03-03T20:35:12.898Z · LW · GW

Crossposted from Twitter, might not engage much with comments on LW and I may or may not moderate replies.

PSA: In my book, everyone has an unlimited number of "I don't understand", "plz say that again in different words", "plz expand upon that", and "plz pause while I absorb that" tokens.

Possession of an unlimited number of such tokens (& their ilk) is one of your sacred rights as a fellow curious mind seeking to understand the world around you. Specifically, no amount of well-intentioned requests for clarification or thinking time will cause me to think you're an idiot.

I might conclude that there's more ground to cover than I hoped; I may despair of communicating quickly; I might rebudget my attention. But asking for clarification or time-to-consider won't convince me you're a fool. (In fact, I consider it evidence to the contrary!). 

If you ask loads of Qs, I sometimes get frustrated. Sometimes b/c my goals were frustrated, but more often it's b/c I tend to feel strong visceral frustration when asked for clarification re a concept I kinda understand, don't fully understand, and wish to understand. (I find this breed of frustration quite valuable -- it tends to spur me to gain deeper understandings of things I care about. My apologies to those who ask me Qs, watch me get viscerally frustrated, and believe that the frustration is directed at them. Known rent-paying bug.)

I'm not saying that spending these tokens will never cost you. Noticing a big inferential gap can sap my desire to continue, which may sting, etc. etc. But if I do cease my attempts to convey, it will be with sadness -- towards my lack of time, not your lack of smarts.

(Also, tbc, the usual outcome is not that I despair of communicating. In my experience, communicating w/ high caliber thinkers on technical topics involves regular and repeated expenditure of these tokens.)

Comment by So8res on So8res' Shortform Feed · 2021-01-31T02:57:00.552Z · LW · GW

Crossposted from Twitter, might not engage much with comments on LW and I may or may not moderate replies

Hypothesis: English is harmed by conventions against making up new plausible-sounding words, as this contributes to conventions like pretending numbers are names (System 1 deliberation; Type II errors) and naming things after people (Bayesian reasoning, Cartesian products).

I used to think naming great concepts after people was a bad idea (eg, "frequency decomposition" is more informative & less scary than "Fourier transformation"). I now suspect that names are more-or-less the only socially accepted way to get new words for new technical concepts.

I'd love a way to talk about how a change in perspective feels Fourier-ish without the jargon 'Fourier', but given that current social conventions forbid adding 'freqshift' (or w/e) to our language, perhaps I'll instead celebrate that we don't call them "Type IV transformations"

tbc, I still think that naming great concepts after people is silly, but now I suspect that if the math & science communities halted that practice we'd be worse off, at least until it stopped being lame to invent new words for new concepts.

Comment by So8res on So8res' Shortform Feed · 2021-01-31T02:54:38.262Z · LW · GW

Crossposted from Twitter, might not engage much with comments on LW and I may or may not moderate replies. 

Hypothesis: we're rapidly losing the cultural technology to put people into contact with new ideas/worldviews of their own volition, ie, not at the recommendation of a friend or acquaintance.

Related hypothesis: it's easier for people to absorb & internalize a new idea/worldview when the relationship between them and the idea feels private. Ex: your friend is pushing a viewpoint onto you, and you feel some social pressure to find at least one objection.

See also "it's difficult to make a big update when people are staring at you". The old internet (& libraries) put people in contact with new ideas privately; the new internet puts you in contact with new ideas that your friends are peddling.

(Perhaps the shift in the internet's memetic focus -- eg from atheism in the 00's to social justice in the 10's -- is explained in part by the older memes thriving when found privately, and the newer thriving when pushed by a friend?)

Comment by So8res on What I’ll be doing at MIRI · 2019-11-15T18:22:15.426Z · LW · GW
I have discussed with MIRI their decision to make their research non-disclosed-by-default and we agreed that my research agenda is a reasonable exception.

Small note: my view of MIRI's nondisclosed-by-default policy is that if all researchers involved with a research program think it should obviously be public then it should obviously be public, and that doesn't require a bunch of bureaucracy. I think this while simultaneously predicting that when researchers have a part of themselves that feels uncertain or uneasy about whether their research should be public, they will find that there are large benefits to instituting a nondisclosed-by-default policy. But the policy is there to enable researchers, not to annoy them and make them jump through hoops.

(Caveat: within ML, it's still rare for risk-based nondisclosure to be treated as a real option, and many social incentives favor publishing-by-default. I want to be very clear that within the context of those incentives, I expect many people to jump to "this seems obviously safe to me" when the evidence doesn't warrant it. I think it's important to facilitate an environment where it's not just OK-on-paper but also socially-hedonic to decide against publishing, and I think that these decisions often warrant serious thought. The aim of MIRI's disclosure policy is to remove undue pressures to make publication decisions prematurely, not to override researchers' considered conclusions.)

Comment by So8res on On motivations for MIRI's highly reliable agent design research · 2017-01-29T16:01:12.000Z · LW · GW

The second statement seems pretty plausible (when we consider human-accessible AGI designs, at least), but I'm not super confident of it, and I'm not resting my argument on it.

The weaker statement you provide doesn't seem like it's addressing my concern. I expect there are ways to get highly capable reasoning (sufficient for, e.g., gaining decisive strategic advantage) without understanding low-K "good reasoning"; the concern is that said systems are much more difficult to align.

Comment by So8res on On motivations for MIRI's highly reliable agent design research · 2017-01-29T01:53:34.000Z · LW · GW

As I noted when we chatted about this in person, my intuition is less "there is some small core of good consequentialist reasoning (it has “low Kolmogorov complexity” in some sense), and this small core will be quite important for AI capabilities" and more "good consequentialist reasoning is low-K and those who understand it will be better equipped to design AGI systems where the relevant consequentialist reasoning happens in transparent boxes rather than black boxes."

Indeed, if I thought one had to understand good consequentialist reasoning in order to design a highly capable AI system, I'd be less worried by a decent margin.

Comment by So8res on My current take on the Paul-MIRI disagreement on alignability of messy AI · 2017-01-13T02:11:45.000Z · LW · GW

Weighing in late here, I'll briefly note that my current stance on the difficulty of philosophical issues is (in colloquial terms) "for the love of all that is good, please don't attempt to implement CEV with your first transhuman intelligence". My strategy at this point is very much "build the minimum AI system that is capable of stabilizing the overall strategic situation, and then buy a whole lot of time, and then use that time to figure out what to do with the future." I might be more optimistic than you about how easy it will turn out to be to find a reasonable method for extrapolating human volition, but I suspect that that's a moot point either way, because regardless, thou shalt not attempt to implement CEV with humanity's very first transhuman intelligence.

Also, +1 to the overall point of "also pursue other approaches".

Comment by So8res on MIRI's 2016 Fundraiser · 2016-11-03T22:24:06.990Z · LW · GW

Thanks :-)

Comment by So8res on MIRI's 2016 Fundraiser · 2016-11-01T23:30:26.238Z · LW · GW

Thanks!

Comment by So8res on MIRI's 2016 Fundraiser · 2016-10-19T23:21:01.143Z · LW · GW

Fixed, thanks.

Comment by So8res on MIRI's 2016 Fundraiser · 2016-10-04T20:41:49.220Z · LW · GW

Huh, thanks for the heads up. If you use an ad-blocker, try pausing that and refreshing. Meanwhile, I'll have someone look into it.

Comment by So8res on MIRI's 2016 Fundraiser · 2016-09-26T18:39:53.142Z · LW · GW

Thanks!

Comment by So8res on Double Corrigibility: better Corrigibility · 2016-05-02T19:25:55.266Z · LW · GW

FYI, this is not what the word "corrigibility" means in this context. (Or, at least, it's not how we at MIRI have been using it, and it's not how Stuart Russell has been using it, and it's not a usage that I, as one of the people who originally brought that word into the AI alignment space, endorse.) We use the phrase "utility indifference" to refer to what you're calling "corrigibility", and we use the word "corrigibility" for the broad vague problem that "utility indifference" was but one attempt to solve.

By analogy, imagine people groping around in the dark attempting to develop probability theory. They might call the whole topic the topic of "managing uncertainty," and they might call specific attempts things like "fuzzy logic" or "multi-valued logic" before eventually settling on something that seems to work pretty well (which happened to be an attempt called "probability theory.") We're currently reserving the "corrigibilty" word for the analog of "managing uncertainty"; that is, we use the "corrigibility" label to refer to the highly general problem of developing AI algorithms that cause a system to (in an intuitive sense) reason without incentives to deceive/manipulate, and to reason (vaguely) as if it's still under construction and potentially dangerous :-)

Comment by So8res on Safety engineering, target selection, and alignment theory · 2015-12-31T16:28:33.075Z · LW · GW

By your analogy, one of the main criticism of doing MIRI-style AGI safety research now is that it's like 10th century Chinese philosophers doing Saturn V safety research based on what they knew about fire arrows.

This is a fairly common criticism, yeah. The point of the post is that MIRI-style AI alignment research is less like this and more like Chinese mathematicians researching calculus and gravity, which is still difficult, but much easier than attempting to do safety engineering on the Saturn V far in advance :-)

Comment by So8res on Why CFAR? The view from 2015 · 2015-12-18T17:37:15.501Z · LW · GW

Yes, precisely. (Transparency illusion strikes again! I had considered it obvious that the default outcome was "a few people are nudged slightly more towards becoming AI alignment researchers someday", and that the outcome of "actually cause at least one very talented person to become AI alignment researcher who otherwise would not have, over the course of three weeks" was clearly in "resounding success" territory, whereas "turn half the attendees into AI alignment researchers" is in I'll-eat-my-hat territory.)

Comment by So8res on Why CFAR? The view from 2015 · 2015-12-18T02:44:40.765Z · LW · GW

I don't claim that it developed skill and talent in all participants, nor even in the median participant. I do stand by my claim that it appears to have had drastic good effects on a few people though, and that it led directly to MIRI hires, at least one of which would not have happened otherwise :-)

Comment by So8res on MIRI's 2015 Winter Fundraiser! · 2015-12-15T16:26:08.074Z · LW · GW

Thanks! :-p It's convenient to have the 2015 fundraisers end before 2015 ends, but we may well change the way fundraisers work next year.

Comment by So8res on MIRI's 2015 Winter Fundraiser! · 2015-12-15T04:45:26.581Z · LW · GW

Thanks! Our languages and frameworks definitely have been improving greatly over the last year or so, and I'm excited to see where we go now that we've pulled a sizable team together.

Comment by So8res on LessWrong 2.0 · 2015-12-13T23:55:29.262Z · LW · GW

I have the requisite decision-making power. I hereby delegate Vaniver to come up with a plan of action, and will use what power I have to see that that plan gets executed, so long as the plan seems unlikely to do more harm than good (but regardless of whether I think it will work). Vaniver and the community will need to provide the personpower and the funding, of course.

Comment by So8res on MIRI's 2015 Winter Fundraiser! · 2015-12-11T23:08:03.037Z · LW · GW

Thanks! And thanks again for your huge donation in the summer; I was not expecting more.

Comment by So8res on MIRI's 2015 Winter Fundraiser! · 2015-12-10T19:34:16.359Z · LW · GW

Thanks!

Comment by So8res on MIRI's 2015 Winter Fundraiser! · 2015-12-10T19:34:09.903Z · LW · GW

Thanks!

Comment by So8res on MIRI's 2015 Winter Fundraiser! · 2015-12-08T15:28:56.733Z · LW · GW

Thanks!

Comment by So8res on MIRI's 2015 Winter Fundraiser! · 2015-12-08T04:05:19.330Z · LW · GW

Thanks!

Comment by So8res on MIRI's 2015 Winter Fundraiser! · 2015-12-08T00:23:55.128Z · LW · GW

Thanks!

Comment by So8res on The trouble with Bayes (draft) · 2015-10-26T19:23:49.907Z · LW · GW

I mostly agree here, though I'm probably less perturbed by the six year time gap. It seems to me like most of the effort in this space has been going towards figuring out how to handle logical uncertainty and logical counterfactuals (with some reason to believe that answers will bear on the question of how to generate priors), with comparatively little work going into things like naturalized induction that attack the problem of priors more directly.

Can you say any more about alternatives you've been considering? I can easily imagine a case where we look back and say "actually the entire problem was about generating a prior-like-thingy" but I have a harder time visualizing different tacts altogether (that don't eventually have some step that reads "then treat observations like Bayesian evidence").

Comment by So8res on The trouble with Bayes (draft) · 2015-10-25T23:25:40.495Z · LW · GW

Yeah, I also have nontrivial odds on "something UDTish is more fundamental than Bayesian inference" / "there are no probabilities only values" these days :-)

Comment by So8res on The trouble with Bayes (draft) · 2015-10-24T18:19:32.502Z · LW · GW

If the Bayesian's ignoring information, then you gave them the wrong prior. As far as I can tell, the objection is that the prior over theta which doesn't ignore the information depends on pi, and intuitions say that Bayesians should think that pi should be independent from theta. But if theta can be chosen in response to pi, then the Bayesian prior over theta had better depend on pi.

I wasn't saying that this problem is "adversarial" in the "you're punishing Bayesians therefore I don't have to win" way; I agree that that would be a completely invalid argument. I was saying "if you want me to succeed even when theta is chosen by someone who doesn't like me after pi is chosen, I need a prior over theta which depends on pi." Then everything works out, except that Robins and Wasserman complain that this is torturing Bayesiansim to give a frequentist answer. To that, I shrug. You want me to get the frequentist result ("no matter which theta you pick I converge") then the result will look frequentist. Not much surprise there.

This is a very natural problem that comes up constantly.

You realize that the Bayesian gets the right answer way faster than the frequentist in situations where theta is discrete, or sufficiently smooth, or parametric, right? I doubt you find problems like this where theta is non-parametric and utterly discontinuous "naturally" or "constantly". But even if you do, the Bayesian will still succeed with a prior over theta that is independent of pi, except when the pi is so complicated and theta that is so discontinuous and so precisely tailored to hiding information in places that pi makes it very very difficult to observe that the only way you can learn theta is by knowing that it's been tailored to that particular pi. (The frequentist is essentially always assuming that theta is tailored to pi in this way, because they're essentially acting like theta might have been selected by an adversary, because that's what you do if you want to converge in all cases.) And even in that case the Bayesian can succeed by putting a prior on theta that depends on pi. What's the problem?

Imagine there's a game where the two of us will both toss an infinite number of uncorrelated fair coins, and then check which real numbers are encoded by these infinite bit sequences. Using any sane prior, I'll assign measure zero to the event "we got the same real number." If you're then like "Aha! But what if my coin actually always returns the same result as yours?" then I'm going to shrug and use a prior which assigns some non-zero probability to a correlation between our coins.

Robins and Wasserman's game is similar. We're imagining a non-parametric theta that's very difficult to learn about, which is like the first infinite coin sequence (and their example does require that it encode infinite information). Then we also imagine that there's some function pi which makes certain places easier or harder to learn about, which is like the second coin sequence. Robins and Wasserman claim, roughly, that for some finite set of observations and sufficiently complicated pi, a reasonable Bayesian will place ~zero probability on theta just happening to hide all its terrible discontinuities in that pi in just such a way that the only way you can learn theta is by knowing that it is one of the thetas that hides its information in that particular pi; this would be like the coin sequences coinciding. Fine, I agree that under sane priors and for sufficiently complex functions pi, that event has measure zero -- if theta is as unstructured as you say, it would take an infinite confluence of coincident events to make it one of the thetas that happens to hide all its important information precisely such that this particular pi makes it impossible to learn.

If you then say "Aha! Now I'm going to score you by your performance against precisely those thetas that hide in that pi!" then I'm going to shrug and require a prior which assigns some non-zero probability to theta being one of the thetas that hides its info in pi.

That normally wouldn't require any surgery to the intuitive prior (I place positive but small probability on any finite pair of sequences of coin tosses being identical), but if we're assuming that it actually takes an infinite confluence of coincident events for theta to hide its info in pi and you still want to measure me against thetas that do this, then yeah, I'm going to need a prior over theta that depends on pi. You can cry "that's violating the spirit of Bayes" all you want, but it still works.

And in the real world, I do want a prior which can eventually say "huh, our supposedly independent coins have come up the same way 2^trillion times, I wonder if they're actually correlated?" or which can eventually say "huh, this theta sure seems to be hiding lots of very important information in the places that pi makes it super hard to observe, I wonder if they're actually correlated?" so I'm quite happy to assign some (possibly very tiny) non-zero prior probability on a correlation between the two of them. Overall, I don't find this problem perturbing.

You can't really say "oh I believe in the likelihood principle," and then rule out examples where the principle fails as unnatural or adversarial.

I agree completely!