Reading through Backdoors as an analogy for deceptive alignment [LW · GW] prompted me to think about a LW feature I might be interested in. I don't have much math background, and have always found it very effortful to parse math-heavy posts. I expect there are other people in a similar boat.

In modern programming IDEs it's common to have hoverovers for functions and variables, and I think it's sort of crazy that we don't have that for math. So, I'm considering a LessWrong feature that:

• takes in a post (i.e. when you save or go to publish a draft)
• identifies the LaTeX terms in the post
• creates a glossary for what each term means. (This should probably require confirmation by the author)
• makes a hoverover for each term so when you mouseover it reminds you.

On "Backdoors", I asked the LessWrong-integrated LLM: "what do the Latex terms here mean"?

It replied :

The LaTeX symbols in this passage represent mathematical notations. Let me explain each of them:

1. $F$ : This represents a class of functions. The curly F denotes that it's a set or collection of functions.
2. $f\in F$ : This means that $f$ is a function that belongs to (is an element of) the class $F$.
3. $f^{\ast }$ : The asterisk superscript typically denotes a modified or special version of something. Here, it represents the backdoored function.
4. $\epsilon$ : This Greek letter (epsilon) is often used to denote a small quantity or tolerance.
5. $x^{\ast }$ : Similar to $f^{\ast }$, this represents a special input, in this case, the backdoor trigger.

These symbols are used to formally describe the game between the attacker and defender:

• $F$ is the set of all possible functions the attacker can choose from.
• $f$ is the original function chosen by the attacker.
• $f^{\ast }$ is the backdoored function created by the attacker.
• $\epsilon$ is the tolerance for how similar $f$ and $f^{\ast }$ must be on most inputs.
• $x^{\ast }$ is the backdoor trigger input, on which $f$ and $f^{\ast }$ behave differently.

This looks at least pretty close to the intended meaning, but curious if it feels like it's wrong or missing nuance in any places @Jacob_Hilton [LW · GW] @Mark Xu [LW · GW]. Also curious whether this would be helpful to people.

The “prompt shut down” clause seemed like one of the more important clauses in the SB 1047 bill. I was surprised other people I talked to didn't think seem to think it mattered that much, and wanted to argue/hear-arguments about it.

The clauses says AI developers, and compute-cluster operators, are required to have a plan for promptly shutting down large AI models.

People's objections were usually:

"It's not actually that hard to turn off an AI – it's maybe a few hours of running around pulling plugs out of server racks, and it's not like we're that likely to be in the sort of hard takeoff scenario where the differences in a couple hours of manually turning it off will make the difference."

I'm not sure if this is actually true, but, assuming it's true, it still seems to me like the shutdown clause is the one of the more uncomplicatedly-good parts of the bill.

Some reasons:

1. I think the ultimate end game for AI governance will require being able to quickly notice and shut down rogue AIs. That's what it means for the acute risk period to end. 

2. In the more nearterm, I expect the situation where we need to stop running an AI to be fairly murky. Shutting down an AI is going to be very costly. People don't like doing costly things. People also don't like doing things that involve lots of undocumented, complex manual decisions that are going to be a pain. If a company (or compute cluster) doesn't have an explicit plan for how to shut down an AI, I think they're a lot less likely to do it. In particular if it'd be a big economic loss, and it's not entirely obvious they have to. 

If a government is trying to impose this cost from the outside, and a company doesn't want to, they'll probably make a bunch of arguments about how unreasonable and/or impossible the request is.

3. I also think "shut it all down" is something that might be important to do, and while not currently in the overton window, might be in the overton window later. 

I think making "prompt shutdown" a concrete task that companies and governments are thinking about make it significantly more likely to happen. And I think/hope it'll serve as a building-block scaffold, such that later both governments and companies will have an easier time considering plans that include "prompt shutdown" as a component.

More "straightforwardly good."

There's a lot in the bill that I think is probably good, but, does depend on how things get enforced. For example, I think it's good to require companies to have a plan to reasonably-assure that their AIs are good. But, I've heard some people be concerned "aren't basically all SSP-like plans basically fake? is this going to cement some random bureaucratic bullshit rather than actual good plans?." And yeah, that does seem plausible. 

I'd take the risk of that on current margins. But "if you're running a big model, you need to have the capacity to turn it off quickly" seems like a just pretty reasonable, necessary piece of legislation?

Motif coming up for me: a lot of skill ceilings are much higher than you might think, and worth investing in.

Some skills that you can be way better at:

• Listening to people, and hearing what they're actually trying to say, and gaining value from it
• Noticing subtle things that are important. You can learn to notice like 5 different things happening inside you or around you, that occured in <1 second.
• Being concrete, in ways that help you resolve confusion and gain momentum on solving problems.
• Each stage of OODA Looping is quite deep
  • (i.e. "Observe", "Orient", "Decide", and "Act" each have a lot of deep subskills. The depth of "Noticing" is a subset of the overall set of "Observation" skills")

There was a particular mistake I made over in this thread [LW(p) · GW(p)]. Noticing the mistake didn't change my overall position (and also my overall position was even weirder than I think people thought it was). But, seemed worth noting somewhere.

I think most folk morality (or at least my own folk morality), generally has the following crimes in ascending order of badness:

• Lying
• Stealing
• Killing
• Torturing people to death (I'm not sure if torture-without-death is generally considered better/worse/about-the-same-as killing)

But this is the conflation of a few different things. One axis I was ignoring was "morality as coordination tool" vs "morality as 'doing the right thing because I think it's right'." And these are actually quite different. And, importantly, you don't get to spend many resources on morality-as-doing-the-right-thing unless you have a solid foundation of the morality-as-coordination-tool.

There's actually a 4x3 matrix you can plot lying/stealing/killing/torture-killing into which are:

• harming the ingroup
• harming the outgroup (who you may benefit from trading with)
• harming powerless people who don't have the ability to trade or collaborate with you

And you basically need to tackle these in this order. If you live in a world where even people in your tribe backstab each other all the time, you won't have spare resources to spend on the outgroup or the powerless until your tribe has gotten it's basic shit together and figured out that lying/stealing/killing each other sucks.

If your tribe has it's basic shit together, then maybe you have the slack to ask the question: "hey, that outgroup over there, who we regularly raid and steal their sheep and stuff, maybe it'd be better if we traded with them instead of stealing their sheep?" and then begin to develop cosmopolitan norms.

If you eventually become a powerful empire (or similar), eventually you may notice that you're going around exploiting or conquering and... maybe you just don't actually want to do that anymore? Or maybe, within your empire, there's an underclass of people who are slaves or slave-like instead of being formally traded with. And maybe this is locally beneficial. But... you just don't want to do that anymore, because empathy or because you've come to believe in principles that say not to, or something. Sometimes this is because the powerless people would actually be more productive if they were free builders/traders, but sometimes it just seems like the right thing to do.

Avoiding harming the ingroup and productive outgroup are things that you're locally incentived to do because cooperation is very valuable. In an iterated strategy game, these are things you're incentived to do all the way along.

Avoiding harming the powerless is something that you are limited in your ability to do until the point where it starts making sense to cash in your victory points.

I think this is all pretty non-explicit in most discussions of morality/ethics/what-people-should-do, and conflation of "actions that are bad because it ruins ability to coordinate" and "actions that are bad because empathy and/or principles tell me they are" is common.

Periodically I describe a particular problem with the rationalsphere with the programmer metaphor of:

"For several years, CFAR took the main LW Sequences Git Repo and forked it into a private branch, then layered all sorts of new commits, ran with some assumptions, and tweaked around some of the legacy code a bit. This was all done in private organizations, or in-person conversation, or at best, on hard-to-follow-and-link-to-threads on Facebook.

"And now, there's a massive series of git-merge conflicts, as important concepts from CFAR attempt to get merged back into the original LessWrong branch. And people are going, like 'what the hell is focusing and circling?'"

And this points towards an important thing about _why_ think it's important to keep people actually writing down and publishing their longform thoughts (esp the people who are working in private organizations)

And I'm not sure how to actually really convey it properly _without_ the programming metaphor. (Or, I suppose I just could. Maybe if I simply remove the first sentence the description still works. But I feel like the first sentence does a lot of important work in communicating it clearly)

We have enough programmers that I can basically get away with it anyway, but it'd be nice to not have to rely on that.

There's a skill of "quickly operationalizing a prediction, about a question that is cruxy for your decisionmaking."

And, it's dramatically better to be very fluent at this skill, rather than "merely pretty okay at it."

Fluency means you can actually use it day-to-day to help with whatever work is important to you. Day-to-day usage means you can actually get calibrated re: predictions in whatever domains you care about. Calibration means that your intuitions will be good, and _you'll know they're good_.

Fluency means you can do it _while you're in the middle of your thought process_, and then return to your thought process, rather than awkwardly bolting it on at the end.

I find this useful at multiple levels-of-strategy. i.e. for big picture 6 month planning, as well as for "what do I do in the next hour."

I'm working on this as a full blogpost but figured I would start getting pieces of it out here for now.

A lot of this skill is building off on CFAR's "inner simulator" framing. Andrew Critch recently framed this to me as "using your System 2 (conscious, deliberate intelligence) to generate questions for your System 1 (fast intuition) to answer." (Whereas previously, he'd known System 1 was good at answering some types of questions, but he thought of it as responsible for both "asking" and "answering" those questions)

But, I feel like combining this with "quickly operationalize cruxy Fatebook predictions" makes it more of a power tool for me. (Also, now that I have this mindset, even when I can't be bothered to make a Fatebook prediction, I have a better overall handle on how to quickly query my intuitions)

I've been working on this skill for years and it only really clicked together last week. It required a bunch of interlocking pieces that all require separate fluency:

1. Having three different formats for Fatebook (the main website, the slack integration, and the chrome extension), so, pretty much wherever I'm thinking-in-text, I'll be able to quickly use it.

2. The skill of "generating lots of 'plans'", such that I always have at least two plausibly good ideas on what to do next.

3. Identifying an actual crux for what would make me switch to one of my backup plans.

4. Operationalizing an observation I could make that'd convince me of one of these cruxes.

I disagree with this particular theunitofcaring post "what would you do with 20 billion dollars?", and I think this is possibly the only area where I disagree with theunitofcaring overall philosophy and seemed worth mentioning. (This crops up occasionally in her other posts but it is most clear cut here).

I think if you got 20 billion dollars and didn't want to think too hard about what to do with it, donating to OpenPhilanthropy project is a pretty decent fallback option.

But my overall take on how to handle the EA funding landscape has changed a bit in the past few years. Some things that theunitofcaring doesn't mention here, which seem at least warrant thinking about:

[Each of these has a bit of a citation-needed, that I recall hearing or reading in reliable sounding places, but correct me if I'm wrong or out of date]

1) OpenPhil has (at least? I can't find more recent data) 8 billion dollars, and makes something like 500 million a year in investment returns. They are currently able to give 100 million away a year.

They're working on building more capacity so they can give more. But for the foreseeable future, they _can't_ actually spend more money than they are making.

2) OpenPhil doesn't want to be responsible for more than 50% of an orgs' budget, because being fully dependent on a single foundation creates a distorted relationship (where the org feels somewhat constrained by the foundation's own goals/whims/models).

If you have a diverse funding base, you can just do the thing you think is best. If you have a small funder base, if you aren't perfectly aligned with the funder, there is pressure to shift towards projects that they think are best (even if the funder is trying _not_ to put such pressure on you)

I'm not sure how big a concern this _should_ be, but AFAICT it's currently their policy.

This means there's a fair bit of value, if you had $20 billion, to setting up an alternative foundation to OpenPhil, just from the perspective of making sure the best orgs can _actually_ get fully funded.

3) OpenPhil has high standards for where to donate.

This is good.

But the fact that they have 8 billion, make another 500 million a year and spend down only around 100 million, means that the funding niche that actually needs filling right now is not more-of-OpenPhils-strategy.

There's a weird situation in the current landscape where it feels like money is unlimited... but there are still EA-relevant projects that need money. Typically ones that are younger, where data is more scarce.

Figuring out which of those actually deserve money is hard (esp. without creating weird incentivizes down the line, where _anyone_ with a half-baked project can show up and get your money). But this seems more like the domain where another major funder would be valuable.

...

Now, this is all hypothetical (theunitofcaring doesn't have 20 billion and neither do I). But this does point at an important shift on how to think about donating, if you're a small-to-medium sized donor.

Awhile ago I wrote "Earning to Give is Costly Signalling". Power laws mean that the richest people dwarf the donations of smaller time donors. Therefore, most of the value of EA donors is convincing rich people to give (and think) in an EA fashion.

Now I think it's weirder than that.

Given that EA collectively has access to billions of dollars (plus smaller-but-still-larger-than-we-know-what-to-do-with donor pools from a few other multi-millionaires)...

If you're a small-to-medium donor, roles that make sense to play are:

• Provide a costly signal for _new_ charities that aren't already on OpenPhil, BERI et al's radar.
• Help seed-fund new projects that you have a lot of local information on, that you think make a credible case for being high impact
• Donate to existing orgs, to help fill out the 50% funding gap (this is still partly about making sure they get funded, and also a sort of continued costly signal of the org's value to the larger funders). Many orgs also have tax-relevant status where it matters what proportion of their budget comes from private vs public donations, so making sure they have a diverse donor base is helpful.

This last option is basically EA business as usual, which is still important, but it's only one of several possible strategies that should be under consideration.

I also think it's important to consider using the money for your own personal development, or the development of people you know who you think could do well. Hire a personal trainer, or a tutor. Save up runway so that you can afford to take time off to think, and plan, or start a project.

A major goal I had for the LessWrong Review was to be "the intermediate metric that let me know if LW was accomplishing important things", which helped me steer.

I think it hasn't super succeeded at this.

I think one problem is that it just... feels like it generates stuff people liked reading, which is different from "stuff that turned out to be genuinely important."

I'm now wondering "what if I built a power-tool that is designed for a single user to decide which posts seem to have mattered the most (according to them), and, then, figure out which intermediate posts played into them." What would the lightweight version of that look like?

Another thing is, like, I want to see what particular other individuals thought mattered, as opposed to a generate aggregate that doesn't any theory underlying it. Making the voting public veers towards some kind of "what did the cool people think?" contest, so I feel anxious about that, but, I do think the info is just pretty useful. But like, what if the output of the review is a series of individual takes on what-mattered-and-why, collectively, rather than an aggregate vote?

Something struck me recently, as I watched Kubo, and Coco - two animated movies that both deal with death, and highlight music and storytelling as mechanisms by which we can preserve people after they die.

Kubo begins "Don't blink - if you blink for even an instant, if you a miss a single thing, our hero will perish." This is not because there is something "important" that happens quickly that you might miss. Maybe there is, but it's not the point. The point is that Kubo is telling a story about people. Those people are now dead. And insofar as those people are able to be kept alive, it is by preserving as much of their personhood as possible - by remembering as much as possible from their life.

This is generally how I think about death.

Cryonics is an attempt at the ultimate form of preserving someone's pattern forever, but in a world pre-cryonics, the best you can reasonably hope for is for people to preserve you so thoroughly in story that a young person from the next generation can hear the story, and palpably feel the underlying character, rich with inner life. Can see the person so clearly that he or she comes to live inside them.

Realistically, this means a person degrades with each generation. Their pattern is gradually distorted. Eventually it is forgotten.

Maybe this horrendously unsatisfying - it should be. Stories are not very high fidelity storage device. Most of what made the person an agent is gone.

But not necessarily - if you choose to not just remember humorous anecdotes about a person, but to remember what they cared about, you can be a channel by which that person continues to act upon the world. Someone recently pointed this out as a concrete reason to respect the wishes of the dead - as long as there are people enacting that person's will, there is some small way in which they meaningfully still exist.

This is part of how I chose to handle the Solstices that I lead myself: Little Echo, Origin of Stories, and Endless Lights are stories/songs touching on this theme. They don't work for everyone but they work for me. It's an unsatisfying concept but it's what we have.

This is what struck me:

I know no stories of my great great grandparents.

I do know stories of ancient generals and philosophers and artists and other famous people - people who lived such a captivating life that people wrote biographies about them.

I know stories about my grandmothers. I know stories about my great grandmothers. But one step beyond that... nothing. I never knew my great great grandparents, never had reason to ask about them. And I think it is probably too late - I think I could perhaps collect some stories of great-great-grandparents on my father's side. On my mothers... it's possible I could track them down but I doubt it.

And as things go, this isn't hugely upsetting to me. These are people I never met, and in all honesty it seems less pressing to preserve them than to cultivate the relationships I have in the near and now, and to save what lives I can who have yet to die in the first, physical fashion.

But, these are people who are dead forever. When fades at last the last lit sun, there will not be anyone to remember them.

And that's sad.

Yesterday I was at a "cultivating curiosity" workshop beta-test. One concept was "there are different mental postures you can adopt, that affect how easy it is not notice and cultivate curiosities."

It wasn't exactly the point of the workshop, but I ended up with several different "curiosity-postures", that were useful to try on while trying to lean into "curiosity" re: topics that I feel annoyed or frustrated or demoralized about.

The default stances I end up with when I Try To Do Curiosity On Purpose are something like:

1. Dutiful Curiosity (which is kinda fake, although capable of being dissociatedly autistic and noticing lots of details that exist and questions I could ask)

2. Performatively Friendly Curiosity (also kinda fake, but does shake me out of my default way of relating to things. In this, I imagine saying to whatever thing I'm bored/frustrated with "hullo!" and try to acknowledge it and and give it at least some chance of telling me things)

But some other stances to try on, that came up, were:

3. Curiosity like "a predator." "I wonder what that mouse is gonna do?"

4. Earnestly playful curiosity. "oh that [frustrating thing] is so neat, I wonder how it works! what's it gonna do next?"

5. Curiosity like "a lover". "What's it like to be that you? What do you want? How can I help us grow together?"

6. Curiosity like "a mother" or "father" (these feel slightly different to me, but each is treating [my relationship with a frustrating thing] like a small child who is bit scared, who I want to help, who I am generally more competent than but still want to respect the autonomy of."

7. Curiosity like "a competent but unemotional robot", who just algorithmically notices "okay what are all the object level things going on here, when I ignore my usual abstractions?"... and then "okay, what are some questions that seem notable?" and "what are my beliefs about how I can interact with this thing?" and "what can I learn about this thing that'd be useful for my goals?"

I started writing this a few weeks ago. By now I have other posts that make these points more cleanly in the works, and I'm in the process of thinking through some new thoughts that might revise bits of this.

But I think it's going to be awhile before I can articulate all that. So meanwhile, here's a quick summary of the overall thesis I'm building towards (with the "Rationalization" and "Sitting Bolt Upright in Alarm" [LW · GW] post, and other posts and conversations that have been in the works).

(By now I've had fairly extensive chats with Jessicata and Benquo and I don't expect this to add anything that I didn't discuss there, so this is more for other people who're interested in staying up to speed. I'm separately working on a summary of my current epistemic state after those chats)

• The rationalsphere isn't great at applying rationality to its own internal politics
• We don't seem to do much better than average. This seems like something that's at least pretty sad, even if it's a true brute fact about the world.
• There have been some efforts to fix this fact, but most of it has seemed (to me) to be missing key facts about game theory, common knowledge, theory of mind, and some other topics that I see as necessary to solve the problem.
• Billions of dollars are at stake, which creates important distortions that need addressing
• The rationality and EA communities are valuable, in large part, because there is an opportunity for important ideas to influence the world-stage, moving millions or billions of dollars (or causing millions of dollars worth of stuff happening). But, when billions of dollars are at stake, you start attract world-class opportunists trying to coopt you, (as well as community members start feeling pressure to conform to social reality on the world-stage), which demands world-class ability to handle subtle political pressures to preserve that value.
• [epistemic status: I'm not sure whether I endorse the rhetoric here. Maybe you don't need to be world class, but you probably need to be at least 75th percentile, and/or become more illegible to the forces that would try to coopt you]
• By default, we don't seem very good at allocating attention re: these issues. But, the attempts to address that I've seen seem doomy.
• One of the default failure modes that I've seen is, when people don't pay attention to a given call-for-clarity about "hey, we seem to be acting in ways that distort truth in predictable ways", is to jump all the way to statements like "EA has a lying problem," which I think is both untrue and anti-helpful for preserving a truthseeking space.
• In that case Sarah later wrote up a followup post that was more reasonable and Benquo wrote up a post that articulated the problem more clearly. [Can't find the links offhand]. But it was a giant red flag for me that getting people to pay attention required sensationalizing the problem. It seemed to me that this was following an incentive gradient identical to political news. This seemed roughly as bad for truthseeking as the original problem Sarah was trying to address was, both because:
• "lying" is a wrong characterization of what was happening (see here for my sense of why it's useful to distinguish lies from motivated cognition or other forms of deception [LW(p) · GW(p)])
• "clickbaitiness" is a bad process for allocating attention
• The "Rationalization/Sitting-bolt-upright" post was intended to provide an outlet for that sort of impulse that was at less counterproductive (in the interim before figuring out a more robust solution).
• A first guess at a "robust solution" is something like "develop clear norms for good, fair practices to critiquing organizations." If you meet those norms, posts on LessWrong that deal with local politics can get curated.
• By default, people use language for both truthseeking and for politics. It takes special effort to keep things truth-focused
• A primary lesson I learned from the sequences is that most people's beliefs and statements are not about truth at all. ("Science as attire [LW · GW]", "Fable of Science and Politics [LW · GW]", etc. Most of the places where the rationalsphere seems most truth-tracking are where it sidesteps this issue, rather than really solving it. Attempting to directly jump towards "well we just use words for truth, not politics", sound to me about as promising as writing the word 'cold' on a fridge [LW · GW].
• Relatedly, I think people struggle to stay in a truthseeking frame when they are feeling defensive. One person being defensive makes it 2-30x harder to remain truth-oriented. Multiple people being defensive at least add up that difficulty linearly, and potentially compound in weirder ways. I think this is challenging enough that it requires joint effort to avoid.
• A truthseeking space that can actually discuss politics sanely needs both individuals who are putting special effort to avoid being defensive, and conversation partners that practice avoiding unnecessarily* provoking defensiveness.
• *where by "unnecessary" I mean: "if your subject matter is inherently difficult to hear, you shouldn't avoid saying it. But you should avoid saying it with rhetoric that is especially liable to inflame the conversation. (i.e. "i think your project is net-harmful" is fine. "I think your project is stupid and can't believe you wasted our time on it" is making the conversation 20x harder, unnecessarily.)
• Yes, this is hard and doesn't come naturally to everyone. But I think it's at least approximately as hard as learning to avoid getting defensive is (and I would guess the low-hanging fruit is actually comparatively easy). I think if a truthseeking space doesn't ask people to at least pick up the low-hanging fruit here, it will be less effective as a truthseeking space.
• I don't think this is necessary for all conversations, but it's escalatingly important the less the participants trust each other and the higher the stakes.
• If conversation participants are not making efforts to pass each other's Ideological Turing Test, my default assumption is no progress will get made
• Communicating between frames/aesthetics/ontologies are very hard
• Common knowledge of 'Double Crux [LW · GW]' has made it somewhat easier to resolve gnarly disagreements, but I still frequently observe rationalists (myself included) just completely talking past each other, not noticing, and then either getting really frustrated, or assuming bad faith when the actual problem is significantly different world models.
• There's something of a skill to identifying what framework you are working under, which is somewhat separate from the usual doublecrux process.
• I also think there's a skill to figuring out how to backpropagate facts into your frame/taste/and-or/ontology, which I think helpful for resolving major frame disagreements. (But dramatically more helpful if both parties are doing it)
• Difficulty communicating between frames exacerbates the difficulty of discussing local politics sanely
• Different frames have quite different takes on which rocks and hard places [LW · GW] are more scary. By default, when the frames bump into each other, they see each other as trampling all over each other's obvious needs.
• Meanwhile, someone communicating in a different frame from you will seem to be missing the point, or subtly off, in a way that is hard to put your finger on, which makes the whole communicating process feel like moving through molasses.
• I think having more people with the taste/ontology doublecrux skill would enable more trust that conversations across frames are happening in good faith
• Counterfactual revolutions are super important. Real revolutions are mostly bad.
• Despite all the above, we're unlikely to reach a state where everyone can easily communicate across frames. Even if we did, it wouldn't guarantee that people actually were talking in good faith – sometimes people actually are actively-deceptive, or stuck in a harmful pattern that they can't escape from. This is particularly problematic when they're in power.
• I think we're in a rare, precious world where it's actually achievable for the major power centers in the EA space to communicate sanely to each other.
• So, it's simultaneously important to have a gameboard with rules that everyone respects, but simultaneously, it's important that there be a real threat of people kicking the gameboard over if the game seems rigged.

Conversation with Andrew Critch today, in light of a lot of the nonprofit legal work he's been involved with lately. I thought it was worth writing up:

"I've gained a lot of respect for the law in the last few years. Like, a lot of laws make a lot more sense than you'd think. I actually think looking into the IRS codes would actually be instructive in designing systems to align potentially unfriendly agents."

I said "Huh. How surprised are you by this? And curious if your brain was doing one particular pattern a few years ago that you can now see as wrong?"

"I think mostly the laws that were promoted to my attention were especially stupid, because that's what was worth telling outrage stories about. Also, in middle school I developed this general hatred for stupid rules that didn't make any sense and generalized this to 'people in power make stupid rules', or something. But, actually, maybe middle school teachers are just particularly bad at making rules. Most of the IRS tax code has seemed pretty reasonable to me."

Over in this thread, Said asked [LW(p) · GW(p)] the reasonable question "who exactly is the target audience with this Best of 2018 book?"

By compiling the list, we are saying: “here is the best work done on Less Wrong in [time period]”. But to whom are we saying this? To ourselves, so to speak? Is this for internal consumption—as a guideline for future work, collectively decided on, and meant to be considered as a standard or bar to meet, by us, and anyone who joins us in the future? 

Or, is this meant for external consumption—a way of saying to others, “see what we have accomplished, and be impressed”, and also “here are the fruits of our labors; take them and make use of them”? Or something else? Or some combination of the above?

I'm working on a post that goes into a bit more detail about the Review Phase, and, to be quite honest, the whole process is a bit in flux – I expect us (the LW team as well as site participants) to learn, over the course of the review process, what aspects of it are most valuable. 

But, a quick "best guess" answer for now.

I see the overall review process as having two "major phases":

• Phase 1: Nomination/Review/Voting/Post-that-summarizes-the-voting
• Phase 2: Compilation and Publication

I think the first phase should be oriented entirely around "internal consumption" – figuring out what epistemic standard to hold ourselves to, and how, so that we can do better in the future. (As well as figuring out what ideas we've developed that should be further built upon). Any other benefits are incidental.

The final book/sequence is at least somewhat externally facing. I do expect it to be some people's first introduction to LessWrong, and other people's "one thing they read from LW this year". And at least some consideration should be given to those people's reading experience (which will be lacking a lot of context). But my guess is that should come more in the form of context-setting editor commentary than in decisions about what to include.

I think “here are the fruits of our labors; take them and make use of them” is more of what I was aiming for. (Although "what standards are we internally holding ourselves to, and what work should we build towards?" is still an important function of the finished product). It'd be nice if people were impressed, but a better frame for that goal is "Outsiders looking in can get an accurate picture of how productive our community is, and what sort of things we do", and maybe they are impressed by that or maybe not. 

(I realize this comment doesn't really address the broader questions underlying the discussion, but wanted to at least get this out there. My preference is to spend the next few days actually attempting to write a variety of reviews, so that week I have more examples to point to when writing up a more comprehensive post about what sorts of reviews I think are most useful and why)

I've posted this on Facebook a couple times but seems perhaps worth mentioning once on LW: A couple weeks ago I registered the domain LessLong.com and redirected it to LessWrong.com/shortform. :P

A thing I might have maybe changed my mind about:

I used to think a primary job of a meetup/community organizer was to train their successor, and develop longterm sustainability of leadership.

I still hold out for that dream. But, it seems like a pattern is:

1) community organizer with passion and vision founds a community

2) they eventually move on, and pass it on to one successor who's pretty closely aligned and competent

3) then the First Successor has to move on to, and then... there isn't anyone obvious to take the reins, but if no one does the community dies, so some people reluctantly step up. and....

...then forever after it's a pale shadow of its original self.

For semi-branded communities (such as EA, or Rationality), this also means that if someone new with energy/vision shows up in the area, they'll see a meetup, they'll show up, they'll feel like the meetup isn't all that good, and then move on. Wherein they (maybe??) might have founded a new one that they got to shape the direction of more.

I think this also applies to non-community organizations (i.e. founder hands the reins to a new CEO who hands the reins to a new CEO who doesn't quite know what to do)

So... I'm kinda wondering if second-generation successors should just... err on the side of shutting the thing down when they leave, rather than trying desperately to find a replacement.

The answer isn't obvious. There is value that continues to be created by the third+ generation. I think I've mostly gone from "having a firm opinion that you should be proactively training your successor" to "man, I dunno, finding a suitable successor is actually pretty hard, mrrr?"

Posts I vaguely want to have been written so I can link them to certain types of new users:

• "Why you can chill out about the basilisk and acausal blackmail." (The current Roko's Basilisk [? · GW] kinda tries to be this, but there's a type of person who shows up on LessWrong regularly who's caught in an anxious loop that keeps generating more concerns, and I think the ideal article here is more trying to break them out of the anxious loop than comprehensively explain the game theory.)
• "FAQ: Why you can chill out about quantum immortality and everything adds up to normality." (Similar, except the sort of person who gets worked up about this is usually having a depressive spiral and worried about being trapped in an infinite hellscape)

Crossposted from my Facebook timeline (and, in turn, crossposted there from vaguely secret, dank corners of the rationalsphere)

“So Ray, is LessLong ready to completely replace Facebook? Can I start posting my cat pictures and political rants there?”

Well, um, hmm....

So here’s the deal. I do hope someday someone builds an actual pure social platform that’s just actually good, that’s not out-to-get you, with reasonably good discourse. I even think the LessWrong architecture might be good for that (and if a team wanted to fork the codebase, they’d be welcome to try)

But LessWrong shortform *is* trying to do a bit of a more nuanced thing than that.

Shortform is for writing up early stage ideas, brainstorming, or just writing stuff where you aren’t quite sure how good it is or how much attention to claim for it.

For it to succeed there, it’s really important that it be a place where people don’t have to self-censor or stress about how their writing comes across. I think intellectual progress depends on earnest curiosity, exploring ideas, sometimes down dead ends.

I even think it involves clever jokes sometimes.

But... I dunno, if looked ahead 5 years and saw that the Future People were using shortform mostly for cat videos and political rants, I’d have a sense that we fucked up somehow.

Hopefully the karma system will naturally reward shortform that’s thought provoking and interesting. If we get deluged in politics and cats and worse things I can’t predict, we might add some features that affect what shortform content gets most highly promoted.

But the bottom line is the whole thing is an experiment. The important bits to know for now are:

1) use some good judgment

2) don’t stress too much about it – if it turns out to be a problem we’ll most likely try to solve it via positive-selection tech

3) Shortform is one piece of the overall intellectual pipeline. We’re also looking into things we can also do to improve LessWrong’s reward for the upper end of the rigor spectrum

Just spent a weekend at the Internet Intellectual Infrastructure Retreat. One thing I came away with was a slightly better sense of was forecasting and prediction markets, and how they might be expected to unfold as an institution.

I initially had a sense that forecasting, and predictions in particular, was sort of "looking at the easy to measure/think about stuff, which isn't necessarily the stuff that connected to stuff that matters most."

Tournaments over Prediction Markets

Prediction markets are often illegal or sketchily legal. But prediction tournaments are not, so this is how most forecasting is done.

The Good Judgment Project

Held an open tournament, the winners of which became "Superforecasters". Those people now... I think basically work as professional forecasters, who rent out their services to companies, NGOs and governments that have a concrete use for knowing how likely a given country is to go to war, or something. (I think they'd been hired sometimes by Open Phil?)

Vague impression that they mostly focus on geopolitics stuff?

High Volume and Metaforecasting

Ozzie described a vision where lots of forecasters are predicting things all the time, which establishes how calibrated they are. This lets you do things like "have one good forecaster with a good track record make lots of predictions. Have another meta-forecaster evaluate a small sample of their predictions to sanity check that they are actually making good predictions", which could get you a lot of predictive power for less work than you'd expect."

This seemed interesting, but I still had some sense of "But how do you get all these people making all these predictions? The prediction markets I've seen don't seem to accomplish very interesting things, for reasons Zvi discussed here." Plus I'd heard that sites like Metaculus end up mostly being about gaming the operationalization rules than actually predicting things accurately.

Automation

One thing I hadn't considered is that Machine Learning is already something like high volume forecasting, in very narrow domains (i.e. lots of bots predicting which video you'll click on next). One of Ozzie's expectations is that over time, as ML improves, it'll expand the range of things that bots can predict. So some of the high volume can come from automated forecasters.

Neural nets and the like might also be able to assist in handling the tricky "operationalization bits", where you take a vague prediction like "will country X go to war against country Y" and turn that into the concrete observations that would count for such a thing. Currently this takes a fair amount of overhead on Metaculus. But maybe at some point this could get partly automated.

(there wasn't a clear case for how this would happen AFAICT, just 'i dunno neural net magic might be able to help.' I don't expect neural-net magic to help here in the next 10 years but I could see it helping in the next 20 or 30. I'm not sure if it happens much farther in advance than "actual AGI" though)

I [think] part of the claim was that for both the automated-forecasting and automated-operationalization, it's worth laying out tools, infrastructure and/or experiments now that'll set up our ability to take advantage of them later.

Sweeping Visions vs Near-Term Practicality, and Overly Narrow Ontologies

An aesthetic disagreement I had with Ozzie was:

My impression is that Ozzie is starting with lots of excitement for forecasting as a whole, and imagining entire ecosystems built out of it. And... I think there's something important and good about people being deeply excited for things, exploring them thoroughly, and then bringing the best bits of their exploration back to the "rest of the world."

But when I look at the current forecasting ecosystem, it looks like the best bits of it aren't built out of sweeping infrastructural changes, they're built of small internal teams building tools that work for them, or consulting firms of professionals that hire themselves out. (Good Judgment project being one, and the How To Measure Anything guy being another)

The problem with large infrastructural ecosystems is this general problem you also find on Debate-Mapping sites – humans don't actually think in clean boxes that are easy to fit into database tables. They think in confused thought patterns that often need to meander, explore special cases, and don't necessarily fit whatever tool you built for them to think in.

Relatedly: every large company I've worked at has built internal tools of some sort, even for domains that seem like they sure out to be able to be automated and sold at scale. Whenever I've seen someone try to purchase enterprise software for managing a product map, it's either been a mistake, or the enterprise software has required a lot of customization before it fit the idiosyncratic needs of the company.

Google sheets is really hard to beat as a coordination tool (but a given google sheet is hard to scale)

So for the immediate future I'm more excited by hiring forecasters and building internal forecasting teams than ecosystem-type websites.

More in neat/scary things Ray noticed about himself.

I set aside this week to learn about Machine Learning, because it seemed like an important thing to understand. One thing I knew, going in, is that I had a self-image as a "non technical person." (Or at least, non-technical relative to rationality-folk). I'm the community/ritual guy, who happens to have specialized in web development as my day job but that's something I did out of necessity rather than a deep love.

So part of the point of this week was to "get over myself, and start being the sort of person who can learn technical things in domains I'm not already familiar with."

And that went pretty fine.

As it turned out, after talking to some folk I ended up deciding that re-learning Calculus was the right thing to do this week. I'd learned in college, but not in a way that connected to anything and gave me a sense of it's usefulness.

And it turned out I had a separate image of myself as a "person who doesn't know Calculus", in addition to "not a technical person". This was fairly easy to overcome since I had already given myself a bunch of space to explore and change this week, and I'd spent the past few months transitioning into being ready for it. But if this had been at an earlier stage of my life and if I hadn't carved out a week for it, it would have been harder to overcome.

Man. Identities. Keep that shit small yo.

High Stakes Value and the Epistemic Commons

I've had this in my drafts for a year. I don't feel like the current version of it is saying something either novel or crisp enough to quite make sense as a top-level post, but wanted to get it out at least as a shortform for now.

There's a really tough situation I think about a lot, from my perspective as a LessWrong moderator. These are my personal thoughts on it.

The problem, in short: 

Sometimes a problem is epistemically confusing, and there are probably political ramifications of it, such that the most qualified people to debate it are also in conflict with billions of dollars on the line and the situation is really high stakes (i.e. the extinction of humanity) such that it really matters we get the question right.

Political conflict + epistemic murkiness means that it's not clear what "thinking and communicating sanely" about the problem look like, and people have (possibly legitimate) reasons to be suspicious of each other's reasoning.

High Stakes means that we can't ignore the problem.

I don't feel like our current level of rationalist discourse patterns are sufficient for this combo of high stakes, political conflict, and epistemic murkiness. 

Spelling out some concrete examples

Interventions that help with AI extinction risk are often hard to evaluate. Reasonable people can disagree whether a project ranges from "highly net positive" to "highly net negative". Smart people I know have fairly different strategic perspectives on how humanity can survive the 21st century.

Sometimes these disagreements are political – is pivotal acts [LW · GW] a helpful frame or a harmful one [LW · GW]? How suspicious should we be of safetywashing [LW · GW]? 

Sometimes these disagreements are more technical. How will differential technology play out? I've heard some arguments that improving alignment techniques on current-generation ML systems may be negative, because a) it won't actually help align powerful AI systems past the sharp left turn, and meanwhile b) makes it easier and more profitable to deploy AI in ways that could start to escalate beyond our control (killing us in slightly more [LW · GW] mundane ways than the fast takeoff scenarios).

I've heard arguments that even interpretability, which you'd think is a purely positive source of information, is also helpful for capabilities (in particular if the interpretability is actually any good). And maybe you actually need a lot of interpretability before the alignment benefits outweigh the capability gains.

Some disagreements are the intersection of political, technical, and psychological. 

You might argue that people in AGI companies are motivated by excitement over AI, or making money, and are only paying lip service to safety. Your beliefs about this might include "their technical agenda doesn't make any sense to you" as well as "you have a strong guess about what else might be motivating their agenda." [LW(p) · GW(p)] 

You might think AI Risk advocates are advocating pivotal acts because of a mix of trauma [LW · GW], finding politics distasteful [LW · GW], and technical mistakes regarding the intersection of boundaries and game theory [LW · GW].

This is all pretty gnarly, because 

• The stakes here matter a lot. We're talking about the end of the world and/or the cosmic endowment.
• Common professional politeness norms typically paper over conflict rather than leaning into it. There aren't (as much) consensus "professional" norms for dealing directly with high stakes conflict that preserve epistemic.
• A lot of conversation has been going on for a long time, but not everyone's participated in the same arguments, so some people feel like "We've had the object-level arguments and I don't really know where to go other than to say 'it sure looks to me like you're psychologically motivated here'", and others are like "why are you jumping to assumptions about me when AFAICT we haven't even hashed out the object level?"
• People disagree about what counts as good concrete technical arguments, and I think at least some (thought IMO not all) of that disagreement is for fairly reasonable reasons.

I have some guesses about how to think about this, but I feel some confusion about them. And it feels pedagogically bad for this to end with "and therefore, [Insert some specific policy or idea]" rather than "okay, what is the space of considerations and desiderata here?" a la Hold Off On Proposing Solutions [LW · GW].

Seems like different AI alignment perspectives sometimes are about "which thing seems least impossible."

Straw MIRI researchers: "building AGI out of modern machine learning is automatically too messy and doomed. Much less impossible to try to build a robust theory of agency first."

Straw Paul Christiano: "trying to get a robust theory of agency that matters in time is doomed, timelines are too short. Much less impossible to try to build AGI that listens reasonably to me out of current-gen stuff."

(Not sure if either of these are fair, or if other camps fit this)

Using "cruxiness" instead of operationalization for predictions.

One problem with making predictions is "operationalization." A simple-seeming prediction can have endless edge cases.

For personal predictions, I often think it's basically not worth worrying about it. Write something rough down, and then say "I know what I meant." But, sometimes this is actually unclear, and you may be tempted to interpret a prediction in a favorable light. And at the very least it's a bit unsatisfying for people who just aren't actually sure what they meant.

One advantage of cruxy predictions (aside from "they're actually particularly useful in the first place), is that if you know what decision a prediction was a crux for, you can judge ambiguous resolution based on "would this actually have changed my mind about the decision?"

("Cruxiness instead of operationalization" is a bit overly click-baity. Realistically, you need at least some operationalization, to clarify for yourself what a prediction even means in the first place. But, I think maybe you can get away with more marginal fuzziness if you're clear on how the prediction was supposed to inform your decisionmaking)

⚖ A year from now, in the three months prior, will I have used "cruxiness-as-operationalization" on a prediction, and found it helpful. (Raymond Arnold: 50%)

My personal religion involves two* gods – the god of humanity (who I sometimes call "Humo") and the god of the robot utilitarians (who I sometimes call "Robutil"). 

When I'm facing a moral crisis, I query my shoulder-Humo and my shoulder-Robutil for their thoughts. Sometimes they say the same thing, and there's no real crisis. For example, some naive young EAs try to be utility monks, donate all their money, never take breaks, only do productive things... but Robutil and Humo both agree that quality intellectual world requires slack and psychological health. (Both to handle crises [LW · GW] and to notice subtle things [LW · GW], which you might need, even in emergencies [LW · GW])

If you're an aspiring effective altruist, you should definitely at least be doing all the things that Humo and Robutil agree on. (i.e. get to to the middle point of Tyler Alterman's story here [EA · GW]).

But Humo and Robutil in fact disagree on some things, and disagree on emphasis. 

They disagree on how much effort you should spend to avoid accidentally recruiting people you don't have much use for [EA · GW].

They disagree on how many high schoolers it's acceptable to accidentally fuck up psychologically, while you experiment with a new program to get them into.

They disagree on how hard to push yourself to grow better/stronger/wiser/faster, and how much you should sacrifice to do so.

Humo and Robutil each struggle to understand things differently. Robutil eventually acknowledges you need Slack, but it didn't occur to him initially. His understanding was born in the burnout and tunnel-vision of thousands of young idealists, and Humo eventually (patiently, kindly) saying "I told you so." (Robutil responds "but you didn't provide any arguments about how that maximized utility!". Humo responds "but I said it was obviously unhealthy!" Robutil says "wtf does 'unhealthy' even mean?")

It took Robutil longer still to consider that perhaps you not only need to prioritize your own wellbeing and your friendships, but you need to prioritize them for their own sake, not just as part of a utilitarian calculus

Humo struggles to acknowledge that if you spend all your time making sure to uphold deontological commitments to avoid harming the people in your care, then this is in fact measured in real human beings who suffer and die because you took longer to scale up your program. 

In my headcanon, Humo and Robutil are gods who are old and wise, and they got over their naive struggles long ago. They respect each other as brothers. They understand that each of their perspectives is relevant to the overall project of human flourishing. They don't disagree as much as you'd naively expect, but they speak different languages and emphasize things differently. 

Humo might acknowledge that I can't take care of everyone, or even respond compassionately to all the people who show up in my life I don't have time to help. But he says so with a warm, mournful [LW · GW] compassion, whereas Robutil says in with brief, efficient ruthlessness [LW(p) · GW(p)].

I find it useful to query them independently, and to imagine the wise version of each of them as best I can – even if my imagining is but a crude shadow of their idealized platonic selves.

I’ve noticed myself using “I’m curious” as a softening phrase without actually feeling “curious”. In the past 2 weeks I’ve been trying to purge that from my vocabulary. It often feels like I'm cheating, trying to pretend like I'm being a friend when actually I'm trying to get someone to do something. (Usually this is a person I'm working with it and it's not quite adversarial, we're on the same team, but it feels like it degrades the signal of true open curiosity)

Hmm, sure seems like we should deploy "tagging" right about now, mostly so you at least have the option of the frontpage not being All Coronavirus All The Time.

So there was a drought of content during Christmas break, and now... abruptly... I actually feel like there's too much content on LW. I find myself skimming down past the "new posts" section because it's hard to tell what's good and what's not and it's a bit of an investment to click and find out.

Instead I just read the comments, to find out where interesting discussion is.

Now, part of that is because the front page makes it easier to read comments than posts. And that's fixable. But I think, ultimately, the deeper issue is with the main unit-of-contribution being The Essay.

A few months ago, mr-hire said (on writing that provokes comments)

Ideas should become comments, comments should become conversations, conversations should become blog posts, blog posts should become books. Test your ideas at every stage to make sure you're writing something that will have an impact.

This seems basically right to me.

In addition to comments working as an early proving ground for an ideas' merit, comments make it easier to focus on the idea, instead of getting wrapped up in writing something Good™.

I notice essays on the front page starting with flowery words and generally trying to justify themselves as an essay, when all they actually needed was to be a couple short paragraphs. Sometimes even a sentence.

So I think it might be better if the default way of contributing to LW was via comments (maybe using something shaped sort of like this feed), which then appears on the front page, and if you end up writing a comment that's basically an essay, then you can turn it into an essay later if you want.

Is... there compelling difference between stockholm syndrome and just, like, being born into a family?

I notice that academic papers have stupidly long, hard-to-read abstracts. My understanding is that this is because there is some kind of norm about papers having the abstract be one paragraph, while the word-count limit tends to be... much longer than a paragraph (250 - 500 words).

Can... can we just fix this? Can we either say "your abstract needs to be a goddamn paragraph, which is like 100 words", or "the abstract is a cover letter that should be about one page long, and it can have multiple linebreaks and it's fine."

(My guess is that the best equilibrium is "People keep doing the thing currently-called-abstracts, and start treating them as 'has to fit on one page', with paragraph breaks, and then also people start writing a 2-3 sentence thing that's more like "the single actual-paragraph that you'd read if you were skimming through a list of papers.")

I had a very useful conversation with someone about how and why I am rambly. (I rambled a lot in the conversation!).

Disclaimer: I am not making much effort to not ramble in this post.

A couple takeaways:

1. Working Memory Limits

One key problem is that I introduce so many points, subpoints, and subthreads, that I overwhelm people's working memory (where human working memory limits is roughly "4-7 chunks").

It's sort of embarrassing that I didn't concretely think about this before, because I've spent the past year SPECIFICALLY thinking about working memory limits, and how they are the key bottleneck on intellectual progress.

So, one new habit I have is "whenever I've introduced more than 6 points to keep track of, stop and and figure out how to condense the working tree of points down to <4.

(Ideally, I also keep track of this in advance and word things more simply, or give better signposting for what overall point I'm going to make, or why I'm talking about the things I'm talking about)

...

2. I just don't finish sente

I frequently don't finish sentences, whether in person voice or in text (like emails). I've known this for awhile, although I kinda forgot recently. I switch abruptly to a new sentence when I realize the current sentence isn't going to accomplish the thing I want, and I have a Much Shinier Sentence Over Here that seems much more promising.

But, people don't understand why I'm making the leap from one half-finished thought to another.

So, another simple habit is "make sure to finish my god damn sentences, even if I become disappointed in them halfway through"

...

3. Use Mindful Cognition Tuning to train on *what is easy for people to follow*, as well as to improve the creativity/usefulness of my thoughts.

I've always been rambly. But a thing that I think has made me EVEN MORE rambly in the past 2 years is a mindful-thinking-technique, where you notice all of your thoughts on the less-than-a-second level, so that you can notice which thought patterns are useful or anti-useful.

This has been really powerful for improving my thought-quality. I'm fairly confident that I've become a better programmer and better thinker because of it.

But, it introduces even more meta-thoughts for me to notice while I'm articulating a sentence, which distract me from the sentence itself.

What I realized last weekend was: I can use Mindful Cognition to notice what types of thoughts/sentences are useful for *other people's comprehension of me*, not just how useful m original thought processes are.

The whole point of the technique is to improve your feedback loop (both speed and awareness), which makes it easier to deliberate practice. I think if I just apply that towards Being More Comprehensible, it'll change from being a liability in rambliness to an asset.

[not trying to be be comprehensible people that don't already have some conception of Kegan stuff. I acknowledge that I don't currently have a good link that justifies Kegan stuff within the LW paradigm very well]

Last year someone claimed to me is that a problem with Kegan is that there really are at least 6 levels. The fact that people keep finding themselves self-declaring as "4.5" should be a clue that 4.5 is really a distinct level. (the fact that there are at least two common ways to be 4.5 also is a clue that the paradigm needs clarification)

My garbled summary of this person's conception is:

• Level 4: (you have a system of principles you are subject to, that lets you take level 3 [social reality??] as object)
• Level 5: Dialectic. You have the ability to earnestly dialogue between a small number of systems (usually 2 at a time), and either step between them, or work out new systems that reconcile elements from the two of them.
• Level 6: The thing Kegan originally meant by "level 5" – able to fluidly take different systems as object.

Previously, I had felt something like "I basically understand level 5 fine AFAICT, but maybe don't have the skills do so fluidly. I can imagine there being some special sauce that I don't currently have but it doesn't feel very mysterious. It's not obvious what would be different about me that made me level 5."

Once I heard this description, I thought "oh, yeah this new 4.5/5 makes a lot of sense, and clearly describes where I'm currently at. Like, the Noticing Frames subsequence (and past year of my life) was basically me gaining the skill of doing Dialectic reasonably well. And from that vantage point, it makes more sense that there's a step beyond that that's something like "the process of noticing frames/systems and dialoguing between them eventually becomes a simpler object in my suite of available actions, such that I can do it seamlessly."

After a recent 'doublecrux meetup' (I wasn't running it but observed a bit), I was reflecting on why it's hard to get people to sufficiently disagree on things in order to properly practice doublecrux.\

As mentioned recently, it's hard to really learn doublecrux unless you're actually building a product that has stakes. If you just sorta disagree with someone... I dunno you can do the doublecrux loop but there's a sense where it just obviously doesn't matter.

But, it still sure is handy to have practiced doublecruxing before needing to do it in an important situation. What to do?

Two options that occur to me are

• Singlecruxing
• First try to develop a plan for building an actual product together, THEN find a thing to disagree about organically through that process.

[note: I haven't actually talked much with the people who's major focus is teaching doublecrux, not sure how much of this is old hat, or if there's a totally different approach that sort of invalidates it]

SingleCruxing

One challenge about doublecrux practice is that you have to find something you have strong opinions about and also someone else has strong opinions about. So... just sidestep that problem but only worrying about something that you have strong opinions about.

Pick a belief that is actually relevant to your plans (such as where you're planning to go to college, or what kind of career to go into, or ideally a project you're actually working on that you're excited about.

What beliefs are you confident in, that are underpinning your entire approach? (i.e. "going to college in the first place is the right move" or "A job in this industry will make me happier than this other industry" or "this project is a good idea because people will buy the product I'm building.")

Instead of practicing discussing this with someone else, you can just ask yourself, with no one else around you, why you believe what you believe, and what would change your mind about it.

Having considered this, I think I like it a lot as an "doublecruxing 101" skill.

One problem with learning doublecrux is that doing it properly takes awhile, and in my experience starts with a phase that's more about model-sharing, before moving to the "actually find your own cruxes and figure out what would change your mind." But, the first part isn't actually all that different from regular debate, or discussion. And it's not quite clear when to transition to the second part (or, it naturally interweaves with the first part. See postformal doublecrux).

This makes it hard to notice and train the specific skills that are unique to doublecrux.

I like the notion that "first you learn singlecrux, then doublecrux" because a) it's just generally a useful skill to ask why you actually believe the things you do and what would change your mind, and b) I think it's much easier to focus on the active, unique ingredients when the topic isn't getting blurred with various other conversational skills, and/or struggling to find a thing that's worth disagreeing about in the first place.

It'd also have the advantage that you can think about something that's actually slightly triggering/uncomfortable for you to consider (which I think is pretty valuable for actually learning to do the skill "for real"), but where you only have to worry about how you feel about it, rather than also have to figure out how to relate to someone else who might also feel strongly.

I think this'd be particular good for local meetups that don't have the benefit of instructors with a lot of practice helping people learn the doublecrux skill.

(I might still have people pair up, but only after thinking privately about it for 5 minutes, and the pairs of people would not be disagreeing with each other, just articulating their thought processes to each other. At any given time, you'd have one"active participant" talking through why they believe what they believe and how they might realistically change their mind about it, and another partner acting more as a facilitator to notice if they're stuck in weird thought pattern loops or something)

Finding a product you both might actually want to build, then disagreeing about it

But, still, sooner or later you might want to practice doublecruxing. What then? How do you reliably find things you disagree about?

The usual method I've observed is have people write down statements that they believe in confidently, that they think other people might disagree with, and then pair up based on shared disagreement. This varies in how well it produces disagreements that feel really 'alive' and meaningful.

But if doublecrux actually is mostly for building products, a possible solution might be instead to pair up based on shared interests in the sorts of projects you might want to build. You (probably?) won't actually build the product, but it seems important that you be able to talk about it as realistically as possible.

Then, you start forming a plan about how to go about building it, while looking for points of disagreement. Lean into that disagreement when you notice it, and explore the surrounding concept-space.

At last night's meetup, I paired with someone and suggested this idea to them. We ended up with the (somewhat meta) actual shared product of "how to improve the Berkeley rationality community." We discussed that for 20 minutes, and eventually found disagreement about whether communities require costly signals of membership to be any good, or whether they could instead be built off other human psychological quirks.

This was not a disagreement I think we would have come up with if we "listed a bunch of things we felt strongly about." And it felt a lot more real.

(I do think there's a risk of most pairs of peoples ending up being "the local community" or something similarly meta as their 'product', but the actual disagreements I expect to still be fairly unique and dependent on the people in question)

I’d like to hire cognitive assistants and tutors more often. This could (potentially) be you, or people you know. Please let me know if you’re interested or have recommendations.

By “cognitive assistant” I mean a range of things, but the core thing is “sit next to me, and notice when I seem like I’m not doing the optimal thing, and check in with me.” I’m interested in advanced versions who have particular skills (like coding, or Applied Quantitivity, or good writing, or research taste) who can also be tutoring me as we go.

I’d like a large rolodex of such people, both for me, and other people I know who could use help. Let me know if you’re interested.

I was originally thinking "people who live in Berkeley" but upon reflection this could maybe be a remote role.

I notice some people go around tagging posts with every plausible tag that possible seems like it could fit. I don't think this is a good practice – it results in an extremely overwhelming and cluttered tag-list, which you can't quickly skim to figure out "what is this post actually about"?, and I roll to disbelieve on "stretch-tagging" actually helping people who are searching tag pages.

I just briefly thought you could put a bunch of AI researchers on a spaceship, and accelerate it real fast, and then they get time dilation effects that increase their effective rate of research.

Then I remembered that time dilation works the other way 'round – they'd get even less time.

This suggested a much less promising plan of "build narrowly aligned STEM AI, have it figure out how to efficiently accelerate the Earth real fast and... leave behind a teeny moon base of AI researchers who figure out the alignment problem."

Man, I watched The Fox and The Hound a few weeks ago. I cried a bit.

While watching the movie, a friend commented "so... they know that foxes are *also* predators, right?" and, yes. They do. This is not a movie that was supposed to be about predation except it didn't notice all the ramifications about its lesson. This movie just isn't taking a stand about predation.

This is a movie about... kinda classic de-facto tribal morality. Where you have your family and your tribe and a few specific neighbors/travelers that you welcomed into your home. Those are your people, and the rest of the world... it's not exactly that they aren't *people*, but, they aren't in your circle of concern. Maybe you eat them sometimes. That's life.

Copper the hound dog's ingroup isn't even very nice to him. His owner, Amos, leaves him out in a crate on a rope. His older dog friend is sort of mean. Amos takes him out on a hunting trip and teaches him how to hunt, conveying his role in life. Copper enthusiastically learns. He's a dog. He's bred to love his owner and be part of the pack no matter what.

My dad once commented that this was a movie that... seemed remarkably realistic about what you can expect from animals. Unlike a lot of other disney movies it didn't require suspending disbelief much.

A baby fox and hound might totally play together because they haven't figured out yet that they're supposed to be enemies.

If that hound then went away for 6 months to learn to hunt, and came back, it might initially be hesitant to hunt down its fox friend, out of a vague/confused memory.

But interspecies friendship isn't *that* strong, and doing-what-your-species/tribe does is often stronger, and yeah later on the hound is like "okay I guess we're hunting this fox now. It's what master wants. I do what master says, that's who I am."

And then...

...well, and then the hound gets attacked by a bear. And the fox comes back to save him. And on one hand, I bet 99+% of foxes would not do that. But, having seen a bunch of youtubes of animals doing impressive things, I'm willing to buy that the occasional hero foxes exist, and I'm willing to buy a fox who remembers enough of his interspecies friend to intervene bravely.

(weakly held, if someone who knew a lot more about foxes than me was like "nope, this is outside the space of what foxes do", I'd believe them)

And what gets me is... man, this is what my morality is built out of. People who were mostly doing what nature of society incentived, who are still on the verge of eating their friends, but there's little sparks of friendship/compassion/abstract-reasoning that build up along the way, combined with the abundance necessary for them to grow. And that's why my morality comes from. Man.

Sometimes the subject of Kegan Levels comes up and it actually matters a) that a developmental framework called "kegan levels" exists and is meaningful, b) that it applies somehow to The Situation You're In.

But, almost always when it comes up in my circles, the thing under discussion is something like "does a person have the ability to take their systems as object, move between frames, etc." And AFAICT this doesn't really need to invoke developmental frameworks at all. You can just ask if a person has a the "move between frames" skill.*

This still suffers a bit from the problem where, if you're having an argument with someone, and you think the problem is that they're lacking a cognitive skill, it's a dicey social move to say "hey, your problem is that you lack a cognitive skill." But, this seems a lot easier to navigate than "you are a Level 4 Person in this 5 Level Scale".

(I have some vague sense that Kegan 5 is supposed to mean something more than "take systems as object", but no one has made a great case for this yet, and in case it hasn't been the thing I'm personally running into)

There's a problem at parties where there'll be a good, high-context conversation happening, and then one-too-many-people join, and then the conversation suddenly dies.

Sometimes this is fine, but other times it's quite sad.

Things I think might help:

• If you're an existing conversation participant:
  • Actively try to keep the conversation small. The upper limit is 5, 3-4 is better. If someone looks like they want to join, smile warmly and say "hey, sorry we're kinda in a high context conversation right now. Listening is fine but probably don't join."
  • If you do want to let a newcomer join in, don't try to get them up to speed (I don't know if I've ever seen that actually work). Instead, say "this is high context so we're not gonna repeat the earlier bits, maybe wait to join in until you've listened enough to understand the overall context", and then quickly get back to the conversation before you lose the Flow.
• If you want to join a conversation:
  • If there are already 5 people, sorry, it's probably too late. Listen if you find it interesting, but if you actively join you'll probably just kill the conversation.
  • Give them the opportunity to gracefully keep the conversation small if they choose. (say something like "hey can I join? It sounds like maybe a high context conversation, no worries if you wanna keep it small.")
  • Listen for longer before joining. Don't just wait till you understand the current topic – try to understand the overall vibe, and what previous topics might be informing the current one. Try to get a sense of what each current participant is getting out the conversation. When you do join, do so in a small way that gives them affordance to shift back to an earlier topic if your new contribution turned out to be not-actually-on-topic.

I'm not sure why it took me so long to realize that I should add a "consciously reflect on why I didn't succeed at all my habits yesterday, and make sure I don't fail tomorrow" to my list of daily habits, but geez it seems obvious in retrospect.

Strategic use of Group Houses for Community Building

(Notes that might one day become a blogpost. Building off The Relationship Between the Village and the Mission [LW · GW]. Inspired to go ahead and post this now because of John Maxwell's "how to make money reducing loneliness [LW · GW]" post, which explores some related issues through a more capitalist lens)

• A good village needs fences:
• A good village requires doing things on purpose. 
• Doing things on purpose requires that you have people who are coordinated in some way
• Being coordinated requires you to be able to have a critical mass of people who are actually trying to do effortful things together (such as maintain norms, build a culture, etc)
• If you don't have a fence that lets some people in and doesn't let in others, and which you can ask people to leave, then your culture will be some random mishmash that you can't control
• There are a few existing sets of fences. 
• The strongest fences are group houses, and organizations. Group houses are probably the easiest and most accessible resource for the "village" to turn into a stronger culture and coordination point. 
• Some things you might coordinate using group houses for:
• Strengthening friendships
• Select people who actually have a decent chance of wanting to be good friends
• Don't stress overmuch about getting the perfect set of people – overly stressing about finding the 'best' people to be friends with is one of the pathologies in the Bay area that make friendship harder. If everyone's doing it, no one has the ability to let a friendship actually grow, which takes time.
• DO find people you enjoy hanging out with, talking to, and share some interests with
• It may take multiple years to find a group house where everyone gets along with everyone. I think it makes sense, earlier on, to focus on exploring (i.e. if you've just moved to the Bay, don't worry about getting a group house culture that is a perfect fit), but within 3 years I think it's achievable for most people to have found a group house that is good for friendship.
• Once you've got a group house that seems like a good longterm home, actually invest it. 
• Do things with your roommates. 
• Allocate time, not just for solving logistical problems, but for getting on the same page emotionally
• "Deep friendships often come from striving and growing together." Look for opportunities for shared activities that are active rather than passive and involve growing skills that you are excited about.
• But, probably don't try to force this. Sometimes you're at the same stage in a life trajectory as someone else, and you're growing in the same way at the same time. But not always. And later on you may want to keep growing in a direction where someone else feels that they've solved their bottleneck and growing more in that direction isn't that relevant to them anymore. That's okay.
• Having a nicer place to live
• I think this is an important "lower Maslow hierarchy" level than the strong friendships one. If your house isn't a nice place to live, you'll probably have a harder time forming friendships with people there. 
• "Nice place to live" means different things to different people. Form a group house with people who have similar desires re: cleanliness and approaches to problem solving and aesthetics, etc.
• Deliberately cultivating your incentives
• What sort of environment you're in shapes what sort of ways you grow. You might care about about this for reasons other than incidentally helping deepen friendships. 
• This depends both on having people who want to cultivate the same sorts of incentives that you do, and on actually coordinating with each other to hold each other to those incentives
• Be wary of your, and other's, desire to have the self-image as someone who wants to grow in a particular way. I've seen a failure mode where people felt vaguely obligated to pay lip service to certain kinds of growth but it wasn't actually what they wanted
• Be wary of "generic emphasis on growth". A thing I've seen a few group houses try is something like "self improvement night" where they try to help each other level up, and it often doesn't work because people are just interested in pretty different skillsets.

Lately I've been noticing myself getting drawn into more demon-thready discussions on LessWrong. This is in part due to UI choice – demon threads (i.e. usually "arguments framed through 'who is good and bad and what is acceptable in the overton window'") are already selected for getting above-average at engagement. Any "neutral" sorting mechanism for showing recent comments is going to reward demon-threads disproportionately.

An option might be to replace the Recent Discussion section with a version of itself that only shows comments and posts from the Questions page (in particular for questions that were marked as 'frontpage', i.e. questions that are not about politics).

I've had some good experiences with question-answering, where I actually get into a groove where the thing I'm doing is actual object-level intellectual work rather than "having opinions on the internet." I think it might be good for the health of the site for this mode to be more heavily emphasized.

In any case, I'm interested in making a LW Team internal option where the mods can opt into a "replace recent discussion with recent question activity" to experiment with living in a world that contains more nudges towards the object level and seeing how that goes.

My current best guess is that the best option includes giving people more choices about Recent Discussion works, and then having the default choice for new users be something a little more magical that is filtered to push things more towards the object level.

I still want to make a really satisfying "fuck yeah" button on LessWrong comments that feels really good to press when I'm like "yeah, go team!" but doesn't actually mean I want to reward the comment in our longterm truthtracking or norm-tracking algorithms.

I think this would seriously help with weird sociokarma cascades.  

Can democracies (or other systems of government) do better by more regularly voting on meta-principles, but having those principles come into effect N years down the line, where N is long enough that the current power structures have less clarity over who would benefit from the change?

Some of the discussion on Power Buys You Distance From the Crime [LW · GW] notes that campaigning to change meta principles can't actually be taken at face value (or at least, people don't take it at face value), because it can be pretty obvious who would benefit from a particular meta principle. (If the king is in power and you suggest democracy, obviously the current power structure will be weakened. If people rely on Gerrymandering to secure votes, changing the rules on Gerrymandering clearly will have an impact on who wins next election)

But what if people voted on changing rules for Gerrymandering, and the rules wouldn't kick in for 20 years. Is that more achievable? Is it better or worse?

The intended benefit is that everyone might roughly agree it's better for the system to be more fair, but not if that fairness will clearly directly cost them. If a rule change occurs far enough in the future, it may be less clear who will benefit from the change.

This is perhaps related to Robin Hanson's Near Mode vs Far mode. I think people are more idealistic in Far Mode... and at least sometimes this just seems good? Lack of clarity over who benefits from a change might improve people's ability to think about longterm benefits and fairness.

Musings on ideal formatting of posts (prompted by argument with Ben Pace)

My thoughts:

1) Working memory is important.

If a post talks about too many things, then in order for me to respond to the argument or do anything useful with it, I need a way to hold the entire argument in my head.

2) Less Wrong is for thinking

This is a place where I particularly want to read complex arguments and hold them in my head and form new conclusions or actions based on them, or build upon them.

3) You can expand working memory with visual reference

Having larger monitors or notebooks to jot down thoughts makes it easier to think.

The larger font-size of LW main posts works against this currently, since there are fewer words on the screen at once and scrolling around makes it easier to lose your train of thought. (A counterpoint is that the larger font size makes it easier to read in the first place without causing eyestrain).

But regardless of font-size:

4) Optimizing a post for re-skimmability makes it easier to refer to.

This is why, when I write posts, I make an effort to bold the key points, and break things into bullets where applicable, and otherwise shape the post so it's easy to skim. (See Sunset at Noon for an example)

Ben's Counter:

Ben Pace noticed this while reviewing an upcoming post I was working on, and his feeling was "all this bold is making me skim the post instead of reading it."

To which all I have to say is "hmm. Yeah, that seems likely."

I am currently unsure of the relative tradeoffs.

New concept for my "qualia-first calibration" app idea that I just crystallized. The following are all the same "type":

1. "this feels 10% likely"

2. "this feels 90% likely"

3. "this feels exciting!"

4. "this feels confusing :("

5. "this is coding related"

6. "this is gaming related"

All of them are a thing you can track: "when I observe this, my predictions turn out to come true N% of the time".

Numerical-probabilities are merely a special case (tho it still gets additional tooling, since they're easier to visualize graphs and calculate brier scores for)

And then a major goal of the app is to come up with good UI to help you visualize and compare results for the "non-numeric-qualia".

Depending on circumstances, it might seem way more important to your prior "this feels confusing" than "this feels 90% likely". (I'm guessing there is some actual conceptual/mathy work that would need doing to build the mature version of this)

"Can we build a better Public Doublecrux?"

Something I'd like to try at LessOnline is to somehow iterate on the "Public Doublecrux" format.

Public Doublecrux is a more truthseeking oriented version of Public Debate. (The goal of a debate is to change your opponent's mind or the public's mind. The goal of a doublecrux is more like "work with your partner to figure out if you should change your mind, and vice vera")

Reasons to want to do _public_ doublecrux include:

• it helps showcase subtle mental moves that are hard to write down explicitly (i.e. tacit knowledge transfer)
• there's still something good and exciting about seeing high profile smart people talk about ideas. Having some variant of that format seems good for LessOnline. And having at least 1-2 "doublecruxes" rather than "debates" or "panels" or "interviews" seems good for culture setting.

Historically I think public doublecruxes have had some problems:

• two people actually changing *their* minds tend to get into idiosyncratic frames that are hard for observers to understand. You're chasing *your* cruxes, rather than presenting "generally compelling arguments." This tends to get into weeds and go down rabbit holes
• – having the audience there makes it a bit more awkward and performative.

...

...

With that in mind, here are some ideas:

• Maybe have the double cruxers in a private room, with videocameras. The talk is broadcast live to other conference-goers, but the actual chat is in a nice cozy room.
• Have _two_ (or three?) dedicated facilitators. One is in the room with the doublecruxers, focused on helping them steer towards useful questions. (this has been tried before seems to go well if the facilitator prepares). The SECOND (and maybe third) facilitator hangs out with the audience outside, and is focused on tracking "what is the audience confused about?". The audience participates in a live google doc where they're organizing the conversational threads and asking questions.
  
  (the first facilitator is periodically surreptitiously checking the google doc or chat and sometimes asking the Doublecruxers questions about it)
• it's possibly worth investing in developing a doublcrux process that's explicitly optimized for public consumption. This might be as simple as having the facilitator periodically asking participants to recap the open threads, what the goal of the current rabbit hole is, etc. But, like, brainstorming and doing "user tests" of it might be worthwhile.

...

Anyway those are some thoughts for now. Curious if anyone's got takes.

Two interesting observations from this week, while interviewing people about their metacognitive practies.

• @Garrett Baker [LW · GW] said that he had practiced memorizing theorems for linear algera awhile back, and he thinks this had (a side effect?) of creating a skill of "memorizing stuff quickly", which then turned into some kind of "working memory management" tool. It sounded something like "He could quickly memorize things and chunk them, and then he could do that on-the-fly while reading math textbooks".
   
• @RobinGoins [LW · GW] had an experience of not being initially able to hold all their possible plans/goals/other in working memory, but then did a bunch of Gendlin Focusing on them, and then had an easier time holding them all. It sounds like the Gendlin Focusing was playing a similar role to the "fast memorization" thing, of "finding a [nonverbal] focusing handle for a complex thing", where the focusing handle was able to efficiently unpack into the full richness of the thing they were trying to think about.

Both of these are interesting because they hint at a skill of "rapid memorization => improved working memory". 

@gwern [LW · GW] has previously written about Dual N Back not actually working that well at improving IQ. It seems like history is littered with corpses of people trying to improve IQ or g, so I'm not too optimistic here. My current assumption/guess is that the Dual N Back stuff trained a particular skill that turned out not to transfer to other domains. 

But, like, even if "rapidly memorize math proofs" didn't generalize to anything other than memorizing math proofs, it feels plausible to me that this could at least help with situations where that particular skill is useful, and might be worth it even without domain transfer.

And I could imagine that there's something of a skill of "learn to rapidly chunk content in a given domain", which doesn't automatically translate to other domains, but which makes it easier to learn to chunk new types of domains, similar to how learning one language doesn't let you speak all languages but makes it easier to learn ones.

I think a bunch of discussion of acausal trade might be better framed as "simulation trade." It's hard to point to "acausal" trade in the real world because, well, everything is at least kinda iterated and at least kinda causally connected. But, there's plenty of places where the thing you're doing is mainly trading with a simulated partner. And this still shares some important components with literal-galaxy-brains making literal acausal trade.

So, AFAICT, rational!Animorphs is the closest thing CFAR has to publicly available documentation. (The characters do a lot of focusing, hypothesis generation-and-pruning. Also, I just got to the Circling Chapter)

I don't think I'd have noticed most of it if I wasn't already familiar with the CFAR material though, so not sure how helpful it is. If someone has an annotated "this chapter includes decent examples of Technique/Skill X, and examples of characters notably failing at Failure Mode Y", that might be handy.

In response to lifelonglearner's comment I did some experimenting with making the page a bit bolder. Curious what people think of this screenshot where "unread" posts are bold, and "read" posts are "regular" (as opposed to the current world, where "unread" posts "regular", and read posts are light-gray).

↑ comment by Raemon · 2019-12-07T01:50:31.645Z · LW(p) · GW(p)

Alternate version where only the title and karma are bolded:

Replies from: Evan Rysdam, adam_scholl, RobbBB

↑ comment by Sunny from QAD (Evan Rysdam) · 2019-12-07T03:54:58.971Z · LW(p) · GW(p)

I think I prefer the status quo design, but not very strongly. Between the two designs pictured here, I at first preferred the one where the authors weren't bolded, but now I think I prefer the one where the whole line is bolded, since "[insert author whose posts I enjoy] has posted something" is as newsworthy as "there's a post called [title I find enticing]".

Something I've noticed about myself is that I tend to underestimate how much I can get used to things, so I might end up just as happy with whichever design is chosen.

↑ comment by Adam Scholl (adam_scholl) · 2019-12-07T06:12:13.527Z · LW(p) · GW(p)

Fwiw, for reasons I can't explain I vastly prefer just the title bolded to the entire line bolded, and significantly prefer the status quo to title bolded.

↑ comment by Rob Bensinger (RobbBB) · 2019-12-07T02:25:16.405Z · LW(p) · GW(p)

I think I prefer bolding full lines b/c it makes it easier to see who authored what?

Replies from: Raemon

↑ comment by Raemon · 2019-12-07T02:35:23.101Z · LW(p) · GW(p)

I initially wanted "bold everywhere" because it helped my brain reliably parse things as "this is a bold line" instead of "this is a line with some bold parts but you have to hunt for them". But, after experimenting a bit I started to feeling having bold elements semi-randomly distributed across the lines made it a lot busier.

Issues with Upvoting/Downvoting

We've talked in the past about making it so that if you have Karma Power 6, you can choose whether to give someone anywhere from 1-6 karma.

Upvoting

I think this is an okay solution, but I also think all meaningful upvotes basically cluster into two choices:

A. "I think this person just did a good thing I want to positively reinforce"

B. "I think this person did a thing important enough that everyone should pay attention to it."

For A, I don't think it obviously matters that you award more than 1 karma, and definitely never more than 3 karma. The karma should be mostly symbolic. For B, I'd almost always want to award them maximum karma. The choice of "well, do they really deserve 1, 2 or 3 karma for their pat-on-the-head?" doesn't seem like a choice we should be forcing people to make.

The value in giving 1, 2 or 3 karma for a "small social reinforcement" is mostly about communicating "Social rewards from longtime trusted community members should feel better to get than social rewards from random newbies." I'm not sure how strong a signal this is.

For "Pay Attention To This" upvotes, similarly, if you have 6 karma power, I don't think it's that interesting a choice to assign 4, 5 or 6.

And, you know, Choices Are Bad [LW · GW].

So, I currently support a paradigm where you just have Big Upvote and Small Upvote. I'm neutral between "small upvote is always 1" or "small upvote grows from 1 to 3 as you gain karma

This feels elegant. The problem is downvoting.

Downvoting

When downvoting, there's a few different things I might be wanting to do (note: I don't endorse all of these, this is just what my S1 is wanting to do).

A. This person made a small mistake, and should be mildly socially punished

B. This person was deeply wrong and should heavily punished

C. This post is getting too much attention relative to how good it is. It's at 25 karma. I want to try to bring to around 15 or something.

D. This content should not be on the site (for any one of a number reasons), should not show up on the frontpage (meaning the karma should be at most zero) or the comment should be autocollapsed (karma should be -5)

When a newcomer shows up and does something I don't like, my natural instinct is to try to keep their comment at 0 (which feels like the right level of "your thing was bad", but in a way that feels more like an awkward silence than a slap in the face. I definitely need to be able to downvote by less than 6. The problem is as a user gains karma power, the amount I need to downvote just scales linearly.

This is all incompatible with the simple "Big Vote, Small Vote" paradigm. Which feels sad from an elegance/symmetry perspective.

So, that's a thing I'm thinking about.

I think learning-to-get-help is an important, often underdeveloped skill. You have to figure out what *can* be delegated. In many cases you may need to refactor your project such that it's in-principle possible to have people help you.

Some people I know have tried consciously developing it by taking turns being a helper/manager. i.e. spend a full day trying to get as much use out of another person as you can. (i.e. on Saturday, one person is the helper. The manager does the best they can to ask the helper for help... in ways that will actually help. On Sunday, they reverse)

The goal is not just to get stuff done for a weekend, but to learn how ask for help, to help, to be helped.

(Some people I know did this for a full day, others did it for an hour. The people who did it for an hour said it didn't quite feel that useful. A person who did it for a full day said that an hour was nowhere near enough time to make it through the initial learning curve of "I don't even know what sort of things are useful to ask for help with.")

So, this is a thing I'm interested in trying.

I think it requires some existing trust and being able to work side-by-side, so I'm mostly extending a request/offer to do this for a weekend with people who already know me and live near me, but am curious if other people try it and get benefit out of it.

With some frequency, LW gets a new user writing a post that's sort of... in the middle of having their mind blown by the prospect of quantum immortality and MWI. I'd like to have a single post to link them to that makes a fairly succinct case for "it adds up to normality", and I don't have a clear sense of what to do other that link to the entire Quantum Physics sequence. 

Any suggestions? Or, anyone feel like writing said post if it doesn't exist yet?

Draft/WIP: The Working Memory Hypothesis re: Intellectual Progress

Strong claim, medium felt

So I'm working with the hypothesis that working memory (or something related) is a major bottleneck on progress within a given field. This has implications on what sort of things fields need.

Basic idea is that you generally need to create new concepts out of existing sub-concepts. You can only create a concept if you can hold the requisite sub-concepts in your head at once. Default working memory limits is 4-7 chunks. You can expand that somewhat by writing things down on whiteboards. I'm not sure exactly what the practical limits are.

But there is some upper limit on how complex a concept people can work with, even the smartest people in the field. So there are some ideas you just can't invent, if they require 30 moving parts to consider at once. If an idea has 30 moving parts, it's necessary to find some way to conceptualize it as having fewer parts.

Fortunately, this is possible. When I first learn a new concept, it's unfamiliar, and I have to track all of it's individual pieces separately – if it has 5 sub-concepts it takes 5 chunks of working memory.

But eventually I learn to think of it as a single concept, and then it only takes 1.

So, an important aspect of a field is distillation – improving explanations to help people think of clusters-of-ideas as a single concept they can reason about as if it took up only one chunk.

Distillation is a sort of tedious process though. So part of why Bell Labs is a great idea is that you can have all these specialists of different fields working in the same building, and whenever one of them wants to learn something they can just walk down the hall and ask the guy who invented it "hey, how does this work", and they get an explanation. And if they don't understand it fully, they can immediately, frictionlessly, ask followup questions.

This doesn't just help them understand new ideas – it reduces those ideas in conceptual complexity until they only take up a single chunk, enabling them to be combined with other ideas.

This has implications for the AI Alignment field – investing in distillation and co-location are both potentially quite valuable. (But, correspondingly – watch out for things that improve the distillation of ideas that feed into AI capabilities)

How (or under what circumstances), can people talk openly about their respective development stages?

A lot of mr-hire's recent posts [LW(p) · GW(p)] (and my own observations and goals) have updated me on the value of having an explicit model of development stages. Kegan levels are one such frame. I have a somewhat separate frame of "which people I consider 'grown up'" (i.e. what sort of things they take responsibility for and how much that matters)

Previously, my take had been "hmm, it seems like people totally do go through development stages, which do typically come in particular order and later development stages are better than early ones. But all the conversations I've seen where anyone brought up development stages seem like terrible conversations."

Basically, the social move of "you seem like you're at a lower development level than me" is often legitimately read as a status attack, and worse, a plausibly-deniable status attack*.

But, also, it's sometimes an important part of the conversation.

(*In particular, when people are within one level of each other, and it's ambiguous. When people are two levels apart from each other, it's usually more obvious to both of them and to third parties which is more cognitively sophisticated)

My recent updates, clarified by mr-hire, were that if you're building an organization (and, perhaps, a community), you need some way to actually account for different people being at different development stages. This requires actually having a model of what's going on. But even the construction of the model can be legitimately read as a plausibly deniable status attack – as the model gets fleshed out it's going to become increasingly clear who it puts into positions of power.

For discussions between individuals about who is "more cognitively sophisticated", my current best guess is that you can actually have this conversation reasonably easily in private (where by "reasonably easily", I mean it maybe takes several hours of building trust and laying groundwork, but there's nothing mysterious about it)

For discussions about how to build and developmental ontology in a mixed environment with no hierarchy and where people start off with very different frames on how to think, I feel pretty confused and/or worried.

I am very confused about how to think (and feel!) about willpower, and about feelings of safety.

My impression from overviews of the literature is something like "The depletion model of willpower is real if you believe it's real. But also it's at least somewhat real even if you don't?"

Like, doing cognitive work costs resources. That seems like it should just be true. But your stance towards your cognitive work affects what sort of work you are doing.

Similarly, I have a sense that physiological responses to potentially threatening situations are real. People who feel defensive have a harder time thinking in truthseeking mode rather than "keep myself safe" mode. But, it also seems plausibly-true that if you naively reinforce feelings of defensiveness they get stronger. i.e. if you make saying "I'm feeling defensive" a get out of jail free card, people will use it, intentionally or no. (there's a weird video about parents tricking babies into thinking they've been hit on the head when they haven't, and the babies start crying as if they're hurt. I have no idea if this is just cute selection effect but sort of illustrates the point)

There's a practical question re: "what sort of norms or advice do you want to encourage about how people deal with defensiveness, and with willpower?". But then there's also just a "what the hell epistemic state do you actually want to have?" as well as "how do you communicate about this?"

If you're a manager at a company, maybe you want to have a model of willpower depletion so you can make good tradeoffs, but you don't want your employees to dwell upon it much? But this obviously collapses as soon as there's multiple management layers. And in any case it's not a sustainable equilibrium. (You can't stop people from learning about willpower. Although you can make a choice not to emphasize it)

Meanwhile, if I'm just trying to manage myself as a person with goals, what should my epistemic state on willpower be?

I'm currently pretty torn between:

• "Try to actually resolve the longstanding major disagreements about what sort of culture is good for LessWrong"
• "Attempt to build real archipelago features [LW · GW] that let people self segregate into whatever discussions they want."
• "Attempt to mostly bypass that discussion by just focusing on the Open Questions feature-set, with an emphasis on object-level questions."

The disagreements about "combat vs collaboration" and other related frames do seem to have real, important things to resolve. I think a lot of the debate can be broken into empirical questions that are (in theory) actually possible to resolve. But... unless people are actually in agreement about a meta-frame that would actually resolve it, mostly it seems like a massive, net negative time sink.

Archipelago hasn't worked, but, well "Real Archipelago hasn't even been attempted". But I'm not sure it actually helps. There's a few key unresolved questions like 'what are the default norms for users that haven't set moderation guidelines' which more or less necessitate solving the first option. There's also the issue wherein at least some ongoing debates have people who prefer different norms.

The latter can maybe be addressed by setting a stronger meta-norm of "if you think the discussion on Post X is important but has counterproductive norms, you can create you own post about it", possibly encouraging people more to create short posts that just say "this is my discussion for topic X, with norm Y". Something about that still feels unsatisfying.

Meanwhile, Open Questions that focus on object level problems mostly don't seem to generate demon threads [LW · GW]. They typically meet my own preferences for collaborativeness (since there's relatively clear criterion for comments of 'is this helping to answer the question that the author asked?'), while usually avoiding most of the issues raised by (my understanding of) people who are annoyed by pressures toward collaborative-ness.

(i.e. my experience is that the open question framework creates an environment that is better suited towards blunt disagreement, at least about factual things, conditional on the questions being object level. And while there's still sometimes disagreement over the best frame to answer a question is, that feels like a much simpler thing to patch)

I notice that I'm increasingly confused that Against Malaria Foundation isn't just completely funded.

It made sense a few years ago. By now – things like Gates Foundation seem like they should be aware of it, and that it should do well on their metrics.

It makes (reasonable-ish) sense for Good Ventures not to fully fund it themselves. It makes sense for EA folk to either not have enough money to fully fund it, or to end up valuing things more complicated than AMF. But it seems like there should be enough rich people and governments for whom "end malaria" is a priority that the $100 million or so that it should just be done by now.

What's up with that?

[cn: spiders I guess?]

I just built some widgets for the admins on LW, so that posts by newbies and reported comments automatically show up in a sidebar where moderators automatically have to pay attention to them, approving or deleting them or sometimes taking more complicated actions.

And... woahman, it's like shining a flashlight into a cave that you knew was going to be kinda gross, but you weren't really prepared to a million spiders suddenly illuminated. The underbelly of LW, posts and comments you don't even see anymore because we installed karma filters on the frontpage.

There's a webcomic called Goblins, where one goblin decided to become a paladin, and gains the ability to Detect Evil. And suddenly is confronted with all the evil lurking about, in the shadows of people's hearts, literal shadows, and sometimes in broad daylight. And he's describing this to a fellow goblin, and they're like "Holy hell, how can you live like that!? Why would you choose to _force_ yourself to see the evil around you?"

And Goblin A nods gravely and says "so that you don't have to."

For this year's LessWrong Review, we're building UI to make it much easier to import linkposts from other blogs, since a lot of important rationalsphere or AI Safety content lives in other places, and backdate it such that it's eligible for the Review.

It's actually pretty easy to automatically import all the text from a url in most cases (We're looking into auto-importing PDFs of papers, which I suspect is doable but haven't checked), and in many cases I think this would basically be preferred, but it's also kinda exploitable in ways I don't know that I'd endorse. (i.e. some authors are probably happy to have people crosspost stuff while nominating it for Best of LessWrong, other authors might feel violated)

Three options are:

• only auto-import the first few paragraphs, ending with a load more
• have an LLM extract some important highlights. (I'm ignoring "have an LLM summarize it" because they suck at that, but I think they're decent at identifying key paragraphs)
• start off by auto-importing the whole post, and then wait until anybody complains.

I'd probably be limiting this to users who are otherwise eligible to nominate (i.e. their account is at least two years old, and maybe they have like 100 karma), so randos can't go crazy with it. Admins will be seeing all posts imported this way so we can be sanity checking things.

Curious what people think.

Have you used the LessWrong Concepts page, or generally used our tagging/wiki features? I'm curious to hear about your experience.

I'm particularly interested in people who read content from them, rather than people who contribute content to them. How do you use them? Do you wish you could get value from them better?

↑ comment by papetoast · 2024-08-18T08:15:37.432Z · LW(p) · GW(p)

How do you use them?

I use it when I am interested in learning about a specific topic. I rarely use the Concepts page, because it contains too many tags, and sometimes I don't even know what tag I am looking for. Instead, I usually already have one or two articles that I have previously read, which feels similar to the topic I am thinking about. I would then search for those posts, look at the tags, and click on the one that is relevant. In the tag page, I start by reading the wiki, but often feel disappointed by the half-done/incompleteness of the wiki. Then I filter by high karma and read the articles from top to bottom, skipping ones that feels irrelevant or uninteresting based on title.

Do you wish you could get value from them better?

I wish the default most relevant ordering is not based on the raw score, but rather a normalized relevance score or something more complicated, because right now it means nothing other that "this post is popular so a lot of people voted on the tags". This default is really bad, every new user has to independently realize that they should change the sorting. LW also does not remember the sorting so I have to change it manually every time, which is irritating but not a big deal.

Replies from: Raemon

↑ comment by Raemon · 2024-08-18T16:58:26.640Z · LW(p) · GW(p)

I rarely use the Concepts page, because it contains too many tags, and sometimes I don't even know what tag I am looking for.
 

Do you feel like you have a missing usecase that the concepts page should be helpful with?

Replies from: papetoast

↑ comment by papetoast · 2024-08-19T07:22:00.031Z · LW(p) · GW(p)

To answer your question directly - not really.

I think index pages are just meant to be used by only a small minority of people in any community. In my mind, the LW concepts page is like the wiki topic groups (not sure what they're called).

The similarities are:

1. It is fun to go through the concepts page and find tags I haven't learned about, this is good for exploration but a rare use case (for me)
2. Because it is an index, it is useful when you have a concept in your mind but couldn't remember the name

But the concepts page has a worse UX than wiki since you have to explicitly search for it, rather than it popping up in the relevant tags page, and also they show up in a cluster

One concrete skill I gained from my 2 weeks of Thinking Physics problems [LW · GW]was:

1. Notice something feels intractably hard
2. Ask "okay, why is this intractably hard?". This might be multiple reasons.
   1. Do those reasons seem intractably hard to fix? If so, recurse and ask "why" again.
   2. Does one of them not seem intractably hard? Then, make a plan for fixing it. Then, if that plan seems cost-effective, do the plan.

• Your intractably hard problem is now solved!

This doesn't seem very novel ("break a problem down into simpler problems" is a pretty canonical tool). But I felt like I got a more visceral understanding of the skill, and how to notice its relevance.

Theory that Jimrandomh was talking about the other day, which I'm curious about:

Before social media, if you were a nerd on the internet, the way to get interaction and status was via message boards / forums. You'd post a thing, and get responses from other people who were filtered for being somewhat smart and confident enough to respond with a text comment.

Nowadays, generally most people post things on social media and then get much more quickly rewarded via reacts, based on a) a process that is more emotional than routed-through-verbal-centers, and b) you are get rewards from a wider swath of the populationl. Which means, in practice, you're getting your incentive gradient from less thoughtful people, both due to the medium, and due to regression to the mean.

This feeds a bit into my model of "Do we want reacts on LessWrong [LW · GW]?", and when/why reacts might be bad for society.

I'd previously talked about how it would be neat if LW reacts specifically gave people affordance to think subtler epistemically-useful thoughts. 

This new model adds a thing like "Maybe we actually just want reacts to be available to people with 1000+ karma or so. So, they increase the signal ratio from people who have demonstrated at least some reasonable threshold of thoughtfulness." (This has the obvious downside of increasing groupthink, which I do take seriously, but there's an unfortunate tradeoff between "increasing groupthink" and "getting your signal from random society which is pretty bad", and I'd currently lean towards the former if I had to pick one. I do eventually want to get a filtering system that selects more directly on "thoughtfulness", more reliably than the karma system does)

The latest magic set has… possibly the subtlest, weirdest take on the Magic color wheel so far. The 5 factions are each a different college within a magical university, each an enemy-color-pair.

The most obvious reference here is Harry Potter. And in Harry Potter, the houses map (relatively) neatly to various magic colors, or color pairs.

Slytherin is basically canonical MTG Black. Gryffindor is basically Red. Ravenclaw is basically blue. Hufflepuff sort of green/white. There are differences between Hogwarts houses and Magic colors, but they are aspiring to very similar archetypes. And that’s what I was initially expecting out of “Wizards of the Coast makes a magic set inspired by Magical School YA Fiction.”

But, each of the factions in Strixhaven is *quite weird*, at least by MTG standards. At first I was very confused. Now that I've had more time to think about it I am pretty impressed.

Each faction is basically a department focused on particular clusters of classes and interests. They are each defined by *a major philosophical argument that divides the field*, where people argue what the point of the field is and what paradigm it should be operating under.

One of the more straightforward ones is, say, the college of the arts, which is Blue/Red. Everyone involved agrees you're supposed to make good, skillful art. But, they argue over whether the point of art is to make you *think*, and philosophically engage with things, or to make you *feel*, and convey raw emotion.

A weirder one is the humanities department – history, psychology, anthropology. The humanities in this school are White/Red, and their defining debate is "is core human condition (er, 'humanoid' condition) primarily about how people relate to the systems and rules they created, or about the close relations and bonds that individuals created themselves?"

The college of mathematics is Blue/Green, with shared love of fractals, the laws governing nature, etc... who debate whether mathematics is a "natural, platonic thing" that sentients merely discovered, or is it a tool they created?

And then...

...there's the communication department. Which is White/Black. Whose central debate is about whether the point of communication is to serve the public good and benefit society, or to manipulate the social fabric to benefit yourself.

I mentioned this last bit to Jim Babcock, who said "WHAT!? *Neither* of those is the point of communication!"

And I said "Yeah it sure would be better if the communications department was Blue, wouldn't it? Good thing this is just a fantasy world created for fun and not at all reflective of the real world."

After starting up PredictionBook, I've noticed I'm underconfident at 60% (I get 81% of my 60% predictions right) and underconfident at 70% (only get 44% right).

This is neat... but I'm not quite sure what I'm actually supposed to do. When I'm forming a prediction, often the exact number feels kinda arbitrary. I'm worried that if I try to take into account my under/overconfidence, I'll end up sort of gaming the system rather than learning anything. (i.e. look for excuses to shove my confidence into a bucket that is currently over/underconfident, rather than actually learning "when I feel X subjectively, that corresponds to X actual confidence."

Curious if folk have suggestions.

Someone recently mentioned that strong-upvotes have a particular effect in demon-thread-y comment sections, where if you see a Bad Comment, and that that comment has 10 karma, you might think "aaah! the LessWrong consensus is that a Bad Comment is in fact Good! And this must be defended against."

When, in fact, 10 karma might be, like, one person strong-upvoting a thing.

This was a noteworthy point. I think the strong upvotes usually "roughly does their job" in most cases, but once things turn "contested" they quickly turn into applause/boo lights in a political struggle. And it might be worth looking into ways to specifically curtail their usefulness in that case somehow.

After this weeks's stereotypically sad experience with the DMV....

(spent 3 hours waiting in lines, filling out forms, finding out I didn't bring the right documentation, going to get the right documentation, taking a test, finding out somewhere earlier in the process a computer glitched and I needed to go back and start over, waiting more, finally getting to the end only to learn I was also missing another piece of identification which rendered the whole process moot)

...and having just looked over a lot of 2018 posts [? · GW] investigating coordination failure... 

I find myself wondering if it's achievable to solve one particular way in which bureaucracy is terrible: the part where each node/person in the system only knows a small number of things, so you have to spend a lot of time rehashing things, and meanwhile can't figure out if your goal is actually achievable.

(While attempting to solve this problem, it's important to remember that at least some of the inconvenience of bureaucracy may be an active ingredient rather than inefficiency. But at least in this case it didn't seem so: drivers licenses aren't a conserved resource that the DMV wants to avoid handing out. If I had learned early on that I couldn't get my license last Monday it would have not only saved me time, but saved DMV employee hassle)

I think most of the time there's just no incentive to really fix this sort of thing (while you might have saved DMV employee hassle, you probably wouldn't save them time, since they still just work the same 8 hour shift regardless. And if you're the manager of a DMV you probably don't care too much about your employees having slightly nicer days.

But, I dunno man, really!?. Does it seem like at least Hot New Startups could be sold on software that, I dunno, tracks all the requirements of a bureaucratic process and tries to compile "will this work?" at start time?

I don't know of a principled way to resolve roomate-things like "what is the correct degree of cleanliness", and this feels sad.

You can't say "the correct amount is 'this much' because, well, there isn't actually an objectly correct degree of cleanliness."

If you say 'eh, there are no universal truths, just preferences, and negotiation', you incentivize people to see a lot of interactions as transactional and adversarial that don't actually need to be. It also seems to involve exaggerating and/or downplaying one's own preferences.

The default outcome is something like "the person who is least comfortable with mess ends up doing most of the cleaning". If cleanliness were just an arbitrary preference this might actually be fine, especially if they really do dramatically care more about it. But usually it's more like "everyone cares at least a bit about being clean, one person just happens to care, say, 15% more and be more quick to act." So everyone else gets the benefits without paying the cost.

I think there's a preformal / formal / post-formal thing going on with Double Crux.

My impression is the CFAR folk who created the doublecrux framework see it less as a formal process you should stick to, and more as a general set of guiding principles. The formal process is mostly there to keep you oriented in the right direction.

But I see people (sometimes me) trying to use it as a rough set of guiding principles, and then easily slipping back into all the usual failure modes of not understanding each other, or not really taking seriously the possibility that they might be the wrong one.

Right now in some contexts I've come across as a bit anal about sticking to "Formal Doublecrux rules". Model share. Check for cruxes. Recurse until you find common cruxes. Look for experiments you can actually run to gain new evidence. Aim to converge on truth.

And it does clearly seem that these steps aren't always the best approach for a given conversation. But I often perceive what feel like basic errors, which would have been caught if you were following the formal rules.

So I'm currently, like, on a crusade to make sure the people around me that I end up in the most disagreements with are able to nail the Formal Doublecrux Framework, and once we're all roughly on that page I'll trust us to do a post-formal version of it where we trust each other to get the basics right, so we can relax about the rules.

Counterfactual revolutions are basically good, revolutions are basically bad

(The political sort of revolution, not the scientific sort)

Possible UI:

What if the RecentDiscussion section specifically focused on comments from old posts, rather than posts which currently appear in Latest Posts. This might be useful because you can already see updates to current discussions (since comments turn green when unread, and/or comment counts go up), but can't easily see older comments.

(You could also have multiple settings that handled this differently, but I think this might be a good default setting to ensure comments on old posts get a bit more visibility)

Weird thoughts on 'shortform'

1) I think most of the value of shortform is "getting started writing things that turn out to just be regular posts, in an environment that feels less effortful."

2) relatedly, "shortform" isn't quite the right phrase, since a lot of things end up being longer. "Casual" or "Off-the-cuff" might be better?

Failure Modes of Archipelago

(epistemic status: off the cuff, maybe rewriting this as a post later. Haven't discussed this with other site admins)

In writing Towards Public Archipelago, I was hoping to solve a couple problems:

• I want authors to be able to have the sort of conversational space that they actually want, to incentivize them to participate more
• I want LW's culture to generally encourage people to grow. This means setting standards that are higher than what-people-do-by-default. But, people will disagree about what standards are actually good. So, having an overarching system whereby people can try out and opt-into higher-level-standards that they uphold each other to seems better than fighting what the overall standards of the site should be.

But, I've noticed an obvious failure mode. For Public Archipelago to work as described, you need someone who is:

• willing to enforce rules
• writes regularly, in a way that lends itself towards being a locus of conversation.

(In non-online spaces, you have a different issue, where you need someone who runs some kind of physical in-person space who is willing to enforce norms who is also capable of attracting people to their space)

I have a particular set of norms I'd like to encourage, but most of the posts I write that would warrant enforcing norms are about meta-stuff-re-Less-Wrong. And in those posts, I'm speaking as site admin, which I think makes it important for me to instead be enforcing a somewhat different set of norms with a higher emphasis on fairness.

(i.e. if site admins start deleting your comments on a post about what sort of norms a site should have, that can easily lead to some real bad chilling effects. I think this can work if you're very specific about what sort of conversation you want to have, and make your reasons clear, but there's a high risk of it spilling into other kinds of damaged trust that you didn't intend)

My vague impression is that most of the people who write posts that would benefit from some kind of norm-enforcing are somewhat averse to having to be a norm-enforcer.

Some people are willing to do both, but they are rare.

So the naive implementation of Public Archipelago doesn't work that well.

Problematic Solution #1: Subreddits

Several people suggested subforums as an alternative to author-centric Islands.

First, I think LW is still too small for this to make sense – I've seen premature subreddits kill a forum, because it divided everyone's attention and made it harder to find the interesting conversation.

Second, I don't think this accomplishes the same thing. Subforurms are generally about topics, and the idea I'm focusing on here is norms. In an AI or Math subforum, are you allowed to ask newbie questions, or is the focus on advanced discussion? Are you allowed to criticize people harshly? Are you expected to put in a bunch of work to answer a question yourself before you answer it?

These are questions that don't go away just because you formed a subforum. Reasonable people will disagree on them. You might have five people who all want to talk about math, none of who agree on all three of those questions. Someone has to decide what to enforce.

I'm very worried that if we try to solve this problem with subreddits, people will run into unintentional naming collisions where someone sets up a space with a generic name like "Math", but with one implicit set of answers to norm-questions, and then someone else wants to talk about math with a different set of answers, and they get into a frustrating fight over which forum should have the simplest name (or force all subforums to have oddly specific names, which still might not address all the nuances someone meant to convey)

For this reason, I think managing norms by author(s), or by individual-post makes more sense.

Problematic Solution #2: Cooperation with Admins

If a high-karma user sets their moderation-policy, they have an option to enable "I'm happy for admins to help enforce my policy." This allows people to have norms but outsource the enforcing of them.

We haven't officially tried to do this yet, but in the past month I've thought about how I'd respond in some situations (both on LW and elsewhere) where a user clearly wanted a particular policy to be respected, but where I disagreed with that policy, and/or thought the user's policy wasn't consistent enough for me to enforce it. At the very least, I wouldn't feel good about it.

I could resolve this with a simple "the author is always right" meta-policy, where even if an author seems (to me) to be wanting unfair or inconsistent things, I decide that giving authors control over their space is more important than being fair. This does seem reasonable-ish to me, at least in principle. I think it's good, in broader society, to have police who enforce laws even when they disagree with them. I think it's good, say, to have a federal government or UN or UniGov that enforces the right of individual islands to enforce their laws, and maybe this includes helping them do so.

But I think, at the very least, this requires a conversation with the author in question. I can't enforce a policy I don't understand, and I think policies that may seem simple-to-the-author will turn out to have lots of edge-cases.

The issue is that having that conversation is a fairly non-trivial-inconvenience, which I think will prevent most instances of admin-assisted-author-norms from coming to fruition.

Variant Solution #2B: Cooperation with delegated lieutenants

Instead of relying on admins to support your policy with a vaguely-associated halo of "official site power structure", people could delegate moderation to specific people they trust to understand their policy (either on a per-post or author-wide system).

This involves a chain-of-trust. (The site admins have to make an initial decision about who gains the power to moderate their posts, and if this also includes delegating moderation rights the admins also need to trust the person to choose good people to enforce a policy). But I think that's probably fine?

Variant Solution #2C: Shared / Open Source Norms

Part of the problem with enforcing norms is that you need to first think a bunch about what norms are even good for and which ones you want. This is a hugely non-trivial inconvenience.

A thing that could help this a bunch is to have people who think a lot about norms posting more about their thought process, and which norms they'd like to see enforced and why. People who are then interested in having norms enforced on their post, and maybe even willing to enforce those norms themselves, could have a starting point to describe which ones they care about.

Is there a good LLM tool that just wraps GPT or Claude with a speech-to-text input and text-to-speech output? I'd like to experiment with having an aways-on-thinking assistant that I talk out loud to.

I've recently updated on how useful it'd be to have small icons representing users. Previously some people were like "it'll help me scan the comment section for people!" and I was like "...yeah that seems true, but I'm scared of this site feeling like facebook, or worse, LinkedIn."

I'm not sure whether that was the right tradeoff, but, I was recently sold after realizing how space-efficient it is for showing lots of commenters. Like, in slack or facebook, you'll see things like:

This'd be really helpful, esp. in the Quick Takes and Popular comments sections, where you can see which people you know/like have commented to a thing

I... had a surprisingly good time reading Coinbase's Terms of Service update email?

We’ve recently updated our User Agreement. To continue using our services and take advantage of our upcoming feature launches, you’ll need to sign in to Coinbase and accept our latest terms.

You can read the entire agreement here. At a glance, here’s what this update means for you:

Easier to Understand: We’ve reorganized and modified our user agreement to make it more understandable and in line with our culture of clear communications.

Clarity on Dispute Resolution: We’ve revised our arbitration agreement to streamline the process for resolving problems you may experience.

Transparent Product Terms: We’ve consolidated the terms relating to many of our new products in a single appendix to make them easier to find.

Restrictions on Use of Services: We’ve created a standalone policy to make it easy to understand how you may use our services.

• Response to Regulatory and Legal Changes: We’ve updated our tax language and some of our definitions to reflect the fact that crypto and crypto law are always on the move!

Please read the updated User Agreement and check out our help center article if you have any questions. 

What I like about this was that normally Terms of Service are just incredibly opaque to me, and when someone says they updated them I just shrug helplessly.

And here... well... to be clear I totally haven't checked if this is actually a good distillation of their changes, or how adversarial their terms of service are.But when I imagine a good, benevolent company trying to have a reasonable terms-of-service, and communicating clearly about it, it seems like actually a hard problem. (I'm currently looking at this through a lens that has general alignment parallels – communicating truthfully in an ontology your user can understand is a difficult problem)

And this email seemed like it was trying (to at least pretend to) do a good job with that.It's still not sufficient. (this email would be much better if instead of saying "We changed something about X", it said more explicitly "we changed X to Y, here's a short summary of Y")But it was an interesting signpost along the way to clear communication in a world of complicated interlocking systems

This is a response to Zack Davis in the comments on his recent post [LW(p) · GW(p)]. It was getting increasingly meta, and I wasn't very confident in my own take, so I'm replying over on my shortform.

OP is trying to convey a philosophical idea (which could be wrong, and whose wrongness would reflect poorly on me [LW · GW], although I think not very poorly, quantitatively speaking [LW · GW]) about "true maps as a Schelling point." (You can see a prelude to this in the last paragraph of a comment of mine from two months ago [LW(p) · GW(p)].)

I would have thought you'd prefer that I avoid trying to apply the philosophy idea to a detailed object-level special case (specifically, that of this website) in the comment section of a Frontpaged post (as a opposed to a lower-visibility meta post or private conversation)?? (Maybe this is another illustration of Wei's point that [LW · GW] our traditional norms just end up encouraging hidden agendas.)

I'm legitimately unsure what the correct norm here is at this point. (I was recently also writing a post that made a more general point, but all my examples were from working on LessWrong, and not sure about the best approach because, years from now, I do still want to be able to link to the post regardless of whether the object-examples are still salient)

One thing that I think clearly fits the currently implemented norms (albeit is higher effort), is to write two posts, one focusing on the crisp abstraction and one on the object-level politics. I think doing both in close proximity is both more honest and more helpful (since no one has to second guess if there's a hidden agenda, but you can also be putting forth what is hopefully a good abstraction that will apply in future situations)

I think, when followed, the above norm produces better writing, in part because forcing yourself to look for 1-2 examples other than the current object-level-situation forces you to check if it's a real pattern. (Although, another problem is it may be "all the examples"  are from some variety of local politics, if those are the places one actually has clear enough knowledge)

The main counterpoint is that this all makes writing a lot harder, and I'm not confident it's worth that extra effort. (And I think there are then downstream effects on what gets written and where people bother writing things up that are potentially bad)

Within the current normset (which I think is important to stick to for now, so that law can be predictable), another option is to go ahead and write the more opinionated post and leave it on Personal Blog (which does tend to end up getting seen by the people who actually have context on the local political situation)

The 2018 Long Review (Notes and Current Plans)

I've spent much of the past couple years pushing features that help with the early stages of the intellectual-pipeline – things like shortform [? · GW], and giving authors moderation tools [LW · GW] that let them have the sort of conversation they want (which often is higher-context, and assuming a particular paradigm that the author is operating in)

Early stage ideas benefit from a brainstorming, playful, low-filter environment. I think an appropriate metaphor for those parts of LessWrong are "a couple people in a research department chatting about their ideas."

But longterm incentives and filters matter a lot as well. I've focused on the early stages because that's where the bottleneck seemed to be, but LessWrong is now at a place where I think we should start prioritizing the later stages of the pipeline – something more analogous to publishing papers, and eventually distilling them into textbooks. 

So, here's the current draft of a plan that I've been discussing with other LW Team members:

— The Long Review Format —

Many LessWrong posts are more conceptual than empirical, and it's hard to tell immediately how useful they are. I think they benefit a lot from hindsight. So, once each year, we could reflect as a group about the best posts of the previous year*, and which them seem to have withstood the tests of time as something useful, true, and (possibly), something that should enter in the LessWrong longterm canon that people are expected to be familiar with.

Here's my current best guess for the format:

[note: I currently expect the entire process to be fully public, because it's not really possible for it to be completely private, and "half public" seems like the worst situation to me]

• (1 week) Nomination
  • Users with 1000+ karma can nominate posts from 2018-or-earlier, describing how they found the post useful over the longterm.
• (4 weeks) Review Phase
  • Authors of nominated posts can opt-out of the rest of the review process if they want.
    • Posts with 3* nominations are announced as contenders. For a month, people are encouraged to look at them thoughtfully, writing comments (or posts) that discuss:
      • How has this post been useful?
      • How does it connect to the broader intellectual landscape.
      • Is this post epistemically sound?
      • How could it be improved?
      • What further work would you like to see people do with the content of this post?
    • Authors are encouraged to engage with critique. Ideally, updating the post in response to feedback, and/or discussing what sort of further work they'd be interesting seeing by others.
• (1 Week) Voting
  • Users with 1000+ karma rank each post on...
    • 1-10 scale for "how important is the content"
    • 1-10 scale for "how epistemically virtuous is this post"
    • Yes/No/Veto on "should this post be added to LessWrong canon?"
  • (In the 1-10 scale, 6+ means "I'd be happy to see this included in the 'best of 2018'" roundup, and 10 means 'this is the best I can imagine'")
  • "Yes, add this to canon" means that it hits some minimum threshold of epistemic virtue, as well as "this is something I think all LW readers should be at least passingly familiar with, or if they're not, the burden is on them to read up on it if it comes up in conversation."
• Rewards
  • The votes will all be publicly available. A few different aggregate statistics will be available, including the raw average, and probably some attempt at a "karma-weighted average."
  • The LW moderation team will put together a physical book, and online sequence, of the best posts, as well as the most valuable reviews of each post.
  • The LW team awards up to* $1000 in prizes to the best reviewers, and $3000 in prizes to the top post authors.
    • * this depends on whether we get reviews that seem to genuinely improve the epistemic landscape. Prizes for reviewers will be mostly moderator discretion (plus some inputs like "how much karma and engagement the review got")

And importantly:

Next Year

Even if we stuck to the above plan, I'd see it as more of an experiment than the definitive, longterm review mechanism. I expect we'd iterate a lot the following year.

But one thing I'm particularly interested in is how this builds over the longterm: next year (November 2020), while people would mostly be nominating posts from 2019, there should also be a process for submitting posts for "re-review", if there's been something like a replication crisis, or if a research direction that seemed promising now seems less-so, that's something we can revisit.

I know I'll go to programmer hell for asking this... but... does anyone have a link to a github repo that tried really hard to use jQuery to build their entire website, investing effort into doing some sort of weird 'jQuery based components' thing for maintainable, scalable development?

People tell me this can't be done without turning into terrifying spaghetti code but I dunno I feel sort of like the guy in this xkcd and I just want to know for sure.

I've lately been talking a lot [LW · GW] about doublecrux [LW · GW]. It seemed good to note some updates I'd also made over the past few months about debate.

For the past few years I've been sort of annoyed at debate because it seems like it doesn't lead people to change their opinions – instead, the entire debate framework seems more likely to prompt people to try to win, meanwhile treating arguments as soldiers and digging in their heels. I felt some frustration at the Hanson/Yudkowsky Foom Debate because huge amounts of digital ink were spilled, and neither party changed their mind much.

The counterpoint that's been pointed out to me lately is:

While debate may have that effect, it also produces lots of positive externalities. The process of Hanson and Yudkowsky spelling out their intuitions and arguments and preferred debate frameworks lead to a lot of interested facts and frameworks to chew on.

My review of the CFAR venue:

There is a song that the LessWrong team listened to awhile back, and then formed strong opinions about what was probably happening during the song, if the song had been featured in a movie.

(If you'd like to form your own unspoiled interpretation of the song, you may want to do that now)

...

So, it seemed to us that the song felt like... you (either a single person or small group of people) had been working on an intellectual project.

And people were willing to give the project the benefit of the doubt, a bit, but then you fucked it up in some way, and now nobody believed in you and you were questioning all of your underlying models and maybe also your sanity and worth as a human. (vaguely A Beautiful Mind like)

And then you retreat to your house where you're pretty alone, and it's raining outside. And the house is the sort of house a moderately wealthy but sometimes-alone intellectual might live, with a fair bit of space inside, white walls, whiteboards, small intellectual toys scattered about. A nice carpet.

And you're pacing around the house re-evaluating everything, and it's raining and the rain dapples on the windows and light scatters in on your old whiteboard diagrams that no longer seem to quite make sense.

And then you notice a small mental click of "maybe, if I applied this idea in this slightly different way, that might be promising". And you clear off a big chunk of whiteboard and start to work again, and then over a several day montage you start to figure out a new version of your idea that somehow works better this time and you get into a flow state and then you're just in this big beautiful empty house in the rain, rebuilding your idea, and this time it's pretty good and maybe will be the key to everything.

So anyway the LW team listened to this song a year+ ago and we now periodically listen to it and refer to it as "Building LessWrong in the Rain."

And, last week we had the LW Team Retreat, which was located at the new(ish) CFAR venue, and... a) it was raining all week, b) we basically all agreed that the interior of the CFAR venue looked almost exactly like how we had all imagined it. (Except, I at least had been imagining it a bit more like a Frank Lloyd Wright house, so that from the outside it looked more rectangular instead of a more traditional house/big-cottage or whatever)

...

The house interior is quite well designed. Every room had a purpose, and I'd be mulling about a given room thinking "gee, I sure wish I had X", and then I'd rotate 30º and then X would be, like, within arm's reach.

Most rooms had some manner of delightful thing, whether that be cute magnet puzzles or a weird glowing flower that looked like if I touched it it'd disappear and then I'd start glowing an either be able to jump higher or spit fireballs (I did not touch it)

Small complaints include:

a) the vacuum was quite big and heavy, which resulted in me switching to using a broom when I was cleaning up,

b) the refrigerator was like 500x more dangerous than any other fridge I ever encountered. Normally the amount of blood a refrigerator draws when I touch it gently is zero. The bottom of this fridge cut me 3 times, twice on my toes, once on my thumb while I was trying to clean it.

c) the first aid kit was in a black toolbox with the red "+" facing away from the visible area which made it a bit more counterintuitive to discover than most of the other things in the house.

Jargon Quest:

There's a kind of extensive double crux that I want a name for. It was inspired by Sarah's Naming the Nameless [LW · GW] post, where she mentions Double Cruxxing on aesthetics. You might call it "aesthetic double crux" but I think that might lead to miscommunication.

The idea is to resolve deep disagreements that underlie your entire framing (of the sort Duncan touches on in this post on Punch Buggy. That post is also a reasonable stab at an essay-form version of the thing I'm talking about).

There are a few things that are relevant here, not quite the same thing but clustered together:

• what counts as evidence?
• what counts as good?
• what counts as beautiful?

Each of them suggest a different name (epistemic double crux, values double crux, aesthetic double crux). Maybe a good common name is "Deep Double Crux" or "Framing Double Crux"

The main point is that when you hunker down for a deep double crux, you're expecting to spend a long while, and to try to tease some real subtle shit.

I liked the phrase Aesthetic Double Crux, suggested in the Naming the Nameless post, since it pointed at entire ways of thinking that had multiple facets, but seemed to orient most around what felt elegant and right. But the people who followed up on that focused most on the art interpretation, so it seemed ripe for misinterpretation.

(In the course of writing this I think I basically decided I liked Deep Double Crux best, but decided to leave the post up as a demonstration of thought process.)

We've been getting increasing amounts of spam, and occasionally dealing with Eugins. We have tools to delete them fairly easily, but sometimes they show up in large quantities and it's a bit annoying.

One possible solution is for everyone's first comment to need to be approved. A first stab at the implementation for this would be:

1) you post your comment as normal

2) it comes with a short tag saying "Thanks for joining less wrong! Since we get a fair bit of spam, first comments need to be approved by a moderator, which normally takes [N hours, whatever N turns out to be]. Sorry about that, we'll be with you soon!"

3) Comments awaiting approval show up on moderator's screen at the top of the page or something, with a one-click approval, so that it's very unlikely to be missed. I think this could get the wait time down pretty low even with a smallish number of moderators.

The main downside here is that people's first commenting experience wouldn't be as good. My intent with step #2 was to smooth it over as much as possible. (i.e. if it just said "comment awaiting approval", I'd think it be much worse).

I'm curious a) how bad people think this experience would be, and b) any other issues that seem relevant?

Recently watched Finding Dory. Rambly thoughts and thorough spoilers to follow.

I watched this because of a review by Ozy a long while ago, noting that the movie is about character with a mental disability that has major affects on her. And at various key moments in the movie, she finds herself lost and alone, her mental handicap playing a major role in her predicament. And in other movies they might given her some way to... willpower through her disability, or somehow gain a superpower that makes the disability irrelevant or something.

And instead, she has to think, and figure out what skills she does have she can use to resolve her predicaments. And that this was beautiful/poignant from the standpoint of getting to see representation of characters with disabilities getting to be protagonists in a very real way.

I think the movie generally lived up to that review (with some caveats, see below). But I was also found myself looking at it through the recent "Elephant" and "Mythic" lens. This is "Self-Identify-As-An-Elephant" and "Live In Mythic Mode" The Movie.

Dory has a "rider", maybe, but the rider can't form longterm memories, which makes it much less obvious as the seat-of-identity.

She seems to have the ability to form system-1 impressions that gradually accumulate into familiarity, useful intuitions that help her find her way around, and the ability to form friends after prolonged exposure to them. (My understanding is that this is not a realistic depiction of humans with short term memory loss, but since the movie is about a talking fish I'm willing to cut it some slack).

Her intuition-powers strain credibility a bit. I'm also willing to cut the movie some slack here from the standpoint of "in most Everett branches, Dory dies very young, and the interesting story worth telling was about the Dory who had just enough natural skill and luck to skate by early on, and then develop S1 associations useful enough to continue surviving.

(Aside: this movie has loads of places where Jesus Christ everyone should have just died, and for some reason this was the most stressful cinematic experience I've had in a living memory)

The thing I found most interesting about the movie is the scene where she's lost and alone and sad, and has to figure out what to do, and starts sharing her thought process outside, making it legible to both herself and the audience for the first time.

Looking at how facebook automatically shows particular subcomments in a thread, that have a lot of likes/reacts.

And then looking at how LW threads often become huge and unwieldy when there's 100 comments.

At first I was annoyed by that FB mechanic, but it may in fact be a necessary thing for sufficiently large threads, to make it easy to find the good parts.

Social failure I notice in myself: there'll be people at a party I don't know very well. My default assumption is "talk to them with 'feeler-outer-questions' to figure out what what they are interested in talking about". (i.e. "what do you do?"/"what's your thing?"/"what have you been thinking about lately?"/"what's something you value about as much as your right pinky?"/"What excites you?").

But this usually produces awkward, stilted conversation. (of the above, I think "what have you been thinking about lately?" produces the best outcomes most of the time)

Recently, I was having that experience, and ended up talking to a nearby person I knew better about a shared interested (videogames in this case). And then the people nearby who I didn't know as well were able to join in the conversation and it felt much more natural.

Part of the problem is that if there is no person-I-know nearby, I have to take a random guess at a thing to talk about that the person is interested in talking about.

In this case, I had various social cues that suggested video games would be a plausible discussion prompt, but not enough context to guess which sorts of games were interesting, and not enough shared background knowledge to launch into a discussion of a game I thought was interesting without worrying a bunch about "is this too much / wrong sort of conversation for them."

Not sure what lesson to learn, but seemed noteworthy.

Man I wish the "Battle of the Sexes" game theory thing had a less distracting name. 

Have you changed your mind about frames or aesthetics? 

I'm working on the next post in the "Keep Beliefs Cruxy and Frames Explicit" sequence. I'm not sure if it should be one or two posts. I'm also... noticing that honestly I'm not actually sure what actions to prescribe, and that this is more like a hypothesis and outlining of problems/desiderata.

Two plausible post titles

• Doublecruxing on Frame
• Keeping Frames Explicit

(I'm currently unsure whether aesthetics are best thought of as a type of frame, or a separate thing)

Honestly, I'm not sure whether I've successfully doublecruxed on a frame (i.e. reached convergence, or had both participants change their mind significantly, or come to any kind of shared). I've definitely singlecruxed on a frame (by which I mean, I have outlined the beliefs I'd have to change in order for my frame to change, and thought about hypothetical experiments you might run to check the beliefs)

So I think I'm switching to "think out loud about this for awhile before trying to write the final posts". I'm curious if other people have perspectives on it. 

Repeating the opening question: Have you changed your mind about frames or aesthetics? Have you engaged in major disagreements involving substantially different outlooks (ways of seeing, ways of thinking, or expectations of conversational context/goals) that went well?

Ben Kuhn's Why and How to Start a For Profit Company Serving Emerging Markets [EA · GW] is, in addition to being generally interesting, sort of cute for being two of the canonical Michael Vassar Questions rolled into one, while being nicely operationalized and clear.

("Move somewhere far away and stay their long enough to learn that social reality is arbitrary", and "start a small business and/or startup to a bunch about how pieces of the world fit together" being the two that come easiest to mind)

random anecdote in time management and life quality. Doesn't exactly have obvious life lesson

I use Freedom.to to block lots of sites (I block LessWrong during the morning hours of each day so that I can focus on coding LessWrong :P).

Once a upon a time, I blocked the gaming news website, Rock/Paper/Shotgun, because it was too distracting.

But a little while later I found that there was a necessary niche in my life of "thing that I haven't blocked on Freedom, that is sort of mindlessly entertaining enough that I can peruse it for awhile when I'm brain dead, but not so bottomlessly entertaining that it'll consume too much of my time." If I didn't have such a site, I would find one.

If I didn't have a standardized one, I would find one at random, and it'd be a bit of a crap shoot whether it was 5 minutes of eyes-glazed-skimming, or an hour of tabsplosioning.

The site I ended up settling on as my default blah-time was Kotaku, which was... basically RockPaperShotgun but worse. Gaming news that was sort of pointless and devoid of personality but juuuust over the threshold of "interesting enough that I actually wanted to read it."

Which I thought about a bit and then decided I reflectively endorsed.

Meanwhile, while I could access RockPaperShotgun in the evenings... I didn't, because, well, it wasn't that important and I was trying to cut back on videogames anyway.

Two years later... I dunno I found myself sort of thinking "you know, I wish I was passively gaining more interesting videogame news."

And... I unblocked RockPaperShotgun.

And I was surprised to notice

a) wow, most the content was actually interesting, tailored for the sorts of games I like, and written in a more entertaining voice

b) there were only a couple articles per day, whereas Kotaku used a vaguely facebook-like algorithm of "most of the articles are crap, but every few ones is a gem, which sort of gets me into a skinner-box that (I realized, in retrospect) probably had me reading _more_ than RPS did.

I frequently feel a desire to do "medium" upvotes. Specifically, I want tiers of upvote for:

1) minor social approval (equivalent to smiling at a person when they do something I think should receive _some_ signal of reward, in particular if I think they were following a nice incentive gradient, but where I don't think the thing they were doing was especially important.

2) strong social reward (where I want someone to be concretely rewarded for having done something hard, but I still don't think it's actually so important that it should rank highly in other people's attention

3) "this is worth your time and attention", where the signal is more about other people than the post/comment author.

(It's possible you could split these into two entirely different schemas, but I think that'd result in unnecessary UI complexity without commensurate benefit)