Habryka's Shortform Feed

I am confident, on the basis of private information I can't share, that Anthropic has asked at least some employees to sign similar non-disparagement agreements that are covered by non-disclosure agreements as OpenAI did. 

Or to put things into more plain terms: 

I am confident that Anthropic has offered at least one employee significant financial incentive to promise to never say anything bad about Anthropic, or anything that might negatively affect its business, and to never tell anyone about their commitment to do so.

I am not aware of Anthropic doing anything like withholding vested equity the way OpenAI did, though I think the effect on discourse is similarly bad.

I of course think this is quite sad and a bad thing for a leading AI capability company to do, especially one that bills itself on being held accountable by its employees and that claims to prioritize safety in its plans.

Reputation is lazily evaluated

When evaluating the reputation of your organization, community, or project, many people flock to surveys in which you ask randomly selected people what they think of your thing, or what their attitudes towards your organization, community or project are. 

If you do this, you will very reliably get back data that looks like people are indifferent to you and your projects, and your results will probably be dominated by extremely shallow things like "do the words in your name invoke positive or negative associations".

People largely only form opinions of you or your projects when they have some reason to do that, like trying to figure out whether to buy your product, or join your social movement, or vote for you in an election. You basically never care about what people think about you while engaging in activities completely unrelated to you, you care about what people will do when they have to take any action that is related to your goals. But the former is exactly what you are measuring in attitude surveys.

As an example of this (used here for illustrative purposes, and what caused me to form strong opinions on this, but not intended as the central point of this post): Many leaders in the Effective Altruism community ran various surveys after the collapse of FTX trying to understand what the reputation of "Effective Altruism" is. The results [EA · GW] were basically always the same: People mostly didn't know what EA was, and had vaguely positive associations with the term when asked. The people who had recently become familiar with it (which weren't that many) did lower their opinions of EA, but the vast majority of people did not (because they mostly didn't know what it was). 

As far as I can tell, these surveys left most EA leaders thinking that the reputational effects of FTX were limited. After all, most people never heard about EA in the context of FTX, and seemed to mostly have positive associations with the term, and the average like or dislike in surveys barely budged. In reflections at the time, conclusions looked like this [EA(p) · GW(p)]: 

1. The fact that most people don't really care much about EA is both a blessing and a curse. But either way, it's a fact of life; and even as we internally try to learn what lessons we can from FTX, we should keep in mind that people outside EA mostly can't be bothered to pay attention.
2. An incident rate in the single digit percents means that most community builders will have at least one example of someone raising FTX-related concerns—but our guess is that negative brand-related reactions are more likely to come from things like EA's perceived affiliation with tech or earning to give than FTX.
3. We have some uncertainty about how well these results generalize outside the sample populations. E.g. we have heard claims that people who work in policy were unusually spooked by FTX. That seems plausible to us, though Ben would guess that policy EAs similarly overestimate the extent to which people outside EA care about EA drama.

Or this [EA(p) · GW(p)]:

Yes, my best understanding is still that people mostly don't know what EA is, the small fraction that do mostly have a mildly positive opinion, and that neither of these points were affected much by FTX.^[1] 

This, I think, was an extremely costly mistake to make. Since then, practically all metrics of the EA community's health and growth have sharply declined [EA · GW], and the extremely large and negative reputational effects have become clear.

Most programmers are familiar with the idea of a "lazily evaluated variable" - a value that isn't computed until the exact moment you try to use it. Instead of calculating the value upfront, the system maintains just enough information to be able to calculate it when needed. If you never end up using that value, you never pay the computational cost of calculating it. Similarly, most people don't form meaningful opinions about organizations or projects until the moment they need to make a decision that involves that organization. Just as a lazy variable suddenly gets evaluated when you first try to read its value, people's real opinions about projects don't materialize until they're in a position where that opinion matters - like when deciding whether to donate, join, or support the project's initiatives.

Reputation is lazily evaluated. People conserve their mental energy, time, and social capital by not forming detailed opinions about things until those opinions become relevant to their decisions. When surveys try to force early evaluation of these "lazy" opinions, they get something more like a placeholder value than the actual opinion that would form in a real decision-making context.

This computation is not purely cognitive. As people encounter a product, organization or community that they are considering doing something with, they will ask their friends whether they have any opinions, perform online searches, and generally seek out information to help them with whatever decision they are facing. This is part of the reason for why this metaphorical computation is costly and put off until it's necessary.

So when you are trying to understand what people think of you, or how people's opinions of you are changing, pay much more attention to the attitudes of people who have recently put in the effort to learn about you, or were facing some decision related to you, and so are more representative of where people tend to end up at when they are in a similar position. These will be much better indicators of your actual latent reputation than what happens when you ask people on a survey. 

For the EA surveys, these indicators looked very bleak: 

"Results demonstrated that FTX had decreased satisfaction by 0.5-1 points on a 10-point scale within the EA community"

"Among those aware of EA, attitudes remain positive and actually maybe increased post-FTX —though they were lower (d = -1.5, with large uncertainty) among those who were additionally aware of FTX."

"Most respondents reported continuing to trust EA organizations, though over 30% said they had substantially lost trust in EA public figures or leadership."

If various people in EA had paid attention to these, instead of to the approximately meaningless placeholder variables that you get when you ask people what they think of you without actually getting them to perform the costly computation associated with forming an opinion of you, I think they would have made substantially better predictions.

↑ comment by Zach Stein-Perlman · 2024-12-08T03:23:43.626Z · LW(p) · GW(p)

Edit 2: after checking, I now believe the data strongly suggest FTX had a large negative effect on EA community metrics. (I still agree with Buck: "I don't like the fact that this essay is a mix of an insightful generic argument and a contentious specific empirical claim that I don't think you support strongly; it feels like the rhetorical strength of the former lends credence to the latter in a way that isn't very truth-tracking." And I disagree with habryka's claims that the effect of FTX is obvious.)

---

practically all metrics of the EA community's health and growth have sharply declined [EA · GW], and the extremely large and negative reputational effects have become clear.

I want more evidence on your claim that FTX had a major effect on EA reputation. Or: why do you believe it?

---

Edit: relevant thing habryka said that I didn't quote above:

For the EA surveys, these indicators looked very bleak: 

"Results demonstrated that FTX had decreased satisfaction by 0.5-1 points on a 10-point scale within the EA community"

"Among those aware of EA, attitudes remain positive and actually maybe increased post-FTX —though they were lower (d = -1.5, with large uncertainty) among those who were additionally aware of FTX."

"Most respondents reported continuing to trust EA organizations, though over 30% said they had substantially lost trust in EA public figures or leadership."

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-12-08T08:49:12.458Z · LW(p) · GW(p)

Practically all growth metrics are [EA · GW] down [EA · GW] (and have indeed turned negative on most measures), a substantial fraction [EA(p) · GW(p)] of core contributors are distancing themselves from the EA affiliation, surveys among EA community builders report EA-affiliation as a major recurring obstacle^[1], and many of the leaders who previously thought it wasn't a big deal now concede that it was/is a huge deal.

Also, informally, recruiting for things like EA Fund managers, or getting funding for EA Funds has become substantially harder. EA leadership positions appear to be filled by less competent people, and in most conversations I have with various people who have been around for a while, people seem to both express much less personal excitement or interest in identifying or championing anything EA-related, and report the same for most other people.

Related to the concepts in my essay, when measured the reputational differential also seem to reliably point towards people updating negatively towards EA as they learn more about EA (which shows up in the quotes you mentioned, and which more recently shows up in the latest Pulse survey [EA · GW], though I mostly consider that survey uninformative for roughly the reasons outlined in this post).

 

1. ^{^}

   As reported to me by someone I trust working in the space recently. I don't have a link at hand.

Replies from: angelinahli, lahwran, angelinahli

↑ comment by angelinahli · 2024-12-10T19:40:05.137Z · LW(p) · GW(p)

Hey! Sorry for the silence, I was feeling a bit stressed by this whole thread, and so I wanted to step away and think about this before responding. I’ve decided to revert the dashboard back to its original state & have republished the stale data. I did some quick/light data checks but prioritised getting this out fast. For transparency: I’ve also added stronger context warnings and I took down the form to access our raw data in sheet form but intend to add it back once we’ve fixed the data. It’s still on our stack to Actually Fix this at some point but we’re still figuring out the timing on that.

On reflection, I think I probably made the wrong call here (although I still feel a bit sad / misunderstood but 🤷🏻‍♀️). It was a unilateral + lightly held call I made in the middle of my work day — like truly I spent 5 min deciding this & maybe another ~15 updating the thing / leaving a comment. I think if I had a better model for what people wanted from the data, I would have made a different call. I’ve updated on “huh, people really care about not deleting data from the internet!” — although I get that the reaction here might be especially strong because it’s about CEA (vs the general case). Sorry, I made a mistake.

Future facing thoughts: I generally hold myself to a higher standard for accuracy when putting data on the internet, but I also do value not bottlenecking people in investigating questions that feel important to me (e.g. qs about EA growth rates), so to be clear I’m prioritizing the latter goal right now. I still in general stand by, “what even is the point of my job if I don’t stand by the data I communicate to others?” :) I want people to be able to trust that the work they see me put out in the world has been red-teamed & critiqued before publication.

Although I’m sad this caused an unintended kerfuffle, it’s a positive update for me that “huh wow, people actually care a lot that this project is kept alive!”. This honestly wasn’t obvious to me — this is a low traffic website that I worked on a while ago, and don’t hear about much. Oli says somewhere that he’s seen it linked to “many other times” in the past year, but TBH no one has flagged that to me (I’ve been busy with other projects). I’m still glad that we made this thing in the first place and am glad people find the data interesting / valuable (for general CEA transparency reasons, as an input to these broader questions about EA, etc.). I’ll probably prioritize maintenance on this higher in the future.

Now that the data is back up I’m going to go back to ignoring this thread!

Replies from: angelinahli, habryka4

↑ comment by angelinahli · 2024-12-10T23:13:52.451Z · LW(p) · GW(p)

[musing] Actually another mistake here which I wish I just said in the first comment: I didn't have a strong enough TAP for, if someone says a negative thing about your org (or something that could be interpreted negatively), you should have a high bar for not taking away data (meaning more broadly than numbers) that they were using to form that perception, even if you think the data is wrong for reasons they're not tracking. You can like, try and clarify the misconception (ideally, given time & energy constraints etc.), and you can try harder to avoid putting wrong things out there, but don't just take it away -- it's not on to reader to treat you charitably and it kind of doesn't matter what your motives were.

I think I mostly agree with something like that / I do think people should hold orgs to high standards here. I didn't pay enough attention to this and regret it. Sorry! (I'm back to ignoring this thread lol but just felt like sharing a reflection 🤷🏻‍♀️)

↑ comment by habryka (habryka4) · 2024-12-10T19:53:37.946Z · LW(p) · GW(p)

Thank you! I appreciate the quick oops here, and agree it was a mistake (but fixing it as quickly as you did I think basically made up for all the costs, and I greatly appreciate it).

Just to clarify, I don't want to make a strong statement that it's worth updating the data and maintaining the dashboard. By my lights it would be good enough to just have a static snapshot of it forever. The thing that seemed so costly to me was breaking old links and getting rid of data that you did think was correct. 

Thanks again!

↑ comment by the gears to ascension (lahwran) · 2024-12-08T18:40:06.790Z · LW(p) · GW(p)

I suspect fixing this would need to involve creating something new which doesn't have the structural problems in EA which produced this, and would involve talking to people who are non-sensationalist EA detractors but who are involved with similarly motivated projects. I'd start here and skip past the ones that are arguing "EA good" to find the ones that are "EA bad, because [list of reasons ea principles are good, and implication that EA is bad because it fails at its stated principles]"

I suspect, even without seeking that out, the spirit of EA that made it ever partly good has already and will further metastasize into genpop.

↑ comment by angelinahli · 2024-12-09T01:20:48.174Z · LW(p) · GW(p)

Hi! A quick note: I created the CEA Dashboard which is the 2nd link you reference. The data here hadn’t been updated since August 2024, and so was quite out of date at the time of your comment. I've now taken this dashboard down, since I think it's overall more confusing than helpful for grokking the state of CEA's work. We still intend to come back and update it within a few months.

Just to be clear on why / what’s going on:

• I stopped updating the dashboard in August because I started getting busy with some other projects, and my manager & I decided to deprioritize this. (There are some manual steps needed to keep the data live).
• I’ve now seen several people refer to that dashboard as a reference for how CEA is doing in ways I think are pretty misleading.
• We (CEA) still intend to come back and fix this, and this is a good nudge to prioritize it.

Thanks!

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-12-09T01:27:27.221Z · LW(p) · GW(p)

Oh, huh, that seems very sad. Why would you do that? Please leave up the data that we have. I think it's generally bad form to break links that people relied on. The data was accurate as far as I can tell until August 2024, and you linked to it yourself a bunch over the years, don't just break all of those links.

I am pretty up-to-date with other EA metrics and I don't really see how this would be misleading. You had a disclaimer at the top that I think gave all the relevant context. Let people make their own inferences, or add more context, but please don't just take things down.

Unfortunately, archive.org doesn't seem to have worked for that URL, so we can't even rely on that to show the relevant data trends.

Edit: I'll be honest, after thinking about it for longer, the only reason I can think of why you would take down the data is because it makes CEA and EA look less on an upwards trajectory. But this seems so crazy. How can I trust data coming out of CEA if you have a policy of retracting data that doesn't align with the story you want to tell about CEA and EA? The whole point of sharing raw data is to allow other people to come to their own conclusions. This really seems like such a dumb move from a trust perspective.

Replies from: Benito, angelinahli, angelinahli, angelinahli

↑ comment by Ben Pace (Benito) · 2024-12-09T02:51:31.455Z · LW(p) · GW(p)

I also believe that the data making EA+CEA looks bad is the causal reason why it was taken down. However, I want to add some slight nuance.

I want to contrast a model whereby Angelina Li did this while explicitly trying to stop CEA from looking bad, versus a model whereby she senses that something bad might be happening, she might be held responsible (e.g. within her organization / community), and is executing a move that she's learned is 'responsible' from the culture around her.

I think many people have learned to believe the reasoning step "If people believe bad things about my team I think are mistaken with the information I've given them, then I am responsible for not misinforming people, so I should take the information away, because it is irresponsible to cause people to have false beliefs". I think many well-intentioned people will say something like this, and that this is probably because of two reasons (borrowing from The Gervais Principle):

1. This is a useful argument for powerful sociopaths to use when they are trying to suppress negative information about themselves.
2. The clueless people below them in the hierarchy need to rationalize why they are following the orders of the sociopaths to prevent people from accessing information. The idea that they are 'acting responsibly' is much more palatable than the idea that they are trying to control people, so they willingly spread it and act in accordance with it.

A broader model I have is that there are many such inference-steps floating around the culture that well-intentioned people can accept as received wisdom, and they got there because sociopaths needed a cover for their bad behavior and the clueless people wanted reasons to feel good about their behavior; and that each of these adversarially optimized inference-steps need to be fought and destroyed.

Replies from: sarahconstantin, Kaj_Sotala

↑ comment by sarahconstantin · 2024-12-09T20:37:47.958Z · LW(p) · GW(p)

I agree, and I am a bit disturbed that it needs to be said.

At normal, non-EA organizations -- and not only particularly villainous ones, either! -- it is understood that you need to avoid sharing any information that reflects poorly on the organization, unless it's required by law or contract or something. The purpose of public-facing communications is to burnish the org's reputation. This is so obvious that they do not actually spell it out to employees.

Of COURSE any organization that has recently taken down unflattering information is doing it to maintain its reputation. 

I'm sorry, but this is how "our people" get taken for a ride. Be more cynical, including about people you like.

↑ comment by Kaj_Sotala · 2024-12-10T12:02:25.670Z · LW(p) · GW(p)

I think many people have learned to believe the reasoning step "If people believe bad things about my team I think are mistaken with the information I've given them, then I am responsible for not misinforming people, so I should take the information away, because it is irresponsible to cause people to have false beliefs". I think many well-intentioned people will say something like this, and that this is probably because of two reasons (borrowing from The Gervais Principle):

(Comment not specific to the particulars of this issue but noted as a general policy:) I think that as a general rule, if you are hypothesizing reasons for why somebody might say a thing, you should always also include the hypothesis that "people say a thing because they actually believe in it". This is especially so if you are hypothesizing bad reasons for why people might say it. 

It's very annoying when someone hypothesizes various psychological reasons for your behavior and beliefs but never even considers as a possibility [LW · GW] the idea that maybe you might have good reasons to believe in it. Compare e.g. "rationalists seem to believe that superintelligence is imminent; I think this is probably because that lets them avoid taking responsibility about their current problems if AI will make those irrelevant anyway, or possibly because they come from religious backgrounds and can't get over their subconscious longing for a god-like figure".

Replies from: Benito, habryka4

↑ comment by Ben Pace (Benito) · 2024-12-10T17:44:47.013Z · LW(p) · GW(p)

I feel more responsibility to be the person holding/tracking the earnest hypothesis in a 1-1 context, or if I am the only one speaking; in larger group contexts I tend to mostly ask "Is there a hypothesis here that isn't or likely won't be tracked unless I speak up" and then I mostly focus on adding hypotheses to track (or adding evidence that nobody else is adding).

↑ comment by habryka (habryka4) · 2024-12-10T17:45:56.781Z · LW(p) · GW(p)

(Did Ben indicate he didn’t consider it? My guess is he considered it, but thinks it’s not that likely and doesn’t have amazingly interesting things to say on it.

I think having a norm of explicitly saying “I considered whether you were saying the truth but I don’t believe it” seems like an OK norm, but not obviously a great one. In this case Ben also responded to a comment of mine which already said this, and so I really don’t see a reason for repeating it.)

Replies from: Kaj_Sotala

↑ comment by Kaj_Sotala · 2024-12-17T20:02:53.127Z · LW(p) · GW(p)

(I read

I think many well-intentioned people will say something like this, and that this is probably because of two reasons 

as implying that the list of reasons is considered to exhaustive, such that any reasons besides those two have negligible probability.)

Replies from: Benito

↑ comment by Ben Pace (Benito) · 2024-12-18T07:47:37.032Z · LW(p) · GW(p)

I gave my strongest hypothesis for why it looks to me that many many people believe it's responsible to take down information that makes your org look bad. I don't think alternative stories have negligible probability, nor does what I wrote imply that, though it is logically consistent with that.

There are many anti-informative behaviors that are widespread for which people do for poor reasons, like saying that their spouse is the best spouse in the world, or telling customers that their business is the best business in the industry, or saying exclusively glowing things about people in reference letters, that are best explained by the incentives on the person to present themselves in the best light; at the same time, it is respectful to a person, while in dialogue with them, to keep a track of the version of them who is trying their best to have true beliefs and honestly inform others around them, in order to help them become that person (and notice the delta between their current behavior and what they hopefully aspire to). 

Seeing orgs in the self-identified-EA space take down information that makes them look bad is (to me) not that dissimilar to the other things I listed.

I think it's good to discuss norms about how appropriate it is to bring up cynical hypotheses about someone during a discussion in which they're present. In this case I think raising this hypothesis was worthwhile it for the discussion, and I didn't cut off any way [LW · GW] for the person in question to continue to show themselves to be broadly acting in good faith, so I think it went fine. Li replied to Habryka, and left a thoughtful pair of comments retracting and apologizing, which reflected well on them in my eyes.

Replies from: Kaj_Sotala

↑ comment by Kaj_Sotala · 2024-12-18T08:13:54.110Z · LW(p) · GW(p)

I don't think alternative stories have negligible probability

Okay! Good clarification.

I think it's good to discuss norms about how appropriate it is to bring up cynical hypotheses about someone during a discussion in which they're present.

To clarify, my comment wasn't specific to the case where the person is present. There are obvious reasons why the consideration should get extra weight when the person is present, but there's also a reason to give it extra weight if none of the people discussed are present - namely that they won't be able to correct any incorrect claims if they're not around.

so I think it went fine

Agree.

(As I mentioned in the original comment, the point I made was not specific to the details of this case, but noted as a general policy. But yes, in this specific case it went fine.)

↑ comment by angelinahli · 2024-12-09T01:58:31.354Z · LW(p) · GW(p)

Quick thoughts on this:

• “The data was accurate as far as I can tell until August 2024”
  • I’ve heard a few reports over the last few weeks that made me unsure whether the pre-Aug data was actually correct. I haven’t had time to dig into this.
  • In one case (e.g. with the EA.org data) we have a known problem with the historical data that I haven’t had time to fix, that probably means the reported downward trend in views is misleading. Again I haven’t had time to scope the magnitude of this etc.
• I’m going to check internally to see if we can just get this back up in a week or two (It was already high on our stack, so this just nudges up timelines a bit). I will update this thread once I have a plan to share.

I’m probably going to drop responding to “was this a bad call” and prioritize “just get the dashboard back up soon”.

↑ comment by angelinahli · 2024-12-10T19:43:28.677Z · LW(p) · GW(p)

More thoughts here [LW · GW], but TL;DR I’ve decided to revert the dashboard back to its original state & have republished the stale data. (Just flagging for readers who wanted to dig into the metrics.)

↑ comment by angelinahli · 2024-12-09T03:01:43.213Z · LW(p) · GW(p)

Hey! I just saw your edited text and wanted to jot down a response:

Edit: I'll be honest, after thinking about it for longer, the only reason I can think of why you would take down the data is because it makes CEA and EA look less on an upwards trajectory. But this seems so crazy. How can I trust data coming out of CEA if you have a policy of retracting data that doesn't align with the story you want to tell about CEA and EA? The whole point of sharing raw data is to allow other people to come to their own conclusions. This really seems like such a dumb move from a trust perspective.

I'm sorry this feels bad to you. I care about being truth seeking and care about the empirical question of "what's happening with EA growth?". Part of my motivation in getting this dashboard published [EA · GW] in the first place was to contribute to the epistemic commons on this question.

I also disagree that CEA retracts data that doesn't align with "the right story on growth”. E.g. here's a post [EA · GW] I wrote in mid 2023 where the bottom line conclusion was that growth in meta EA projects was down in 2023 v 2022. It also publishes data on several cases where CEA programs grew slower in 2023 or shrank. TBH I also think of this as CEA contributing to the epistemic commons here — it took us a long time to coordinate and then get permission from people to publish this. And I’m glad we did it!

On the specific call here, I'm not really sure what else to tell you re: my motivations other than what I've already said. I'm going to commit to not responding further to protect my attention, but I thought I'd respond at least once :)

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-12-09T03:42:42.683Z · LW(p) · GW(p)

I would currently be quite surprised if you had taken the same action if I was instead making an inference that positively reflects on CEA or EA. I might of course be wrong, but you did do it right after I wrote something critical of EA and CEA, and did not do it the many other times it was linked in the past year. Sadly your institution has a long history of being pretty shady with data and public comms this way, and so my priors are not very positively inclined.

I continue to think that it would make sense to at least leave the data up that CEA did feel comfortable linking in the last 1.5 years. By my norms invalidating links like this, especially if the underlying page happens to be unscrapeable by the internet archive, is really very bad form.

I did really appreciate your mid 2023 post!

Thoughts on integrity and accountability

[Epistemic Status: Early draft version of a post I hope to publish eventually. Strongly interested in feedback and critiques, since I feel quite fuzzy about a lot of this]

When I started studying rationality and philosophy, I had the perspective that people who were in positions of power and influence should primarily focus on how to make good decisions in general and that we should generally give power to people who have demonstrated a good track record of general rationality. I also thought of power as this mostly unconstrained resource, similar to having money in your bank account, and that we should make sure to primarily allocate power to the people who are good at thinking and making decisions.

That picture has changed a lot over the years. While I think there is still a lot of value in the idea of "philosopher kings", I've made a variety of updates that significantly changed my relationship to allocating power in this way:

• I have come to believe that people's ability to come to correct opinions about important questions is in large part a result of whether their social and monetary incentives reward them when they have accurate models in a specific domain. This means a person can have extremely good opinions in one domain of reality, because they are subject to good incentives, while having highly inaccurate models in a large variety of other domains in which their incentives are not well optimized.
• People's rationality is much more defined by their ability to maneuver themselves into environments in which their external incentives align with their goals, than by their ability to have correct opinions while being subject to incentives they don't endorse. This is a tractable intervention and so the best people will be able to have vastly more accurate beliefs than the average person, but it means that "having accurate beliefs in one domain" doesn't straightforwardly generalize to "will have accurate beliefs in other domains".
  
  One is strongly predictive of the other, and that’s in part due to general thinking skills and broad cognitive ability. But another major piece of the puzzle is the person's ability to build and seek out environments with good incentive structures.
• Everyone is highly irrational in their beliefs about at least some aspects of reality, and positions of power in particular tend to encourage strong incentives that don't tend to be optimally aligned with the truth. This means that highly competent people in positions of power often have less accurate beliefs than much less competent people who are not in positions of power.
• The design of systems that hold people who have power and influence accountable in a way that aligns their interests with both forming accurate beliefs and the interests of humanity at large is a really important problem, and is a major determinant of the overall quality of the decision-making ability of a community. General rationality training helps, but for collective decision making the creation of accountability systems, the tracking of outcome metrics and the design of incentives is at least as big of a factor as the degree to which the individual members of the community are able to come to accurate beliefs on their own.

A lot of these updates have also shaped my thinking while working at CEA, LessWrong and the LTF-Fund over the past 4 years. I've been in various positions of power, and have interacted with many people who had lots of power over the EA and Rationality communities, and I've become a lot more convinced that there is a lot of low-hanging fruit and important experimentation to be done to ensure better levels of accountability and incentive-design for the institutions that guide our community.

I also generally have broadly libertarian intuitions, and a lot of my ideas about how to build functional organizations are based on a more start-up like approach that is favored here in Silicon Valley. Initially these intuitions seemed at conflict with the intuitions for more emphasis on accountability structures, with broken legal systems, ad-hoc legislation, dysfunctional boards and dysfunctional institutions all coming to mind immediately as accountability-systems run wild. I've since then reconciled my thoughts on these topics a good bit.

Integrity

Somewhat surprisingly, "integrity" has not been much discussed as a concept handle on LessWrong. But I've found it to be a pretty valuable virtue to meditate and reflect on.

I think of integrity as a more advanced form of honesty – when I say “integrity” I mean “acting in accordance with your stated beliefs.” Where honesty is the commitment to not speak direct falsehoods, integrity is the commitment to speak truths that actually ring true to yourself, not ones that are just abstractly defensible to other people. It is also a commitment to act on the truths that you do believe, and to communicate to others what your true beliefs are.

Integrity can be a double-edged sword. While it is good to judge people by the standards they expressed, it is also a surefire way to make people overly hesitant to update. If you get punished every time you change your mind because your new actions are now incongruent with the principles you explained to others before you changed your mind, then you are likely to stick with your principles for far longer than you would otherwise, even when evidence against your position is mounting.

The great benefit that I experienced from thinking of integrity as a virtue, is that it encourages me to build accurate models of my own mind and motivations. I can only act in line with ethical principles that are actually related to the real motivators of my actions. If I pretend to hold ethical principles that do not correspond to my motivators, then sooner or later my actions will diverge from my principles. I've come to think of a key part of integrity being the art of making accurate predictions about my own actions and communicating those as clearly as possible.

There are two natural ways to ensure that your stated principles are in line with your actions. You either adjust your stated principles until they match up with your actions, or you adjust your behavior to be in line with your stated principles. Both of those can backfire, and both of those can have significant positive effects.

Who Should You Be Accountable To?

In the context of incentive design, I find thinking about integrity valuable because it feels to me like the natural complement to accountability. The purpose of accountability is to ensure that you do what you say you are going to do, and integrity is the corresponding virtue of holding up well under high levels of accountability.

Highlighting accountability as a variable also highlights one of the biggest error modes of accountability and integrity – choosing too broad of an audience to hold yourself accountable to.

There is tradeoff between the size of the group that you are being held accountable by, and the complexity of the ethical principles you can act under. Too large of an audience, and you will be held accountable by the lowest common denominator of your values, which will rarely align well with what you actually think is moral (if you've done any kind of real reflection on moral principles).

Too small or too memetically close of an audience, and you risk not enough people paying attention to what you do, to actually help you notice inconsistencies in your stated beliefs and actions. The smaller the group that is holding you accountable is, the smaller your inner circle of trust, which reduces the amount of total resources that can be coordinated under your shared principles.

I think a major mistake that even many well-intentioned organizations make is to try to be held accountable by some vague conception of "the public". As they make public statements, someone in the public will misunderstand them, causing a spiral of less communication, resulting in more misunderstandings, resulting in even less communication, culminating into an organization that is completely opaque about any of its actions and intentions, with the only communication being filtered by a PR department that has little interest in the observers acquiring any beliefs that resemble reality.

I think a generally better setup is to choose a much smaller group of people that you trust to evaluate your actions very closely, and ideally do so in a way that is itself transparent to a broader audience. Common versions of this are auditors, as well as nonprofit boards that try to ensure the integrity of an organization.

This is all part of a broader reflection on trying to create good incentives for myself and the LessWrong team. I will probably follow this up with a post that more concretely summarizes my thoughts on how all of this applies to LessWrong concretely.

In summary:

• One lens to view integrity through is as an advanced form of honesty – “acting in accordance with your stated beliefs.”
• To improve integrity, you can either try to bring your actions in line with your stated beliefs, or your stated beliefs in line with your actions, or reworking both at the same time. These options all have failure modes, but potential benefits.
• People with power sometimes have incentives that systematically warp their ability to form accurate beliefs, and (correspondingly) to act with integrity.
• An important tool for maintaining integrity (in general, and in particular as you gain power) is to carefully think about what social environment and incentive structures you want for yourself.
• Choose carefully who, and how many people, you are accountable to:
• Too many people, and you are limited in the complexity of the beliefs and actions that you can justify.
• Too few people, too similar to you, and you won’t have enough opportunities for people to notice and point out what you’re doing wrong. You may also not end up with a strong enough coalition aligned with your principles to accomplish your goals.

Welp, I guess my life is comic sans today. The EA Forum snuck some code into our deployment bundle for my account in-particular, lol: https://github.com/ForumMagnum/ForumMagnum/pull/9042/commits/ad99a147824584ea64b5a1d0f01e3f2aa728f83a

A thing that I've been thinking about for a while has been to somehow make LessWrong into something that could give rise to more personal-wikis and wiki-like content. Gwern's writing has a very different structure and quality to it than the posts on LW, with the key components being that they get updated regularly and serve as more stable references for some concept, as opposed to a post which is usually anchored in a specific point in time. 

We have a pretty good wiki system for our tags, but never really allowed people to just make their personal wiki pages, mostly because there isn't really any place to find them. We could list the wiki pages you created on your profile, but that doesn't really seem like it would allocate attention to them successfully.

I was thinking about this more recently as Arbital is going through another round of slowly rotting away (its search currently being broken and this being very hard to fix due to annoying Google Apps Engine restrictions) and thinking about importing all the Arbital content into LessWrong. That might be a natural time to do a final push to enable people to write more wiki-like content on the site.

Does anyone have any takes on the two Boeing whistleblowers who died under somewhat suspicious circumstances? I haven't followed this in detail, and my guess is it is basically just random chance, but it sure would be a huge deal if a publicly traded company now was performing assassinations of U.S. citizens. 

Curious whether anyone has looked into this, or has thought much about baseline risk of assassinations or other forms of violence from economic actors.

I have updated the OpenAI Email Archives [LW · GW] to now also include all emails that OpenAI has published in their March 2024 and December 2024 blogposts!

I continue to think reading through these is quite valuable, and even more interesting with the March 2024 and December 2024 emails included.

AND THE GAME [LW · GW] IS CLEAR. WRONGANITY SHALL SURVIVE ANOTHER DAY. GLORY TO EAST WRONG. GLORY TO WEST WRONG. GLORY TO ALL.

Btw less.online is happening. LW post and frontpage banner probably going up Sunday or early next week. 

Thoughts on voting as approve/disapprove and agree/disagree:

One of the things that I am most uncomfortable with in the current LessWrong voting system is how often I feel conflicted between upvoting something because I want to encourage the author to write more comments like it, and downvoting something because I think the argument that the author makes is importantly flawed and I don't want other readers to walk away with a misunderstanding about the world.

I think this effect quite strongly limits certain forms of intellectual diversity on LessWrong, because many people will only upvote your comment if they agree with it, and downvote comments they disagree with, and this means that arguments supporting people's existing conclusions have a strong advantage in the current karma system. Whereas the most valuable comments are likely ones that challenge existing beliefs and that are rigorously arguing for unpopular positions.

A feature that has been suggested many times over the years is to split voting into two dimensions. One dimension being "agree/disagree" and the other being "approve/disapprove". Only the "approve/disapprove" dimension matters for karma and sorting, but both are displayed relatively prominently on the comment (the agree/disagree dimension on the the bottom, the approve/disapprove dimension at the top). I think this has some valuable things going for it, and in particular would make me likely to upvote more comments because I could simultaneously signal that while I think a comment was good, I don't agree with it.

An alternative way of doing this that Ray has talked about is the introduction of short reactions that users can click at the bottom of a comment, two of the most prominently displayed ones would be "agree/disagree". Reactions would be by default non-anonymous and so would serve more as a form of shorthand comment instead of an alternative voting system. Here is an example of how that kind of UI might look:

I don't know precisely what the selection menu for choosing reactions should look like. My guess is we want to have a relatively broad selection, maybe even with the ability to type something custom into it (obviously limiting the character count significantly).

I am most worried that this will drastically increase the clutter of comment threads and make things a lot harder to parse. In particular if the order of the reacts is different on each comment, since then there is no reliable way of scanning for the different kinds of information.

A way to improve on this might be by having small icons for the most frequent reacts, but that then introduces a pretty sharp learning curve into the site, and it's always a pain to find icons for really abstract concepts like "agree/disagree".

I think I am currently coming around to the idea of reactions being a good way to handle approve/disapprove, but also think it might make more sense to introduce more as a new kind of vote that has more top-level support than simple reacts would have. Though in the most likely case this whole dimension will turn out to be too complicated and not worth the complexity costs (as 90% of feature ideas do).

After many years of pain, LessWrong now has fixed kerning and a consistent sans-serif font on all operating systems. You have probably seen terrible kerning like this over the last few years on LW: 

It really really looks like there is no space between the first comma and "Ash". This is because Apple has been shipping an extremely outdated version of Gill Sans with terribly broken kerning, often basically stripping spaces completely. We have gotten many complaints about this over the years.

But it is now finally fixed. However, changing fonts likely has many downstream effects on various layout things being broken in small ways. If you see any buttons or text misaligned, let us know, and we'll fix it. We already cleaned up a lot, but I am expecting a long tail of small fixes.

↑ comment by cubefox · 2024-10-29T09:54:07.760Z · LW(p) · GW(p)

Did the font size in comments change? It does seem quite small now...

Replies from: Kaj_Sotala

↑ comment by Kaj_Sotala · 2024-10-29T10:22:39.407Z · LW(p) · GW(p)

Yeah it feels uncomfortably small to read to me now

Replies from: Viliam

↑ comment by Viliam · 2024-10-29T11:26:38.744Z · LW(p) · GW(p)

Something felt uncomfortable today, but I can't put my finger on it. Just a general feeling as if the letters are less sharp or less clearly separated or something like that.

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-10-29T15:37:37.360Z · LW(p) · GW(p)

Guys, for this specific case you really have to say what OS you are using. Otherwise you might be totally talking past each other.

(Font-size didn't change on any OS, but the font itself changed from Calibri to Gill Sans on Windows. Gill Sans has a slightly smaller x-height so probably looks a bit smaller.)

Replies from: Kaj_Sotala, MondSemmel, cubefox, nathan-helm-burger, green_leaf, DanielFilan

↑ comment by Kaj_Sotala · 2024-10-29T19:47:21.456Z · LW(p) · GW(p)

On Windows the font feels actively unpleasant right away, on Android it's not quite as bad but feels like I might develop eyestrain if I read comments for a longer time.

↑ comment by MondSemmel · 2024-10-29T19:09:00.894Z · LW(p) · GW(p)

Up to a few days ago, the comments looked good on desktop Firefox, Windows 11, zoom level 150%. Now I find them uncomfortable to look at.

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-10-29T20:54:41.858Z · LW(p) · GW(p)

Plausible we might want to revert to Calibri on Windows, but I would like to make Gill Sans work. Having different font metrics on different devices makes a lot of detailed layout work much more annoying.

Curious if you can say more about the nature of discomfort. Also curious whether fellow font optimizer @Said Achmiz [LW · GW] has any takes, since he has been helpful here in the past, especially on the "making things render well on Windows" side.

Replies from: SaidAchmiz, MondSemmel

↑ comment by Said Achmiz (SaidAchmiz) · 2024-10-30T06:34:08.373Z · LW(p) · GW(p)

Well, let’s see. Calibri is a humanist sans; Gill Sans is technically also humanist, but more more geometric in design. Geometric sans fonts tend to be less readable when used for body text.

Gill Sans has a lower x-height than Calibri. That (obviously) is the cause of all the “the new font looks smaller” comments.

(A side-by-side comparison of the fonts, for anyone curious, although note that this is Gill Sans MT Pro, not Gill Sans Nova, so the weight [i.e., stroke thickness] will be a bit different than the version that LW now uses.)

Now, as far as font rendering goes… I just looked at the site on my Windows box (adjusting the font stack CSS value to see Gill Sans Nova again, since I see you guys tweaked it to give Calibri priority)… yikes. Yeah, that’s not rendering well at all. Definitely more blurry than Calibri. Maybe something to do with the hinting, I don’t know. (Not really surprising, since Calibri was designed from the beginning to look good on Windows.) And I’ve got a hi-DPI monitor on my Windows machine…

Interestingly, the older version of Gill Sans (seen in the demo on my wiki, linked above) doesn’t have this problem; it renders crisply on Windows. (Note that this is not the flawed, broken-kerning version of the font that comes with Macs!)

I also notice that the comment font size is set to… 15.08px. Seems weird? Bumping it up to 16px improves things a bit, although it’s still not amazing.

If you can switch to the older (but not broken) version of Gill Sans, that’d be my recommendation.

If you can’t… then one option might be to check out one of the many similar fonts to see if perhaps one of them renders better on Windows while still having matching metrics.

Replies from: kave, habryka4

↑ comment by kave · 2024-10-31T01:11:07.069Z · LW(p) · GW(p)

One sad thing about older versions of Gill Sans: Il1 all look the same. Nova at least distinguishes the 1.

IMO, we should probably move towards system fonts, though I would like to choose something that preserves character a little more.

↑ comment by habryka (habryka4) · 2024-10-30T15:45:38.673Z · LW(p) · GW(p)

Interesting, thanks! Checking an older version of Gill Sans probably wouldn't have been something would have thought to do, so your help is greatly appreciated. 

I'll experiment some with getting Gill Sans MT Pro.

↑ comment by MondSemmel · 2024-10-29T21:23:26.268Z · LW(p) · GW(p)

Comparing with this Internet Archive snapshot from Oct 6, both at 150% zoom, both in desktop Firefox in Windows 11: Comparison screenshot, annotated

• The new font seems... thicker, somehow? There's a kind of eye test you do at the optician where they ask you if the letters seem sharper or just thicker (or something), and this font reminds me of that. Like something is wrong with the prescription of my glasses.
• The new font also feels noticeably smaller in some way. Maybe it's the letter height? I lack the vocabulary to properly describe this. At the very least, the question mark looks noticeably weird. And e.g. in "t" and "p", the upper and lower parts of the respective letter are weirdly tiny.
• Incidentally there were also some other differences in the shape and alignment of UI elements (see the annotated screenshot).

Replies from: MondSemmel

↑ comment by MondSemmel · 2024-10-29T21:30:55.973Z · LW(p) · GW(p)

Oh, and the hover tooltip for the agreement votes is now bugged; IIRC hovering over the agreement vote number is supposed to give you some extra info just like with karma, but now it just explains what agreement votes are.

↑ comment by cubefox · 2024-10-29T17:59:56.359Z · LW(p) · GW(p)

Guys, for this specific case you really have to say what OS you are using. Otherwise you might be totally talking past each other.

(Font-size didn't change on any OS, but the font itself changed from Calibri to Gill Sans on Windows. Gill Sans has a slightly smaller x-height so probably looks a bit smaller.)

I tested it on Android, it's the same for both Firefox and Chrome. The font looks significantly smaller than the old font, likely due to the smaller x-height you mentioned. Could the font size of the comments be increased a bit so that it appears visually about as large as the old one? Currently I find it too small to read comfortably. (Subjective font size is often different from the standard font size measure. E.g. Verdana appears a lot larger than Arial at the same standard "size".)

(A general note: some people are short sighted and wear glasses, and the more short-sighted you are, the stronger the glasses contract your field of view to a smaller area. So things that may appear as an acceptable size for people who aren't particularly short-sighted, may appear too small for more short-sighted people.)

↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-10-29T17:09:38.508Z · LW(p) · GW(p)

Yeah, using Firefox on both Android and Windows. Font looks terrible on the comments. Too small, and the the letters are too smushed together. I was going to just change it on the client-side, but then noticed other people complaining.

Couldn't you please just set the comment font to the same as the post font? I would vastly prefer to have it all the same.

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-10-29T17:21:11.655Z · LW(p) · GW(p)

You definitely would not want the comment font be the same as the post font. Legibility would be really terrible for that serif font at the small font-size that you want to display comments as. I am confident it would be much worse for the vast majority of users (feel free to try it yourself). You could change both post font and comment font to a sans-serif, but that would get rid of a lot of the character of the site (and I prefer legibility of serif fonts at larger font sizes).

Replies from: Vladimir_Nesov, nathan-helm-burger, SaidAchmiz, nathan-helm-burger

↑ comment by Vladimir_Nesov · 2024-10-29T23:23:36.537Z · LW(p) · GW(p)

would not want the comment font be the same as the post font [...] the small font-size that you want to display comments as

I had to increase the zoom level by about 20% (from 110% to 130%) after this change to make the comments readable^[1]. This made post text too big to the point where I would normally adjust zoom level downward, but I can't in this case^[2], since the comments are on the same site as the posts. Also the lines in both posts and comments are now too long (with greater zoom).

I sit closer to the monitor than standard to avoid need for glasses^[3], so long lines have higher angular distance. In practice modern sites usually have a sufficiently narrow column of text in the middle so this is almost never a problem. Before the update, LW line lengths were OK (at 110% zoom). At monitor/window width 1920px, substack's 728px seems fine (at default zoom), but LW's 682px get balooned too wide with 130% zoom.

The point is not that accomodating sitting closer to the monitor is an important use case for a site's designer, but that somehow the convergent design of most of the web manages to pass this test, so there might be more reasons for that.

Incidentally, the footnote font size is 12.21px, even smaller than the comment font size of 15.08px.

---

1. The comment font still doesn't feel "sharp", like there's more anti-aliasing at work. It's Gill Sans Nova Medium, size 15.08px (130% zoom applies on top of that). OpenSans Regular 18px on RoyalRoad (100% zoom; as an example sans font) doesn't have this problem. LW post text is fine (at either zoom), Warnock Pro 18.2px. I'm in Firefox on Arch Linux, 1920x1080.
   Here's a zoomed-in screenshot from LW (from 130% zoom in Firefox):
   
   Here's a zoomed-in screenshot from RoyalRoad (from 100% zoom in Firefox):
   ↩︎

2. I previously never felt compelled to figure out how to automate font change in some places of a site. ↩︎

3. That is, with more myopia than I have I would wear glasses, and will less myopia I would put the monitor further back on the desk. ↩︎

↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-10-29T17:47:16.696Z · LW(p) · GW(p)

Small font-size? No! Same font-size! I don't want the comments in a smaller font OR a different font! I want it all the same font as the posts, including the same size.

This looks good to me:

This looks terrible to me:

Replies from: Benito, habryka4

↑ comment by Ben Pace (Benito) · 2024-10-29T18:48:31.577Z · LW(p) · GW(p)

Personally I like the different headspace I'm in for writing posts and comments that the styling gives. One is denser and smaller and less high-stakes, the other is bigger and more presentational, more like a monologue for a large audience.

↑ comment by habryka (habryka4) · 2024-10-29T18:18:33.314Z · LW(p) · GW(p)

You want higher content density for comments than for posts, so you need a smaller font size. You could sacrifice content density, but it would really make skimming comment threads a lot worse.

Replies from: jbash, nathan-helm-burger

↑ comment by jbash · 2024-10-30T01:48:10.762Z · LW(p) · GW(p)

You may want higher density, but I don't think you can say that I want high density at the expense of legibility.

It takes a lot to make me notice layout, and I rarely notice fonts at all... unless they're too small. I'm not as young as I used to be. This change made me think I must have zoomed the browser two sizes smaller. The size contrast is so massive that I have to actually zoom the page to read comfortably when I get to the comment section. It's noticeably annoying, to the point of breaking concentration.

I've mostly switched to RSS for Less Wrong^[1]. I don't see your fonts at all any more, unless I click through on an article. The usual reason I click through is to read the comments (occasionally to check out the quick takes and popular comments that don't show up on RSS). So the comments being inaccessible is doubly bad.

My browser is Firefox on Fedora Linux, and I use a 40 inch 4K monitor (most of whose real estate is wasted by almost every Web site). I usually install most of the available font packages, and it says it's rendering this text in "Gill Sans Nova Medium".

---

1. My big reason for going to RSS was to mitigate the content prioritization system. I want to skim every headline, or at least every headline over some minimum threshold of "good". On the other hand, I don't want to have to look at any old headlines twice to see the new ones. I'm really minimally interested in either the software's or the other users' opinions of which material I should want to see. RSS makes it easier to get a simple chronological view; the built-in chronological view is weird and hard to navigate to. I really feel like I'm having to fight the site to see what I want to see. ↩︎

Replies from: nathan-helm-burger

↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-10-30T03:52:08.624Z · LW(p) · GW(p)

Just want to chime in with agreement about annoyance over the prioritization of post headlines. One thing in particular that annoys me is that I haven't figured out how to toggle off 'seen' posts showing up. What if I just want to see unread ones?

Also, why can't I load more at once instead of always having to click 'load more'?

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-10-30T04:02:58.335Z · LW(p) · GW(p)

The "Recommended" tab filters out read posts by default. We never had much demand for showing recently-sorted posts while filtering out only ones you've read, but it wouldn't be very hard to build. 

Not sure what you mean by "load more at once". We could add a whole user setting to allow users to change the number of posts on the frontpage, but done consistently that would produce a ginormous number of user settings for everything, which would be a pain to maintain (not like, overwhelmingly so, but I would be surprised if it was worth the cost).

↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-10-29T18:42:47.971Z · LW(p) · GW(p)

That doesn't make sense to me, but then, I'm clearly not the target audience since 'skimming comment threads' isn't a thing I ever want to do. I want to read them, carefully and thoughtfully, like I do posts. 

 

This is, I think, related to how I feel that voting (karma or agreement) should be available only at the bottom of posts and comments, so that people are encouraged to actually read the post/comment before voting. Maybe even placed behind a reading comprehension quiz.

Replies from: Sodium

↑ comment by Sodium · 2024-10-29T18:50:30.302Z · LW(p) · GW(p)

I think knowing the karma and agreement is useful, especially to help me decide how much attention to pay to a piece of content, and I don't think there's that much distortion from knowing what others think. (i.e., overall benefits>costs)

Replies from: nathan-helm-burger

↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-10-29T18:55:16.515Z · LW(p) · GW(p)

I'm not saying you shouldn't be able to see the karma and agreement at the top, just that you should only be able to contribute your own opinion at the bottom, after reading and judging for yourself.

↑ comment by Said Achmiz (SaidAchmiz) · 2024-10-29T18:39:48.878Z · LW(p) · GW(p)

You definitely would not want the comment font be the same as the post font.

This… seems straightforwardly false? Every one of GreaterWrong’s eight themes uses a single font for both posts and comments, and it doesn’t cause any problems. (And it’s a different font for each theme!)

Replies from: habryka4, nathan-helm-burger

↑ comment by habryka (habryka4) · 2024-10-29T20:51:15.320Z · LW(p) · GW(p)

(I think it's quite costly and indeed one of the things I like least about the GW design, but also, I was more talking about a straightforward replacement.

On LW we made a lot of subsequent design choices based on different content density, and the specific fonts we chose are optimized for their respective most commonly used font sizes. I am confident the average user experience would become worse if you just replaced the comment font with the body font)

Replies from: SaidAchmiz

↑ comment by Said Achmiz (SaidAchmiz) · 2024-10-30T05:44:03.773Z · LW(p) · GW(p)

I am confident the average user experience would become worse if you just replaced the comment font with the body font)

Yeah, I agree with that, but that’s because of a post body font that wasn’t chosen for suitability for comments also. If you pick, to begin with, a font that works for both, then it’ll work for both.

… of course, if you don’t think that any of the GW themes’ fonts work for both, then never mind, I guess. (But, uh, frankly I find that to be a strange view. But no accounting for taste, etc., so I certainly can’t say it’s wrong, exactly.)

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-10-30T06:00:50.334Z · LW(p) · GW(p)

Sure, I was just responding to this literal quote: 

Couldn't you please just set the comment font to the same as the post font?

↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-10-29T19:06:03.006Z · LW(p) · GW(p)

Good point! I went and looked their themes. I prefer LessWrong's look, except for the comments. 

Again, this doesn't matter much to me since I can customize client-side, I just wanted to let habryka know that some people dislike the new comment font and would prefer the same font and size as the normal post font.

My view on phone (Android, Firefox): https://imgur.com/a/Kt1OILQ 

 

How my client view looks on my computer:

↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-10-29T20:04:40.001Z · LW(p) · GW(p)

How about running a poll to see what users prefer?

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-10-29T20:56:14.941Z · LW(p) · GW(p)

We have done lots of users interviews over the years! Fonts are always polarizing, but people have a strong preference for sans serifs at small font sizes (and people prefer denser comment sections, though it's reasonably high variance).

↑ comment by green_leaf · 2024-10-30T06:39:49.945Z · LW(p) · GW(p)

I use Google Chrome on Ubuntu Budgie and it does look to me like both the font and the font size changed.

↑ comment by DanielFilan · 2024-10-30T00:02:22.326Z · LW(p) · GW(p)

It looks kinda small to me, someone who uses Firefox on Ubuntu.

Replies from: DanielFilan

↑ comment by DanielFilan · 2024-10-31T21:35:47.550Z · LW(p) · GW(p)

Update: I have already gotten over it.

Replies from: T3t

↑ comment by RobertM (T3t) · 2024-11-01T00:43:36.827Z · LW(p) · GW(p)

(We switched back to shipping Calibri above Gill Sans Nova pending a fix for the horrible rendering on Windows, so if Ubuntu has Calibri, it'll have reverted back to the previous font.)

Replies from: DanielFilan

↑ comment by DanielFilan · 2024-11-01T20:07:34.400Z · LW(p) · GW(p)

I believe I'm seeing Gill Sans? But when I google "Calibri" I see text that looks like it's in Calibri, so that's confusing.

Replies from: kave

↑ comment by kave · 2024-11-01T20:59:24.773Z · LW(p) · GW(p)

Yeah, that's a google Easter Egg. You can also try "Comic Sans" or "Trebuchet MS".

Replies from: DanielFilan

↑ comment by DanielFilan · 2024-11-01T21:28:22.293Z · LW(p) · GW(p)

Sure, I'm just surprised it could work without me having Calibri installed.

Replies from: kave

↑ comment by kave · 2024-11-01T21:35:50.115Z · LW(p) · GW(p)

They load it in as a web font (i.e. you load Calibri from their server when you load that search page). We don't do that on LessWrong

↑ comment by Kaj_Sotala · 2024-10-29T13:35:56.008Z · LW(p) · GW(p)

Seeing strange artifacts on some of the article titles on Chrome for Android (but not on desktop)

↑ comment by Thomas Kwa (thomas-kwa) · 2024-11-07T02:08:57.477Z · LW(p) · GW(p)

The new font doesn't have a few characters useful in IPA.

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-11-07T03:21:20.974Z · LW(p) · GW(p)

Ah, we should maybe font-subset some system font for that (same as what we did for greek characters). If someone gives me a character range specification I could add it.

We were down between around 7PM and 8PM PT today. Sorry about that.

It's hard to tell whether we got DDosd or someone just wanted to crawl us extremely aggressively, but we've had at least a few hundred IP addresses and random user agents request a lot of quite absurd pages, in a way that was clearly designed to avoid bot-detection and block methods. 

I wish we were more robust to this kind of thing, and I'll be monitoring things tonight to prevent it from happening again, but it would be a whole project to make us fully robust to attacks of this kind. I hope it was a one-off occurence. 

But also, I think we can figure out how to make it so we are robust to repeated DDos attacks, if that is the world we live in. I do think it would mean strapping in for a few days of spotty reliability while we figure out how to do that.

Sorry again, and boo for the people doing this. It's one of the reasons why running a site like LessWrong is harder than it should be.

A bunch of very interesting emails between Elon, Sam Altman, Ilya and Greg were released (I think in some legal proceedings, but not sure). It would IMO be cool for someone to gather them all and do some basic analysis of them. 

https://x.com/TechEmails/status/1857456137156669765 

https://x.com/TechEmails/status/1857285960997712356 

What is the purpose of karma?

LessWrong has a karma system, mostly based off of Reddit's karma system, with some improvements and tweaks to it. I've thought a lot about more improvements to it, but one roadblock that I always run into when trying to improve the karma system, is that it actually serves a lot of different uses, and changing it in one way often means completely destroying its ability to function in a different way. Let me try to summarize what I think the different purposes of the karma system are:

Helping users filter content

The most obvious purpose of the karma system is to determine how long a post is displayed on the frontpage, and how much visibility it should get.

Being a social reward for good content

This aspect of the karma system comes out more when thinking about Facebook "likes". Often when I upvote a post, it is more of a public signal that I value something, with the goal that the author will feel rewarded for putting their effort into writing the relevant content.

Creating common-knowledge about what is good and bad

This aspect of the karma system comes out the most when dealing with debates, though it's present in basically any karma-related interaction. The fact that the karma of a post is visible to everyone, helps people establish common knowledge of what the community considers to be broadly good or broadly bad. Seeing a an insult downvoted, does more than just filter it out of people's feeds, it also makes it so that anyone who stumbles accross it learns something about the norms of the community.

Being a low-effort way of engaging with the site

On lesswrong, Reddit and Facebook, karma is often the simplest action you can take on the site. This means its usually key for a karma system like that to be extremely simple, and not require complicated decisions, since that would break the basic engagement loop with the site.

Problems with alternative karma systems

Here are some of the most common alternatives to our current karma system, and how they perform on the above dimensions:

Eigenkarma as weighted by a set of core users

The basic idea here is that you try to signal-boost a small set of trusted users, by giving people voting power that is downstream from the initially defined set of users.

There are some problems with this. The first one is whether to assign any voting power to new users. If you don't you remove a large part of the value of having a low-effort way of engaging with your site.

It also forces you to separate the points that you get on your content, from your total karma score, from your "karma-trust score" which introduces some complexity into the system. It also makes it so that increases in the points of your content, no longer neatly correspond to voting events, because the underlying reputation graph is constantly shifting and changing, making the social reward signal a lot weaker.

In exchange for this, you likely get a system that is better at filtering content, and probably has better judgement about what should be made common-knowledge or not.

Prediction-based system

I was talking with Scott Garrabrant today, who was excited about a prediction-based karma system. The basic idea is to just have a system that tries to do its best to predict what rating you are likely to give to a post, based on your voting record, the post, and other people's votes.

In some sense this is what Youtube and Facebook are doing in their systems, though he was unhappy with the transparency of what they were doing.

The biggest sacrifice I see in creating this system, is the loss in the ability to create common knowledge, since now all votes are ultimately private, and the ability for karma to establish social norms, or just common knowledge about foundational facts that the community is built around, is greatly diminished.

I also think it diminishes the degree to which votes can serve as a social reward signal, since there is no obvious thing to inform the user of when their content got votes on. No number that went up or down, just a few thousand weights in some distant predictive matrix, or neural net.

Augmenting experts

A similar formulation to the eigenkarma system is the idea of trying to augment experts, by rewarding users in proportion to how successful they are at predicting how a trusted expert would vote, and then using that predicted expert's vote as the reward signal. Periodically, you do query the trusted expert, and use that to calibrate and train the users who are trying to predict the expert.

This still allows you to build common-knowledge, and allows you to have effective reward signals ("simulated Eliezer upvoted your comment"), but does run into problems when it comes to being a low-effort way of engaging with the site. The operation of "what would person X think about this comment" is a much more difficult one than "did I like this comment?", and as such might deter a large number of users from using your site.

I just came back from talking to Max Harms about the Crystal trilogy, which made me think about rationalist fiction, or the concept of hard sci-fi combined with explorations of cognitive science and philosophy of science in general (which is how I conceptualize the idea of rationalist fiction). 

I have a general sense that one of the biggest obstacles for making progress on difficult problems is something that I would describe as “focusing attention on the problem”. I feel like after an initial burst of problem-solving activity, most people when working on hard problems, either give up, or start focusing on ways to avoid the problem, or sometimes start building a lot of infrastructure around the problem in a way that doesn’t really try to solve it. 

I feel like one of the most important tools/skills that I see top scientist or problem solvers in general use, is utilizing workflows and methods that allow them to focus on a difficult problem for days and months, instead of just hours. 

I think at least for me, the case of exam environments displays this effect pretty strongly. I have a sense that in an exam environment, if I am given a question, I successfully focus my full attention on a problem for a full hour, in a way that often easily outperforms me thinking about a problem in a lower key environment for multiple days in a row.

And then, when I am given a problem set with concrete technical problems, my attention is again much better focused than when I am given the same problem but in a much less well-defined way. E.g. thinking about solving some engineering problem, but without thinking about it by trying to create a concrete proof or counterproof. 

My guess is that there is a lot of potential value in fiction that helps people focus their attention on a problem in a real way. In fiction you have the ability to create real-feeling stakes that depend on problem solving, and things like the final exam in Methods of Rationality show how that can be translated into large amounts of cognitive labor. 

I think my strongest counterargument to this model is something like “sure, it’s easy to make progress on problems when you have someone else give you the correct ontology in which the problem is solvable, but that’s just because 90% of the work of solving problems is coming up with the right ontologies for problems like this”. And I think there is something importantly real about this, but also that it doesn’t fully address the value of exams and fiction and problem sets that I am trying to point to (though I do think it explains a good chunk of their effect). 

Going back to the case of fiction, it is clear to me that fiction is as a literary form much more optimized to hold human attention that most non-fiction is. I think first of all that this constraint means that most fiction (and in particular most popular fiction) isn’t about much else than whatever best holds people’s attention, but it also means that if the bottleneck on a lot of problems is just getting people to hold their attention on the problem for a while, then utilizing the methods that fiction-writing has developed seems like an obvious way of making progress on those problems. 

I feel like another major effect that explains a lot of the effects that I observe is people believing that a problem is solvable. In a fictional setting, if the author promises you that things have a good explanation, then it’s motivating to figure out why. On an exam you are promised that the problems that you are given are solvable, and solvable within a reasonable amount of time. 

I do think this can still be exploited. In the last few chapters of HPMOR, Harry does a mental motion that I would describe as "don't waste mental energy on asking whether a problem is solvable, just pretend it it, and ask what the solution would be if it was solvable", in a way that felt to me like it would work on a lot of real-world problems. 

Is it OK for LW admins to look at DM metadata for spam prevention reasons? 

Sometimes new users show up and spam a bunch of other users in DMs (in particular high-profile users). We can't limit DM usage to only users with activity on the site, because many valuable DMs get sent by people who don't want to post publicly. We have some basic rate limits for DMs, but of course those can't capture many forms of harassment or spam. 

Right now, admins can only see how many DMs users have sent, and not who users have messaged, without making a whole manual database query, which we have a policy of not doing unless we have a high level of suspicion of malicious behavior. However, I feel like it would be quite useful for identifying who is doing spammy things if we could also see who users have sent DMs to, but of course, this might feel bad from a privacy perspective to people. 

So I am curious about what others think. Should admins be able to look at DM metadata to help us identify who is abusing the DM system? Or should we stick to aggregate statistics like we do right now? (React or vote "agree" if you think we should use DM metadata, and react or vote "disagree" if you think we should not use DM metadata).

Oops, I am sorry. We did not intend to take the site down. We ran into an edge-case of our dialogue code that nuked our DB, but we are back up, and the Petrov day celebrations shall continue as planned. Hopefully without nuking the site again, intentionally or unintentionally. We will see.

Final day to donate to Lightcone in the Manifund EA Community Choice program to tap into the Manifold quadratic matching funds. Small donations in-particular have a pretty high matching multiplier (around 2x would be my guess for donations <$300). 

I don't know how I feel in-general about matching funds, but in this case it seems like there is a pre-specified process that makes some sense, and the whole thing is a bit like a democratic process with some financial stakes, so I feel better about it.

I am in New York until Tuesday. DM me if you are in the area and want to meet up and talk about LW, how to use AI for research/thinking/writing, or broader rationality community things.

Currently lots of free time Saturday and Monday. 

Is intellectual progress in the head or in the paper?

Which of the two generates more value:

• A researcher writes up a core idea in their field, but only a small fraction of good people read it in the next 20 years
• A researchers gives a presentation at a conference to all the best researchers in his field, but none of them write up the idea later

I think which of the two will generate more value determines a lot of your strategy about how to go about creating intellectual progress. In one model what matters is that the best individuals hear about the most important ideas in a way that then allows them to make progress on other problems. In the other model what matters is that the idea gets written as an artifact that can be processed and evaluated by reviews and the proper methods of the scientific progress, and then built upon when referenced and cited.

I think there is a tradeoff of short-term progress against long-term progress in these two approaches. I think many fields can go through intense periods of progress when focusing on just establishing communication between the best researchers of the field, but would be surprised if that period lasts longer than one or two decades. Here are some reasons for why that might be the case:

• A long-lasting field needs a steady supply of new researchers and thinkers, both to bring in new ideas, and also to replace the old researchers who retire. If you do not write up your ideas, the ability for a field to evaluate the competence of a researchers has to rely on the impressions of individual researchers. My sense is that relying on that kind of implicit impression does not survive multiple successions and will get corrupted by people trying to use their influence for some other means within two decades.
• You are blocking yourself off from interdisciplinary progress. After a decade a two fields often end up in a rut that needs some new paradigm or at least new idea to allow people to make progress again. If you don't write up your ideas publicly, you lose a lot of opportunities for interdisciplinary researchers to enter your field and bring in ideas from other places.
• You make it hard to improve on research debt because there is no canonical reference that can be updated with better explanations and better definitions. (Current journals don't do particularly well on this, but this is an opportunity that wiki-like systems can take advantage of, or with some kind of set of published definitions like the DSM-5, and new editions of textbooks also help with this)
• If you are a theoretical field, you are making it harder for your ideas to get implemented or transformed into engineering problems. This prevents your field from visibly generating value, which reduces both the total amount of people who want to join your field, and also the interest of other people to invest resources into your field

However, you also gain a large number of benefits, that will probably increase your short-term output significantly:

• Through the use of in-person conversations and conferences the cost of communicating a new idea and letting others build on it is often an order of magnitude smaller
• Your ability to identify the best talent can now be directly downstream of the taste of the best people in the field, which allows you to identify researchers who are not great at writing, but still great at thinking
• The complexity limit of any individual idea in your field is a lot higher, since the ideas get primarily transmitted via high-bandwidth channels
• Your feedback cycles of getting feedback on your ideas from other people in the field is a lot faster, since your ideas don't need to go through a costly writeup and review phase

My current model is that it's often good for research fields to go through short periods (< 2 years) in which there is a lot of focus on just establishing good communications among the best researchers, either with a parallel investment in trying to write up at least the basics of the discussion, or a subsequent clean-up period in which the primary focus is on writing up the core insights that all the best researchers converged on.

Thoughts on minimalism, elegance and the internet:

I have this vision for LessWrong of a website that gives you the space to think for yourself, and doesn't constantly distract you with flashy colors and bright notifications and vibrant pictures. Instead it tries to be muted in a way that allows you to access the relevant information, but still gives you the space to disengage from the content of your screen, take a step back and ask yourself "what are my goals right now?".

I don't know how well we achieved that so far. I like our frontpage, and I think the post-reading experience is quite exceptionally focused and clear, but I think there is still something about the way the whole site is structured, with its focus on recent content and new discussion that often makes me feel scattered when I visit the site.

I think a major problem is that Lesswrong doesn't make it easy to do only a single focused thing on the site at a time, and it doesn't currently really encourage you to engage with the site in a focused way. We have the library, which I do think is decent, but the sequence navigation experience is not yet fully what I would like it to be, and when I go to the frontpage the primary thing I still see is recent content. Not the sequences I recently started reading, or the practice exercises I might want to fill out, or the open questions I might want to answer.

I think ther are a variety of ways to address this, some of which I hope to build very soon:

+ The frontpage should show you not only recent content, but also show you much older historical content (that can be of much higher quality, due to being drawn from a much larger pool). [We have a working prototype of this, and I hope we can push it soon]

+ We should encourage you to read whole sequences at a time, instead of individual posts. If you start reading a sequence, you should be encouraged to continue reading it from the frontpage [This is also quite close to working]

+ There should be some way to encourage people to put serious effort into answering the most important open questions [This is currently mostly bottlenecked on making the open-question system/UX good enough to make real progress in]

+ You should be able to easily bookmark posts and comments to allow you to continue reading something at a later point in time [We haven't really started on this, but it's pretty straightforward, so I still think this isn't too far off]

+ I would love it if there were real rationality exercises in many of the sequences, in a way that would periodically require you to write essays and answer questions and generally check your understanding. This is obviously quite difficult to make happen, both in terms of UI, but also in terms of generating the content

I think if we had all of these, in particular the open questions one, then I think I would feel more like LessWrong is oriented towards my long-term growth instead of trying to give me short-term reinforcement. It would also create a natural space in which to encourage focused work and generally make me feel less scattered when I visit the site, due to deemphasizing the most recent wave of content.

I do think there are problems with deemphasizing more recent content, mostly because this indirectly disincentivizes creating new content, which I do think would obviously be bad for the site. Though in some sense it might encourage the creation of longer-lived content, which would be quite good for the site.

Thoughts on negative karma notifications:

• An interesting thing that I and some other people on the LessWrong team noticed (as well as some users) was that since we created karma notifications we feel a lot more hesitant to downvote older comments, since we know that this will show up for the other users as a negative notification. I also feel a lot more hesitant to retract my own strong upvotes or upvotes in general since the author of the comment will see that as a downvote.
• I've had many days in a row in which I received +20 or +30 karma, followed by a single day where by chance I received a single downvote and ended up at -2. The emotional valence of having a single day at -2 was somehow stronger than the emotional valence of multiple days of +20 or +30.

Here is a thing that I think would be cool to analyze sometime: How difficult would it have been for AI systems to discover and leverage historical hardware-level vulnerabilities, assuming we had not discovered them yet. Like, it seems worth an analysis to understand how difficult things like rowhammer, or more recent speculative execution bugs like Spectre and Meltdown would have been to discover, and how useful they would have been. It's not an easy analysis, but I can imagine the answer coming out obviously one way or another if one engaged seriously with the underlying issue.

Thoughts on impact measures and making AI traps

I was chatting with Turntrout today about impact measures, and ended up making some points that I think are good to write up more generally.

One of the primary reasons why I am usually unexcited about impact measures is that I have a sense that they often "push the confusion into a corner" in a way that actually makes solving the problem harder. As a concrete example, I think a bunch of naive impact regularization metrics basically end up shunting the problem of "get an AI to do what we want" into the problem of "prevent the agent from interferring with other actors in the system".

The second one sounds easier, but mostly just turns out to also require a coherent concept and reference of human preferences to resolve, and you got very little from pushing the problem around that way, and sometimes get a false sense of security because the problem appears to be solved in some of the toy problems you constructed.

I am definitely concerned that Turntrou's AUP does the same, just in a more complicated way, but am a bit more optimistic than that, mostly because I do have a sense that in the AUP case there is actually some meaningful reduction going on, though I am unsure how much.

In the context of thinking about impact measures, I've also recently been thinking about the degree to which "trap-thinking" is actually useful for AI Alignment research. I think Eliezer was right in pointing out that a lot of people, when first considering the problem of unaligned AI, end up proposing some kind of simple solution like "just make it into an oracle" and then consider the problem solved.

I think he is right that it is extremely dangerous to consider the problem solved after solutions of this type, but it isn't obvious that there isn't some good work that can be done that is born out of the frame of "how can I trap the AI and make it marginally harder for it to be dangerous, basically pretending it's just a slightly smarter human?".

Obviously those kinds of efforts won't solve the problem, but they still seem like good things to do anyways, even if they just buy you a bit of time, or help you notice a bit earlier if your AI is actually engaging in some kind of adversarial modeling.

My broad guess is that research of this type is likely very cheap and much more scalable, and you hit diminishing marginal returns on it much faster than you would on AI Alignment research that is tackling the core problem, so it might just be fine to punt it until later. Though if you are acting on very short timelines it probably should still be someones job to make sure that someone at Deepmind tries to develop the obvious transparency technologies to help you spot if your neural net has any large fractions of it dedicated to building sophisticated human modeling, even if this won't solve the problem in the long-run.

This perspective, combined with Wei Dai's recent comments that one job of AI Alignment researchers is to produce evidence that the problem is actually difficult, is that it might be a good idea for some people to just try to develop lots of benchmarks of adversarial behavior that have any chance of triggering before you have a catastrophic failure. Like, it seems obviously great to have a paper that takes some modern ML architecture and can clearly demonstrate in which cases it might engage in adversarial modeling, and maybe some remotely realistic scenarios where that might happen.

My current guess is that current ML architectures aren't really capable of adversarial modeling in this way, though I am not actually that confident of that, and actually would be somewhat surprised if you couldn't get any adversarial behavior out of a dedicated training regime, if you were to try. For example, let's say I train an RL-based AI architecture on chat interactions with humans in which it just tries to prolong the length of the chat session as much as possible. I would be surprised if the AI wouldn't build pretty sophisticated models of human interactions, and try some weird tactics like get the human to believe that it is another human, or pretend that it is performing some long calculation, or deceive the humans in a large variety of ways, at least if it was pretrained with a language model of comparable quality to GPT-2, and had similar resources going to it as Open AI Five. Though it's also unclear to what degree this would actually give us evidence about treacherous turn scenarios.

I've also been quite curious about the application of ML to computer security, where an obvious experiment is to just try to set up some reasonable RL-architecture in which I have an AI interface with a webserver, trying to get access to some set of files that it shouldn't get access to . The problem here is obviously the sparse reward landscape, and there really isn't an obvious training regime here, but showing how even current AI could possibly leverage security vulnerabilities in a lot of systems in a way that could easily give rise to unintented side-effects could be a valuable goal. But in general training RL for almost anything is really hard, so this seems unlikely to work straightforwardly.

Overall, I am not sure what I feel about the perspective I am exploring above. I have a deep sense that a lot of it is just trying to dodge the hard parts of the problem, but it seems fine to put on my hat for short-term, increase marginal difficulty of bad outcomes, for a bit and see how I feel after exploring it for a while.

Printing more rationality books: I've been quite impressed with the success of the printed copies of R:A-Z and think we should invest resources into printing more of the other best writing that has been posted on LessWrong and the broad diaspora.

I think a Codex book would be amazing, but I think there also exists potential for printing smaller books on things like Slack/Sabbath/etc., and many other topics that have received a lot of other coverage over the years. I would also be really excited about printing HPMOR, though that has some copyright complications to it.

My current model is that there exist many people interested in rationality who don't like reading longform things on the internet and are much more likely to read things when they are in printed form. I also think there is a lot of value in organizing writing into book formats. There is also the benefit that the book now becomes a potential gift for someone else to read, which I think is a pretty common way ideas spread.

I have some plans to try to compile some book-length sequences of LessWrong content and see whether we can get things printed (obviously in coordination with the authors of the relevant pieces).

Forecasting on LessWrong: I've been thinking for quite a while about somehow integrating forecasts and prediction-market like stuff into LessWrong. Arbital has these small forecasting boxes that look like this:

I generally liked these, and think they provided a good amount of value to the platform. I think our implementation would probably take up less space, but the broad gist of Arbital's implementation seems like a good first pass.

I do also have some concerns about forecasting and prediction markets. In particular I have a sense that philosophical and mathematical progress only rarely benefits from attaching concrete probabilities to things, and more works via mathematical proof and trying to achieve very high confidence on some simple claims by ruling out all other interpretations as obviously contradictory. I am worried that emphasizing probability much more on the site would make making progress on those kinds of issues harder.

I also think a lot of intellectual progress is primarily ontological, and given my experience with existing forecasting platforms and Zvi's sequence on prediction markets, they are not very good at resolving ontological confusions and often seem to actively hinder them by causing lots of sunk-costs into easy-to-operationalize ontologies that tend to dominate the platforms.

And then there is the question of whether we want to go full-on internal prediction market and have active markets that are traded in some kind of virtual currency that people actually care about. I think there is a lot of value in that direction, but it's obviously also a lot of engineering effort that isn't obviously worth it. It seems likely better to wait until a project like foretold.io has reached maturity and then see whether we can integrate it into LessWrong somehow.

We are rolling out some new designs for the post page: 

Old:

New: 

The key goal was to prioritize the most important information and declutter the page. 

The most opinionated choice I made was to substantially de-emphasize karma at the top of the post page. I am not totally sure whether that is the right choice, but I think the primary purpose of karma is to use it to decide what to read before you click on a post, which makes it less important to be super prominent when you are already on a post page, or when you are following a link from some external website.

The bottom of the post still has very prominent karma UI to make it easy for people to vote after they finished reading a post (and to calibrate on reception before reading the comments).

This redesign also gives us more space in the right column, which we will soon be filling with new side-note UI and an improved inline-react experience. 

The mobile UI is mostly left the same, though we did make the decision to remove post-tags from the top of the mobile UI page to only making them visible below the post, because they took up too much space.

Feel free to comment here with feedback. I expect we will be iterating on the new design quite actively over the coming days and weeks.

↑ comment by papetoast · 2024-08-16T08:46:51.874Z · LW(p) · GW(p)

I like most of the changes, but strongly dislike the large gap before the title. (I similarly dislike the large background in the top 50 of the year posts)

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-08-16T17:15:46.350Z · LW(p) · GW(p)

Well, the gap you actually want to measure is the gap between the title and the audio player (or at the very least the tags), since that's the thing we need to make space for. You are clearly looking at LW on an extremely large screen. This is the more median experience: 

There is still a bunch of space there, but for many posts the tags extend all the way above the post. 

Replies from: papetoast

↑ comment by papetoast · 2024-08-17T04:23:20.296Z · LW(p) · GW(p)

I understand that having the audio player above the title is the path of least resistance, since you can't assume there is enough space on the right to put it in. But ideally things like this should be dynamic, and only take up vertical space if you can't put it on the right, no? (but I'm not a frontend dev)

Alternatively, I would consider moving them vertically above the title a slight improvement. It is not great either, but at least the reason for having the gap is more obvious.

The above screenshots are done in a 1920x1080 monitor

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-08-17T04:56:28.944Z · LW(p) · GW(p)

Yeah, we could make things dynamic, it would just add complexity that we would need to check every time we make a change. It's the kind of iterative improvement we might do over time, but it's not something that should block the roll-out of a new design (and it's often lower priority than other things, though post-pages in-particular are very important and so get a lot more attention than other pages).

Had a very aggressive crawler basically DDos-ing us from a few dozen IPs for the last hour. Sorry for the slower server response times. Things should be fixed now.

Random thoughts on game theory and what it means to be a good person

It does seem to me like there doesn’t exist any good writing on game theory from a TDT perspective. Whenever I read classical game theory, I feel like the equilibria that are being described obviously fall apart when counterfactuals are being properly brought into the mix (like D/D in prisoners dilemmas).

The obvious problem with TDT-based game theory, just as it is with Bayesian epistemology, the vast majority of direct applications are completely computationally intractable. It’s kind of obvious what should happen in games with lots of copies of yourself, but as soon as anything participates that isn’t a precise copy, everything gets a lot more confusing. So it is not fully clear what a practical game-theory literature from a TDT-perspective would look like, though maybe the existing LessWrong literature on Bayesian epistemology might be a good inspiration.

Even when you can’t fully compute everything (and we even don’t really know how to compute everything in principle), you might still be able to go through concrete scenarios and list considerations and perspectives that incorporate TDT-perspectives. I guess in that sense, a significant fraction of Zvi’s writing could be described as practical game theory, though I do think there is a lot of value in trying to formalize the theory and make things as explicit as possible, which I feel like Zvi at least doesn’t do most of the time.

Critch (Academian) tends to have this perspective of trying to figure out what a “robust agent” would do, in the sense of an agent that would at the very least be able to reliably cooperate with copies of itself, and adopt cooperation and coordination principles that allow it to achieve very good equilibria with agents that adopt the same type of cooperation and coordination norms. And I do think there is something really valuable here, though I am also worried that the part where you have to cooperate with agents who haven’t adopted super similar cooperation norms is actually the more important one (at least until something like AGI).

And I do think that the majority of the concepts we have for what it means to be a “good person” are ultimately attempts at trying to figure out how to coordinate effectively with other people, in a way that a more grounded game theory would help a lot with.

Maybe a good place to start would be to brainstorm a list of concrete situations in which I am uncertain what the correct action is. Here is some attempt at that:

• How to deal with threats of taking strongly negative-sum actions? What is the correct response to the following concrete instances?

  • A robber threatens to shoot you if you don’t hand over your wallet
    • Do you precommit to violently attack any robber that robs you, or do you simply hand over your wallet?

• You are in the room with someone holding the launch buttons for the USA’s nuclear arsenal and they are threatening to launch them if you don’t hand over your wallet

• You are head of the U.S. and another nation state is threatening a small-scale nuclear attack on one of your cities if you don’t provide some kind of economic subsidy to them

  • Do you launch a conventional attack?
  • Do you launch a full out nuclear response as a deterrent?
  • Do you launch a small-scale nuclear response?
  • Do you not do anything at all?
  • Does the answer depend on the size of the economic subsidy? What if they ask twice?

• You are at a party and your assigned driver ended up drinking, even though they said they would not (the driver was chosen by a random draw)

  • Do you somehow punish them now, do you punish them later, or not at all?
  • What if they are less likely to remember if you punish them now because they are drunk? Does that matter for the game-theoretic correct action?
  • What if they did this knowingly, reasoning from a CDT perspective that there wouldn’t be any point in punishing them now because they wouldn’t remember the next day
    • What if you would never see them again later?
    • What if you only ever get to interact with them after they made the choice to be drunk?

I feel like I have some hint of an answer to all of these, but also feel like any answer that I can come up with makes me exploitable in a way that makes me feel like there is no meta-level on which there is an ideal strategy.

Making yourself understandable to other people

(Epistemic status: Processing obvious things that have likely been written many times before, but that are still useful to have written up in my own language)

How do you act in the context of a community that is vetting constrained? I think there are fundamentally two approaches you can use to establish coordination with other parties:

1. Professionalism: Establish that you are taking concrete actions with predictable consequences that are definitely positive

2. Alignment: Establish that you are a competent actor that is acting with intentions that are aligned with the aims of others

I think a lot of the concepts around professionalism arise when you have a group of people who are trying to coordinate, but do not actually have aligned interests. In those situations you will have lots of contracts and commitments to actions that have well-specified outcomes and deviations from those outcomes are generally considered bad. It also encourages a certain suppression of agency and a fear of people doing independent optimization in a way that is not transparent to the rest of the group.

Given a lot of these drawbacks, it seems natural to aim for establishing alignment with others, it is however much less clear how to achieve that. Close group of friends can often act in alignment because they have credibly signaled to each other that they care about each others experiences and goals. This also tends to involve costly signals of sacrifice that are only economical if the goals of the participants were actually aligned. I also suspect that there is a real "merging of utility functions" going on, where close friends and partners self-modify to share each others values.

For larger groups of people, establishing alignment with each other seems much harder, in particular in the presence of adversarial actors. You can request costly signals, but it is often difficult to find good signals that are not prohibitively costly for many members of your group (this task is much easier for smaller groups, since you have less spread in the costs of different actions). You are also under much more adversarial pressure, since with more people you likely have access to more resources which attracts more adversarial actors.

I expect this is the reason why we see larger groups often default to professionalism norms with very clearly defined contracts.

I think the EA and Rationality communities have historically optimized hard for alignment and not professionalism, since that enabled much better overall coordination, but as the community grew and attracted more adversarial actors those methods didn't scale very well and so we currently expect alignment-level coordination capabilities while only having access to professionalism-level coordination protocols and resources.

We've also seen an increase in people trying to increase the amount of alignment, by looking into things like circling and specializing in mediation and facilitation, which I think is pretty promising and has some decent traction.

I also think there is a lot of value in building better infrastructure and tools for more "professionalism" style interactions, where people offer concrete services with bounded upside. A lot of my thinking on the importance of accountability derives from this perspective.

This FB post by Matt Bell on the Delta Variant helped me orient a good amount: 

https://www.facebook.com/thismattbell/posts/10161279341706038

As has been the case for almost the entire pandemic, we can predict the future by looking at the present. Let’s tackle the question of “Should I worry about the Delta variant?” There’s now enough data out of Israel and the UK to get a good picture of this, as nearly all cases in Israel and the UK for the last few weeks have been the Delta variant. [1] Israel was until recently the most-vaccinated major country in the world, and is a good analog to the US because they’ve almost entirely used mRNA vaccines.

- If you’re fully vaccinated and aren’t in a high risk group, the Delta variant looks like it might be “just the flu”. There are some scary headlines going around, like “Half of new cases in Israel are among vaccinated people”, but they’re misleading for a couple of reasons. First, since Israel has vaccinated over 80% of the eligible population, the mRNA vaccine still is 1-((0.5/0.8)/(0.5/0.2)) = 75% effective against infection with the Delta variant. Furthermore, the efficacy of the mRNA vaccine is still very high ( > 90%) against hospitalization or death from the Delta variant. Thus, you might still catch Delta if you’re vaccinated, but it will be more like a regular flu if you do. J&J likely has a similar performance in terms of reduced hospitalizations and deaths, as the UK primarily vaccinated its citizens with the AstraZeneca vaccine, which is basically a crappier version of J&J, and is still seeing a 90+% reduction in hospitalizations and deaths among the vaccinated population. [2]

[...]

This seems like potentially a big deal: https://mobile.twitter.com/DrEricDing/status/1402062059890786311

> Troubling—the worst variant to date, the #DeltaVariant is now the new fastest growing variant in US. This is the so-called “Indian” variant #B16172 that is ravaging the UK despite high vaccinations because it has immune evasion properties. Here is why it’s trouble—Thread. #COVID19
 

@Elizabeth [LW · GW] was interested in me crossposting this comment from the EA Forum since she thinks there isn't enough writing on the importance of design on LW. So here it is.

Atlas reportedly spent $10,000 on a coffee table. Is this true? Why was the table so expensive?

Atlas at some point bought this table, I think: https://sisyphus-industries.com/product/metal-coffee-table/. At that link it costs around $2200, so I highly doubt the $10,000 number.

Lightcone then bought that table from Atlas a few months ago at the listing price, since Jonas thought the purchase seemed excessive, so Atlas actually didn't end up paying anything. I am really glad we bought it from them, it's probably my favorite piece of furniture in the whole venue we are currently renovating.

If you think it was a waste of money, I have made much worse interior design decisions (in-general furniture is really annoyingly expensive, and I've bought couches for $2000 that turned out to just not work for us at all and were too hard to sell), and I consider this one a pretty strong hit. (To clarify, the reason why it's so expensive is because it's a kinetic sculpture with a moving magnet and a magnetic ball that draws programmable patterns into the sand at the center of the table, so it's not just like, a pretty coffee table)

The table is currently serving as a centerpiece of our central workspace social room, and has a pretty large effect on good conversations happening since it seems to hit the right balance of being visually interesting without being too distracting while also being functional, and despite this kind of sounding ridiculous, if for some reason it was impossible for Lightcone to pay for this table (which I don't think it is since I think interior design matters), I would pay for it from my own personal funds. 

In general, as someone who has now helped prepare on the order of 20 venues for workshops and conferences, it seems pretty obvious to me that interior design matters quite a bit for workshop venues. I think it would indeed be pretty crazy to pay $2000 for every coffee table in your venue, but a single central design piece can make a huge difference to a room, and I've spent hundreds of hours trying to design rooms to facilitate good conversations with my counterfactual earning rate being in the hundreds of dollars per hours, and I think it definitely is sometimes worth my time/money to buy an occasional expensive piece of furniture.

I like this shortform feed idea!

We launched the books on Product Hunt today! 

Leaving this here: 2d9797e61e533f03382a515b61e6d6ef2fac514f