Posts

Larks's Shortform 2022-08-30T13:43:39.046Z
2021 AI Alignment Literature Review and Charity Comparison 2021-12-23T14:06:50.721Z
2020 AI Alignment Literature Review and Charity Comparison 2020-12-21T15:27:19.303Z
2019 AI Alignment Literature Review and Charity Comparison 2019-12-19T03:00:54.708Z
2018 AI Alignment Literature Review and Charity Comparison 2018-12-18T04:46:55.445Z
2017 AI Safety Literature Review and Charity Comparison 2017-12-24T18:52:31.816Z
2018 AI Safety Literature Review and Charity Comparison 2017-12-20T22:04:47.174Z
2016 AI Risk Literature Review and Charity Comparison 2016-12-15T00:19:21.966Z
Contrarian LW views and their economic implications 2014-10-08T23:48:04.250Z
Confirmation Bias Presentation 2014-06-05T21:07:31.918Z
Meetup : Princeton NJ Meetup 2014-03-23T00:22:06.409Z
Meetup : Princeton NJ Meetup 2014-02-02T22:32:10.247Z
Meetup : Princeton NJ Meetup 2013-10-22T02:10:25.174Z
Giving What We Can September Internship 2013-02-18T20:03:56.667Z
[minor] Separate Upvotes and Downvotes Implimented 2013-01-29T10:31:21.726Z
[Link]: 80,000 hours blog 2012-02-26T14:34:58.457Z
Counterfactual Coalitions 2012-02-16T21:42:52.639Z
Best Intro to LW article for transhumanists 2011-10-28T02:30:35.471Z
In Defense of Objective Bayesianism: MaxEnt Puzzle. 2011-01-06T00:56:50.739Z
Link: Facing the Mind-Killer 2010-12-18T00:57:08.114Z
Oxford (UK) Rationality & AI Risks Discussion Group 2010-11-02T19:10:30.494Z
A Player of Games 2010-09-23T22:52:38.849Z
Burning Man Meetup: Bayes Camp 2010-08-25T06:14:54.005Z

Comments

Comment by Larks on Let’s think about slowing down AI · 2024-01-17T18:33:30.860Z · LW · GW

This post seems like it was quite influential. This is basically a trivial review to allow the post to be voted on.

Comment by Larks on Effective Aspersions: How the Nonlinear Investigation Went Wrong · 2023-12-22T03:07:17.373Z · LW · GW

L'Ésswrong, c'est moi.

Comment by Larks on Nonlinear’s Evidence: Debunking False and Misleading Claims · 2023-12-17T21:07:57.014Z · LW · GW

I agree in general, but think the force of this is weaker in this specific instance because NonLinear seems like a really small org. Most of the issues raised seem to be associated with in-person work and I would be surprised if NonLinear ever went above 10 in-person employees. So at most this seems like one order of magnitude in difference. Clearly the case is different for major corporations or orgs that directly interact with many more people. 

Comment by Larks on Nonlinear’s Evidence: Debunking False and Misleading Claims · 2023-12-17T03:47:56.173Z · LW · GW

I think there will be some degree to which clearly demonstrating that false accusations were made will ripple out into the social graph naturally (even with the anonymization), and will have consequences. I also think there are some ways to privately reach out to some smaller subset of people who might have a particularly good reason to know about this. 

If this is an acceptable resolution, why didn't you just let the problems with NonLinear ripply out into the social graph naturally?

Comment by Larks on Sharing Information About Nonlinear · 2023-09-13T13:34:51.016Z · LW · GW

If most firms have these clauses, one firm doesn't, and most people don't understand this, it seems possible that most people would end up with a less accurate impression of their relative merits than if all firms had been subject to equivalent evidence filtering effects.

In particular, it seems like this might matter for Wave if most of their hiring is from non-EA/LW people who are comparing them against random other normal companies.

Comment by Larks on 2021 AI Alignment Literature Review and Charity Comparison · 2023-03-01T04:33:25.982Z · LW · GW

Sorry, not for 2022.

Comment by Larks on Larks's Shortform · 2022-09-01T17:28:25.127Z · LW · GW

I would typically aim for mid-December, in time for the American charitable giving season.

Comment by Larks on Larks's Shortform · 2022-08-30T13:43:39.301Z · LW · GW

After having written an annual review of AI safety organisations for six years, I intend to stop this year. I'm sharing this in case someone else wanted to in my stead.

Reasons

  • It is very time consuming and I am busy.
  • I have a lot of conflicts of interests now.
  • The space is much better funded by large donors than when I started. As a small donor, it seems like you either donate to:
    • A large org that OP/FTX/etc. support, in which case funging is ~ total and you can probably just support any.
    • A large org than OP/FTX/etc. reject in which case there is a high chance you are wrong.
    • A small org OP/FTX/etc. haven't heard of, in which case I probably can't help you either.
  • Part of my motivation was to ensure I stayed involved in the community but this is not at threat now.

Hopefully it was helpful to people over the years. If you have any questions feel free to reach out.

Comment by Larks on Common misconceptions about OpenAI · 2022-08-26T01:23:42.189Z · LW · GW

Thanks!

Comment by Larks on Common misconceptions about OpenAI · 2022-08-25T19:01:20.063Z · LW · GW

Alignment research: 30

Could you share some breakdown for what these people work on? Does this include things like the 'anti-bias' prompt engineering?

Comment by Larks on Accounting For College Costs · 2022-05-04T00:39:10.849Z · LW · GW

I would expect that to be the case for staff who truly support faculty. But many of them seem to be there to directly support students, rather than via faculty. The number of student mental health coordinators (and so on) you need doesn't scale with the number of faculty you have. The largest increase in this category is 'student services', which seems to be definitely of this nature.

Comment by Larks on Accounting For College Costs · 2022-05-03T21:01:45.675Z · LW · GW

Thanks very much for writing this very diligent analysis.

I think you do a good job of analyzing the student/faculty ratio, but unless I have misread it seems like this is only about half the answer. 'Support' expenses rose by even more than 'Instruction', and the former category seems less linked to the diversity of courses offered than to things like the proliferation of Deans, student welfare initiatives, fancy buildings, etc.

Comment by Larks on 2021 AI Alignment Literature Review and Charity Comparison · 2021-12-26T17:23:18.391Z · LW · GW

Thanks, that's very kind of you!

Comment by Larks on 2021 AI Alignment Literature Review and Charity Comparison · 2021-12-24T22:31:18.665Z · LW · GW

Is your argument about personnel overlap that one could do some sort of mixed effect regression, with location as the primary independent variable and controls for individual productivity? If so I'm so somewhat skeptical about the tractability: the sample size is not that big, the data seems messy, and I'm not sure it would capture necessarily the fundamental thing we care about. I'd be interested in the results if you wanted to give it a go though!

More importantly, I'm not sure this analysis would be that useful. Geography-based-priors only really seem useful for factors we can't directly observe; for an organization like CHAI our direct observations will almost entirely screen off this prior. The prior is only really important for factors where direct measurement is difficult, and hence we can't update away from the prior, but for those we can't do the regression. (Though I guess we could do the regression on known firms/researchers and extrapolate to new unknown orgs/individuals).

The way this plays out here is we've already spent the vast majority of the article examining the research productivity of the organizations; geography based priors only matter insomuchas you think they can proxy for something else that is not captured in this.

As befits this being a somewhat secondary factor, it's worth noting that I think (though I haven't explicitly checked) in the past I have supported bay area organisations more than non-bay-area ones.   

Comment by Larks on 2021 AI Alignment Literature Review and Charity Comparison · 2021-12-24T21:42:51.618Z · LW · GW

Thanks, fixed in both copies.

Comment by Larks on 2021 AI Alignment Literature Review and Charity Comparison · 2021-12-24T11:40:08.723Z · LW · GW

Thanks, fixed.

Comment by Larks on 2021 AI Alignment Literature Review and Charity Comparison · 2021-12-23T17:51:08.573Z · LW · GW

Should be fixed, thanks.

Comment by Larks on 2021 AI Alignment Literature Review and Charity Comparison · 2021-12-23T17:11:04.926Z · LW · GW

Changed in both copies as you request.

Comment by Larks on 2021 AI Alignment Literature Review and Charity Comparison · 2021-12-23T17:06:01.089Z · LW · GW
  • I prioritized posts by named organizations.
    • Diffractor does not list any institutional affiliations on his user page.
    • No institution I noticed listed the post/sequence on their 'research' page.
    • No institution I contacted mentioned the post/sequence.
  • No post in the sequence was that high in the list of 2021 Alignment Forum posts, sorted by karma.
  • Several other filtering methods also did not identify the post

However upon reflection it does seem to be MIRI-affiliated so perhaps should have been affiliated; if I have time I may review and edit it in later.

Comment by Larks on Rationalist Storybooks: A Challenge · 2021-11-11T04:29:47.875Z · LW · GW

13 years later: did anyone end up actually making such a book?

Comment by Larks on Covid 7/8: Delta Takes Over · 2021-07-08T15:15:53.293Z · LW · GW

The labels on the life satisfaction chart appear to be wrong; January 2021 comes before December 2020.

Comment by Larks on Covid 6/17: One Last Scare · 2021-06-18T04:36:29.940Z · LW · GW

Yes.

Comment by Larks on 2 innovative life extension approaches using cryonics technology · 2021-04-02T01:33:32.387Z · LW · GW

Well, with hemispherectomy, those problems are no more. Hemispherectomy is a procedure where half of the brain is removed. It has been performed multiple times without any apparent complications (example).

I was skeptical until I read the example. Now I am convinced!

Comment by Larks on Demand offsetting · 2021-03-22T03:38:41.640Z · LW · GW

It's hard to sell 1 million eggs for one price, and 1 million for another price.

Are you sure this is the case? It's common for B2B transactions to feature highly customised and secret pricing and discounts. And in this case they're not selling the same product from the customer's point of view: one buyer gets a million ethical eggs, while another gets a million ordinary (from their point of view) eggs.

Comment by Larks on Why aren't we all using Taffix? · 2021-03-01T18:40:40.396Z · LW · GW

Thanks for writing this; ordered.

Comment by Larks on How my school gamed the stats · 2021-02-22T15:33:31.342Z · LW · GW

A teacher in year 9 walked up to a student who was talking, picked them up and threw them out of an (open) first floor window. 

Worth clarifying for US readers that 'first floor' in the UK would be 'second floor' in the US, because UK floor indexing starts at zero. So this event is much worse than it sounds.

Comment by Larks on How Should We Respond to Cade Metz? · 2021-02-14T02:12:28.204Z · LW · GW

Scott wrote a response.

Comment by Larks on 2020 AI Alignment Literature Review and Charity Comparison · 2020-12-25T22:29:06.886Z · LW · GW

Thanks, added.

Comment by Larks on Covid 12/24: We’re F***ed, It’s Over · 2020-12-24T22:24:48.895Z · LW · GW

At the moment, the poor person and the rich person are both buying things. If the rich person buys more vaccine, that means they will buy less of the other things, so the poor person will be able to have more of them. So the question is about the ratios of how much the two guys care about the vaccine and how much they care about the other thing... and the answer is the rich guy will pay up for the vaccine when his vaccine:other ratio is higher than the other guys. This is the efficient allocation.

It might be the case that it is separately desirable to redistribute wealth from the rich guy to the poor guy. This would indeed allow the poor guy to buy more things. But, conditional on a certain wealth distribution, it is best to allow market forces to allocate goods within that distribution. 

(For simplicity I have ignored macroeconomics in this post, but the same argument broadly goes through if you don't.)

Comment by Larks on 2020 AI Alignment Literature Review and Charity Comparison · 2020-12-22T02:08:40.462Z · LW · GW

Hey Daniel, thanks very much for the comment. In my database I have you down as class of 2020, hence out of scope for that analysis, which was class of 2018 only. I didn't include 2019 or 2020 classes because I figured it takes time to find your footing, do research, write it up etc., so absence of evidence would not be very strong evidence of absence. So please don't consider this as any reflection on you. Ironically I actually did review one of your papers in the above - this one - which I did indeed think was pretty relevant! (Cntrl-F 'Hendrycks' to find the paragraph in the article). Sorry if this was not clear from the text.

Comment by Larks on The Darwin Game - Rounds 10 to 20 · 2020-11-18T05:12:44.955Z · LW · GW

Larks, excellent name choice for your AttackBot.

Thanks! I figured it was in the spirit of a DefectBot to defect linguistically as well, and there was a tiny chance someone might be doing naive string-matching. 

Comment by Larks on The Darwin Game - Rounds 0 to 10 · 2020-10-25T04:02:44.678Z · LW · GW

You will have to wait for next time's obituary I'm afraid! I think Isusr should have a good grasp on the philosophical and ethical traditions I was attempting to channel with CooperateBot - while the insights are deep, I think the lengthy code is quite clear on the matter.

Comment by Larks on The Darwin Game - Rounds 0 to 10 · 2020-10-24T16:20:30.793Z · LW · GW

I actually have no idea - I guess we are just two naturally very cooperative people!

Comment by Larks on The Darwin Game - Rounds 0 to 10 · 2020-10-24T05:14:56.279Z · LW · GW

Cool competition! It makes me wish I had had more time to put into CooperateBot. At present I would say it instantiated a relatively naive view of cooperation, and could do much better if I invested more time considering the true nature of generosity. Looking at the obituary I suspect that CooperateBot may not last much longer.

Comment by Larks on Should I prefer to get a tax refund, or not to? · 2020-10-23T00:55:22.934Z · LW · GW

Holding constant the total amount of taxes you pay, it is better not to get a refund. This is the perspective you should take at the beginning of the year. 

Holding constant the amount of taxes you have already paid, it is better to get a refund. This is the perspective you should take at the end of the year. 

Comment by Larks on Covid-19 6/4: The Eye of the Storm · 2020-06-05T02:27:05.227Z · LW · GW

I attempted to produce a rough estimate of this here (excerpted below):

... One (BERI funded!) study suggested that banning large gatherings reduced r0 by around 28%.
Unfortunately, protests seem in many ways ideal for spreading the disease. They involve a large number of people in a relatively small area for an extended period of time. Even protests which were advertised as being socially distanced often do not end up that way. While many people wear masks, photos of protests make clear that many do not, and those that are are often using cloth masks that are significantly less effective than surgical or n95s in the face of repeated exposure. Additionally, protests often involve people shouting or chanting, which cause infectious droplets to be released from people's mouths. Exposure to tear gas can apparently also increase susceptibility, as well as cramped indoor conditions for those arrested.
It's hard to estimate how many new cases will be caused by the protests, because there doesn’t seem to be good statistics on the number of people at protests, so we can't model the physical dynamics easily. A simple method would be to assume we have lost the benefits of the ban on large gatherings over the last week or so. On the one hand, this may be an over-estimate, because fortunately most people continue to socially distance, and protests take place mainly outside. On the other hand, protesters are actively seeking out (encouraging others to seek out) boisterous large gatherings in a way they were not pre-March, which could make things even worse. On net I suspect it may under-estimate the incremental spread, but given the paucity of other statistics we will use it as our central scenario.
If the r0 was around 0.9 before, this suggests the protests might have temporarily increased it to around 1.25, and hopefully it will quickly return to 0.9 after the protests end. Even if we assume no chain infections during the protest - so no-one who has been infected at a protest goes on to infect another protester - this means the next step in disease prevalence would be a 25% increase instead of a 10% decrease. Unfortunately the exponential nature of infection means this will have a large impact. If you assume around 1% of the US was infected previously, had we stayed on the previous r0=0.9 we would end up with around 9% more of the population infected from here on before the disease was fully suppressed. In contrast, with this one-time step-up in r0, we will see around 12.5% of the population infected from here - an additional 3.5% of the population.
Assuming an IFR of around 0.66%, that's a change from around 190,000 deaths to more like 265,000. Protesters skew younger than average, suggesting that this IFR may be an over-estimate, but on the other hand, they are also disproportionately African American, who seem to be more susceptible to the disease, and the people they go on to infect will include older people.
Comment by Larks on 2018 AI Alignment Literature Review and Charity Comparison · 2019-12-19T03:02:37.941Z · LW · GW

See next year's post here.

Comment by Larks on Free Money at PredictIt? · 2019-09-26T22:42:43.311Z · LW · GW

I still found this helpful as it allowed me to exit my directional Yang and Buttigieg positions with negative transaction cost.

Comment by Larks on Honoring Petrov Day on LessWrong, in 2019 · 2019-09-26T22:10:21.171Z · LW · GW

I would like to add that I think this is bad (and have the codes). We are trying to build social norms around not destroying the world; you are blithely defecting against that.

Comment by Larks on Power Buys You Distance From The Crime · 2019-08-03T16:19:46.446Z · LW · GW
This case is more complicated than the corporate cases because the powerful person (me) was getting merely the appearance of what she wanted (a genuine relationship with a compatible person), not the real thing. And because the exploited party was either me or Connor, not a third party like bank customers. No one thinks the Wells Fargo CEO was a victim the way I arguably was.

I think you have misunderstood the Wells Fargo case. These fake accounts generally didn't bring in any material revenue; they were just about internal 'number of new accounts targets'. It was directly a case of bank employees being incentivised to defraud management and investors, which they then did. If ordinary Wells employees had not behaved fraudulently, all the targets would have been missed, informing management/investors about their mis-calibration, and more appropriate targets would have been set. In this case power didn't buy distance from the crime, but only in the sense that it meant you couldn't tell you were being cheated.

For more on this I recommend the prolific Matt Levine:

There's a standard story in most bank scandals, in which small groups of highly paid traders gleefully and ungrammatically conspire to rip-off customers and make a lot of money for themselves and their bank. This isn't that. This looks more like a vast uprising of low-paid and ill-treated Wells Fargo employees against their bosses.
...
So that's about 2.1 million fake deposit and credit-card accounts, of which about 100,000 -- fewer than 5 percent -- brought in any fee income to Wells Fargo. The total fee income was $2.4 million, or about $1.14 per fake account. And that overstates the profitability: Wells Fargo also enrolled people for debit cards and online banking, but the CFPB doesn't bother to count those incidents, or suggest that any of them led to any fees. Which makes sense: You'd expect online banking and debit cards to be free, if you never use them or even know about them. Meanwhile, all this dumb stuff seems to have occupied huge amounts of employee time that could have been spent on more productive activities. If you divide the $2.4 million among the 5,300 employees fired for setting up fake accounts, you get about $450 per employee. Presumably it cost Wells Fargo way more than that just to replace them.
In the abstract, you can see why Wells Fargo would emphasize cross-selling of multiple "solutions" to customers. It is a good sales practice; it both indicates and encourages customer loyalty. If your customers have a checking account, and a savings account, and a credit card and online banking, all in one place, then they'll probably use each of those products more than if they had only one. And when they want a new, lucrative product -- a mortgage, say, or investment advice -- they're more likely to turn to the bank where they keep the rest of their financial life. 
But obviously no one in senior management wanted this. Signing customers up for online banking without telling them about it doesn't help Wells Fargo at all. No one feels extra loyalty because they have a banking product that they don't use or know about. Even signing them up for a credit card without telling them about it generally doesn't help Wells Fargo, because people don't use credit cards that they don't know about. Cards with an annual fee are a different story -- at least you can charge them the fee! -- but it seems like customers weren't signed up for many of those.  This isn't a case of management pushing for something profitable and getting what they asked for, albeit in a regrettable and illegal way. This is a case of management pushing for something profitable but difficult, and the workers pushing back with something worthless but easy.

Comment by Larks on Financial engineering for funding drug research · 2019-05-17T03:04:28.949Z · LW · GW

Is this very different from founding a pharmaceutical company?

Comment by Larks on Strategic implications of AIs' ability to coordinate at low cost, for example by merging · 2019-05-02T20:51:24.945Z · LW · GW

Critch wrote a related paper:

Existing multi-objective reinforcement learning (MORL) algorithms do not account for objectives that arise from players with differing beliefs.Concretely, consider two players with different beliefs and utility functions who may cooperate to build a machine that takes actions on their behalf. A representation is needed for how much the machine’s policy will prioritize each player’s interests over time. Assuming the players have reached common knowledge of their situation, this paper derives a recursion that any Pareto optimal policy must satisfy. Two qualitative observations can be made from the recursion: the machine must (1) use each player’s own beliefs in evaluating how well an action will serve that player’s utility function, and (2) shift the relative priority it assigns to each player’s expected utilities over time, by a factor proportional to how well that player’s beliefs predict the machine’s inputs. Observation (2) represents a substantial divergence from naive linear utility aggregation (as in Harsanyi’s utilitarian theorem, and existing MORL algorithms), which is shown here to be inadequate for Pareto optimal sequential decision-making on behalf of players with different beliefs.

Toward negotiable reinforcement learning: shifting priorities in Pareto optimal sequential decision-making

Comment by Larks on Strategic implications of AIs' ability to coordinate at low cost, for example by merging · 2019-05-02T20:44:47.325Z · LW · GW

War only happens if two agents don’t have common knowledge about who would win (otherwise they’d agree to skip the costs of war).

They might also have poorly aligned incentives, like a war between two countries that allows both governments to gain power and prestige, at the cost of destruction that is borne by the ordinary people of both countries. But this sort of principle-agent problem also seems like something AIs should be better at dealing with.

Comment by Larks on Literature Review: Distributed Teams · 2019-04-16T16:06:48.856Z · LW · GW

In light of this:

Build over-communication into the process.
In particular, don’t let silence carry information. Silence can be interpreted a million different ways (Cramton 2001).

Thanks for writing this! I found it very interesting, and I like the style. I particularly hadn't properly appreciated how semi-distributed was worth than either extreme. It's disappointing to hear, but seemingly obvious in retrospect and good to know.

Comment by Larks on 2018 AI Alignment Literature Review and Charity Comparison · 2019-01-05T18:45:23.520Z · LW · GW

Thanks for sharing, seems like a reasonable take to me.

Comment by Larks on 2018 AI Alignment Literature Review and Charity Comparison · 2019-01-05T18:43:10.417Z · LW · GW

I definitely being near AI hubs is helpful, and I'd be interested in supporting any credible new groups that started in other hubs.

Thanks for that extra info on CHAI staff. In general my objections to the bay area are partly about the EA/LW culture there, and partly about the broader culture. I did end up donating to CHAI despite this!

Comment by Larks on 2018 AI Alignment Literature Review and Charity Comparison · 2019-01-05T18:22:33.116Z · LW · GW

Thanks! I have fixed most of the typos.

Comment by Larks on In Defense of Finance · 2018-12-19T00:37:01.560Z · LW · GW

Especially as in recent years investor preferences for lower risk, low beta, low vol, higher yield stocks (like utilities and staples) has been well documented as strong.

One related fact is that (at least some of the banks) feel like they haven't gotten credit for the risk reductions they have done. Given their lower leverage now, they feel like they should have higher multiples, whereas actually their relative multiples have compressed vs the market vs pre-crisis. Of course, this may be because their risk was under-estimated pre-crisis, so their relative multiple was too high.

Comment by Larks on In Defense of Finance · 2018-12-19T00:33:25.226Z · LW · GW

At least in public equity markets the size premium is very weak. My team looked at it and decided not to use it as a factor to guide our investing. It's closely correlated to an illiquidity premia, which I do believe in.

Comment by Larks on Get genotyped for free ( If your IQ is high enough) · 2018-12-15T19:04:15.139Z · LW · GW

Update: Jeff says they only ever sent him part of his data.