Running Effective Structured Forecasting Sessions 2019-09-06T21:30:25.829Z · score: 21 (5 votes)
How to write good AI forecasting questions + Question Database (Forecasting infrastructure, part 3) 2019-09-03T14:50:59.288Z · score: 31 (12 votes)
AI Forecasting Resolution Council (Forecasting infrastructure, part 2) 2019-08-29T17:35:26.962Z · score: 30 (12 votes)
Could we solve this email mess if we all moved to paid emails? 2019-08-11T16:31:10.698Z · score: 32 (15 votes)
AI Forecasting Dictionary (Forecasting infrastructure, part 1) 2019-08-08T16:10:51.516Z · score: 41 (22 votes)
Conversation on forecasting with Vaniver and Ozzie Gooen 2019-07-30T11:16:58.633Z · score: 41 (10 votes)
Does improved introspection cause rationalisation to become less noticeable? 2019-07-30T10:03:00.202Z · score: 28 (8 votes)
Prediction as coordination 2019-07-23T06:19:40.038Z · score: 43 (13 votes)
jacobjacob's Shortform Feed 2019-07-23T02:56:35.132Z · score: 15 (2 votes)
When does adding more people reliably make a system better? 2019-07-19T04:21:06.287Z · score: 35 (10 votes)
How can guesstimates work? 2019-07-10T19:33:46.002Z · score: 26 (8 votes)
Can we use ideas from ecosystem management to cultivate a healthy rationality memespace? 2019-06-13T12:38:42.809Z · score: 37 (7 votes)
AI Forecasting online workshop 2019-05-10T14:54:14.560Z · score: 32 (6 votes)
What are CAIS' boldest near/medium-term predictions? 2019-03-28T13:14:32.800Z · score: 32 (9 votes)
Formalising continuous info cascades? [Info-cascade series] 2019-03-13T10:55:46.133Z · score: 17 (4 votes)
How large is the harm from info-cascades? [Info-cascade series] 2019-03-13T10:55:38.872Z · score: 23 (4 votes)
How can we respond to info-cascades? [Info-cascade series] 2019-03-13T10:55:25.685Z · score: 15 (3 votes)
Distribution of info-cascades across fields? [Info-cascade series] 2019-03-13T10:55:17.194Z · score: 15 (3 votes)
Understanding information cascades 2019-03-13T10:55:05.932Z · score: 55 (19 votes)
Unconscious Economics 2019-02-27T12:58:50.320Z · score: 70 (27 votes)
How important is it that LW has an unlimited supply of karma? 2019-02-11T01:41:51.797Z · score: 30 (12 votes)
When should we expect the education bubble to pop? How can we short it? 2019-02-09T21:39:10.918Z · score: 41 (12 votes)
What is a reasonable outside view for the fate of social movements? 2019-01-04T00:21:20.603Z · score: 36 (12 votes)
List of previous prediction market projects 2018-10-22T00:45:01.425Z · score: 33 (9 votes)
Four kinds of problems 2018-08-21T23:01:51.339Z · score: 41 (19 votes)
Brains and backprop: a key timeline crux 2018-03-09T22:13:05.432Z · score: 88 (23 votes)
The Copernican Revolution from the Inside 2017-11-01T10:51:50.127Z · score: 142 (68 votes)


Comment by jacobjacob on Rationality and Levels of Intervention · 2019-10-12T10:55:22.782Z · score: 2 (1 votes) · LW · GW

Nitpick. Mildly triggered by:

These are posts about moving from thought A to thought B, and whether thought B is allowed given thought A.

“Allowed” is of course a very social term, and one that sounds a lot like “will my teacher accept it if I make this inference?”

Which is different from the mathematical mindset of what happens if I make that inference, and is that things interesting/elegant/useful. What does it capture to have those kinds of inference rules, and does it capture the kind of process I want to run or not?

Moreover, when it comes to Bayesian reasoning and its various generalisations, the correct inference is _inevitable_, and not optional. There is one single credence which is correct to hold given your priors and the evidence you’ve observed. (Compare this to old school rationality, like Popper and Feynman, thought more in terms of you being “allowed” to hold a variety of beliefs as long as you hadn’t been refuted by experiment. I can’t find the reference post for this now, though.)

Comment by jacobjacob on Rationality and Levels of Intervention · 2019-10-12T10:54:12.031Z · score: 6 (3 votes) · LW · GW

Just riffing a bit on the same project you started :)

There’s integrity and accountability -- integrity (Level 3) as following a certain decision theory and making it common knowledge that you do, such that others can reliably simulate you, and coordinate and make trades with you; and accountability as choosing who you want to do your individual holistic regulation (Level 4).

On another note, predictions and calibration training is often pitched as a kind of Level 1/2 intervention, but I’m more bullish on it as a Level 2 intervention with important Level 5 consequences.

It’s certainly often helpful to quantify your beliefs, and to form an all-things-considered opinion as an ensemble model of all the things you might trust. But to restrict your trains-of-thought to always follow an all-things-considered view, never veering off into resonating with a single model or world-view, is, as you point out, not that great. However, spreading the meme of being able to zoom out to an all-things-considered, quantitative opinion when necessary, and engaging with that level regularly enough to build a track-record of being able to do that, seems like a core part of having a healthy Bayesian community, even if you actually use it quite infrequently compared to other modes of thinking (just like professional mathematicians riff on a post-rigorous level but can drop down to the rigorous level when need be). This is part of my current framing for the forecasting class I’m teaching at CFAR mainlines.

There’s also a long list of other CFAR techniques one could analyse.

Eliezer’s and Abram’s posts are interesting Level 1 interventions, but look at lot like improvements to your slow, deliberate, conscious thinking processes, perhaps eventually becoming ingrained in your S1. I’d compare that with TAPs, which seem to intervene quite directly at Level 2 (and probably with backchaining effects to Level 1): “what thoughts do I want to follow from other thoughts?” [1]

This also seems to me to be the core of what makes CBT therapy work, whereby you uncover unwanted trains (“Get invite to social event” → “Visualise public shame from making an embarrassing comment” → “Flinch away from invite”), and then intervene to change their trajectory.

This causes the question of whether there are any more direct interventions at Level 1. Interventions determining which thoughts, in and of themselves, are even desirable or not. I interpret Selective reporting and Lines of retreat as analysing such interventions. The former (a bit extrapolated) as noting that if there are some unitary thoughts we cannot think, regardless of whether we actually believe them, this can cause large mistakes elsewhere in our belief system. The latter tries to tackle the problem when the blocker is motivational rather than social, by embedding the thoughts in conditionals and building a backup plan before considering whether it has to be used.

Then there's goal factoring, closely related to separation of concerns. Don't take actions which confusedly optimise for orthogonal goals, separate out your desires and optimize them separately. This probably has implications at Levels 1 through 4.

I could go on through the CFAR techniques and might at a later point, but that will do for now.

[1] This looks more like “epistemic TAPs”, or “internal TAPs”, which haven’t yet become a standard part of the mainline curriculum, where TAPs are often more external, and for things like “Deciding to take the stairs instead of the elevator as soon as I come into the office and look at them”.

Comment by jacobjacob on Rationality and Levels of Intervention · 2019-10-11T14:36:42.679Z · score: 4 (2 votes) · LW · GW

As we’re thinking about _intervention_, we’re hoping to _change_ something, or accomplish some _effect_. And in this vein, it’s interesting to note how the levels aren’t that independent.

For example, incentives tend to backpropagate from one level to the other. I expect that if you regularly give someone negative reinforcement for expressing half-formed ideas (Level 3 intervention), they might not just stop expressing ideas, but also stop _having_ original ideas altogether (Level 1 / 2 effect).

Or if you establish a meme of sharing the causes of your beliefs (Level 3 intervention), your community as a whole will run into fewer info-cascades (Level 5 effect).

Some of the most powerful interventions are those which create loops between levels. Helping people become stronger rationalists (Level 1 / 2) will enable them to make important changes to their and their community’s environment (Level 4 / 5) which will then feedback into their ability to think true thoughts and enact further important changes.

Similarly, bad equilibria emerge when Level 5 interventions change the optimal strategy at Level 3, and people doubling down on that then further entrenches those Level 5 changes.

Comment by jacobjacob on Introduction to Introduction to Category Theory · 2019-10-07T00:50:08.333Z · score: 11 (6 votes) · LW · GW

This sounds like a very exciting project and a solution to an open exposition problem.

I look forward to reading the posts!

I also wonder if you know what kinds of things would motivate you to write them?

Interesting discussion? Getting data about your ability to tutor concepts? Money (maybe we could organize a Patreon with interested LW users if so)?

Comment by jacobjacob on Honoring Petrov Day on LessWrong, in 2019 · 2019-09-27T10:18:55.885Z · score: 2 (1 votes) · LW · GW

Ah. I thought it was the entire site. (Though it did say "Frontpage" in the post.)

Comment by jacobjacob on Honoring Petrov Day on LessWrong, in 2019 · 2019-09-27T09:54:18.650Z · score: 26 (11 votes) · LW · GW

All your reasons look like People Are Bad. I think it suffices that The World is Complex and Coordination is Hard.

Consider, for example:

  • Someone thinks Petrov day is not actually a good ritual and wants to make a statement about this
  • Someone thinks the reasoning exhibited in OP/comments is naïve and wouldn't stand up to the real test, and so wants to punish people/teach them a lesson about this
  • Someone comes up with a clever argument involving logical decision theories and Everett branches meaning they should push... but they made a mistake and the argument is wrong
  • Someone thinks promising but unstable person X is about to press the button, and that this would be really bad for X's community position, and so instead they take it upon themselves to press the button to enable the promising but unstable person to be redeemed and flourish
  • Someone accidentally gives away/loses their launch codes (e.g. just keeps their gmail inbox open at work)
  • A group of people tries to set up a scheme to reliable prevent a launch, however this grows increasingly hard and confusing and eventually escalates into one of the above failure modes
  • Several people try to set up precommitments that will guarantee a stable equilibrium; however, one of them makes a mistake and interpret the result as launching being the only way for them to follow-through on it
  • Someone who feels that "this is trivial if there's no incentive to press the button" tries to "make things more interesting" and sets off a chain of events that culmninates in a failure mode like the above
  • ...

Generating this list only took a few minutes and wasn't that high effort. Lots of the examples have a lot of ways of being realised.

So overall, adding karma bounty for launching could be cool, but I don't think it's as necessary as some people think.

Comment by jacobjacob on [Site Feature] Link Previews · 2019-09-19T13:29:38.898Z · score: 3 (2 votes) · LW · GW

I liked the little "LW".

Though why does it appear at the end of the link?

Since all links begin in the same place but end in different places there's an annoying micro-suspense before I reach the end of the link and figure out whether it's LW or not.

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-09-04T20:24:59.987Z · score: 19 (7 votes) · LW · GW

Someone tried to solve a big schlep of event organizing.

Through this app, you:

  • Pledge money when signing up to an event
  • Lose it if you don't attend
  • Get it back if you attend + a share of the money from all the no-shows

For some reason it uses crypto as the currency. I'm also not sure about the third clause, which seems to incentivise you to want others to no-show to get their deposits.

Anyway, I've heard people wanting something like this to exist and might try it myself at some future event I'll organize.

H/T Vitalik Buterin's Twitter

Comment by jacobjacob on AI Forecasting Resolution Council (Forecasting infrastructure, part 2) · 2019-08-30T18:56:46.340Z · score: 2 (1 votes) · LW · GW

Thanks for pointing that out. I was aware of such superhuman programs, but the last sentence failed to make the self-play condition sufficiently clear. Have updated it now to reflect this.

Comment by jacobjacob on Epistemic Spot Check: The Fate of Rome (Kyle Harper) · 2019-08-25T18:01:26.324Z · score: 10 (9 votes) · LW · GW

I think of it like "95% confidence interval of where the mean of Elizabeth's estimate would land after 10 hours of further research".

I've found personally this format is often useful when giving quick probability estimates. Precision is costly.

Comment by jacobjacob on What experiments would demonstrate "upper limits of augmented working memory?" · 2019-08-23T09:34:45.754Z · score: 2 (1 votes) · LW · GW

I'm curious what software this was made with, and how habryka made it readable?

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-08-14T00:28:29.812Z · score: 13 (8 votes) · LW · GW

What important book that needs fact-checking is nobody fact-checking?

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-13T22:52:06.415Z · score: 2 (1 votes) · LW · GW

I also don't know if "this answers my objection" means "oh, then I'd use it" or "other problems still seem to big" (though I'd bet on the latter).

Comment by jacobjacob on Matthew Barnett's Shortform · 2019-08-13T18:01:57.861Z · score: 3 (2 votes) · LW · GW
  • The development of modern formal logic (predicate logic, modal logic, the equivalence of higher-order logics and set-theory, etc.), which is of course deeply related to Zermelo, Fraenkel, Turing and Church, but which involved philosophers like Quine, Putnam, Russell, Kripke, Lewis and others.
  • The model of scientific progress as proceeding via pre-paradigmatic, paradigmatic, and revolutionary stages (from Kuhn, who wrote as a philosopher, though trained as a physicist)
Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-13T17:29:45.533Z · score: 5 (2 votes) · LW · GW
As I understand it, the point of a costly signal is that it's supposed to be relatively more affordable if you actually have that quality.

Yeah. In hindsight the terminology of "costly signal" is a bit unfortunate, because the payment here would actually work a bit like Mario's jump or LessWrong karma -- it's a very simple mechanism which can co-opted to solve a large number of different problems. In particular, the money is not intended to be burnt (as would be the case with status signals, or proof-of-work as mentioned in some other comments), but actually paid to you.

Overall appreciate you writing up those points, they're pretty helpful in understanding how people might (and might not) use this.

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-13T09:38:25.915Z · score: 2 (1 votes) · LW · GW

That's a great point, I will do that.

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T10:43:48.652Z · score: 2 (3 votes) · LW · GW

This was crossposted to the EA forum replacing all mentions of "rationality" with "EA", mutatis mutandis.

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T10:25:21.150Z · score: 2 (1 votes) · LW · GW

Yup, I think one of the main use cases is to enable a way of contacting people much higher status/more busy than you that doesn't require a ton of networking or makes their lives terrible. (N.b. I have lots of uncertainty over whether there's actually demand for this among that cohort, which is partly why I'm writing this.)

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T10:21:38.373Z · score: 2 (1 votes) · LW · GW

Updated the OP to clarify this. Will hold off on replying until I know whether this changes Scott's mind or not!

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T10:14:19.161Z · score: 2 (1 votes) · LW · GW

Yeah, referrals are important.

They also have some problems:

  • Requires you to already know the right people
  • If the referer doesn't know how valuable the recipient will think this is, they might just avoid it as opposed to risking burning reputation (same problem as outlined above)
  • It's still costly for the recipient. E.g. it doesn't have any way of compensating the recipient for giving them the "reduce option value" vs "slightly harm relationship" trade-off

There are probably large differences between how important each of these problems are, though I'm moderately confident that at least the first presents are real and important user case. If the options are "Pay $50 to message X" or "Try to network with people who can then introduce you to X", the first might be better and save us some social games.

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T09:54:08.788Z · score: 2 (1 votes) · LW · GW

Thanks, this is a good data-point.

Though I want to ask: if people know that you have this problem, as things stand currently, they might just avoid messaging you (since they don't have any way of compensating you for marginally making the burden on you worse)? Moreover, your time and energy are presumably exchangeable for money (though not indefinitely so)?

So it still seems paid emails might help with that?

(PS. I don't think it's only about information asymmetries and having too many emails, though I realise the OP quite strongly implies that. Might rewrite to reflect that.)

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T09:46:46.737Z · score: 3 (2 votes) · LW · GW

I don't get this.

If businesses can simply buy their way around the problem they'll do exactly that.

There's a finite amount of money such that, if you got paid that amount, you'd be happy receiving an unsolicited ad email. There's also a finite amount of money such that, if they had to pay that amount, it wouldn't be worth it for advertisers to send you an email.

In equilibrium (probably given lots of additional specification of details) this makes the content you receive worth it.

I don't see where this goes wrong in a way that's solved by PoW.

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T09:35:44.117Z · score: 6 (4 votes) · LW · GW

This seems to be saying:

"assuming a large team of full-time devs and ability to push whatever solution out to everyone who uses email, what should we build?"

which is quite different from what the post is asking for:

"should just the rationality community coordinate to make this move, to marginally adding on something to current email, using an already existing software tool?"

Comment by jacobjacob on Conversation on forecasting with Vaniver and Ozzie Gooen · 2019-08-10T18:54:01.910Z · score: 2 (1 votes) · LW · GW

I wonder whether you have any examples, or concrete case studies, of things that were successfully action-guiding to people/organisations? (Beyond forecasts and blog-posts, though those are fine to.)

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-08-09T10:56:01.318Z · score: 7 (4 votes) · LW · GW

Yup. I think it follows that when people do talk to groups in that way, they're not primarily aiming to communicate useful information in an optimal manner (if they were, there'd be better ways). They must be doing something else.

Plausible candidates are: building common knowledge and signalling of various kinds .

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-08-08T15:04:01.840Z · score: 9 (5 votes) · LW · GW

If you're building something and people's first response is "Doesn't this already exist?" that might be a good sign.

It's hard to find things that actually are good (as opposed to just seeming good), and if you succeed, there ought to be some story for why no one else got there first.

Sometimes that story might be:

People want X. A thing existed which called itself "X", so everyone assumed X already existed. If you look close enough, "X" is actually pretty different from X, but from a distance they blend together; causing people looking for ideas to look elsewhere.

Comment by jacobjacob on Benito's Shortform Feed · 2019-08-08T15:02:37.589Z · score: 20 (5 votes) · LW · GW

I'd take a bet at even odds that it's single-digit.

To clarify, I don't think this is just about grabbing power in government or military. My outside view of plans to "get a PhD in AI (safety)" seems like this to me. This was part of the reason I declined an offer to do a neuroscience PhD with Oxford/DeepMind. I didn't have any secret for why it might be plausibly crucial.

Comment by jacobjacob on The Copernican Revolution from the Inside · 2019-08-04T14:13:54.733Z · score: 6 (3 votes) · LW · GW

An interesting blog post from Sabine Hossenfelder points to a similar situation with the Michelson-Morley experiment in 1887, which set the stage for special relativity.

Comment by jacobjacob on How can we respond to info-cascades? [Info-cascade series] · 2019-08-04T13:41:04.745Z · score: 2 (1 votes) · LW · GW
A further result from the same paper is that if the actor posing the proposition can set the payout odds and the threshold in response to the common prior and known info-distribution, ...

This is a really cool result, but I'm confused about why it holds. Is the idea something like: the actor themself is uncertain about the value of the project, and the kickstarter also helps them find out whether it's worth doing, so up-cascades are costly in expectation (they might land themselves having to run some awful project)?

But if is the mechanism, it seems to apply to any rational actor using a kickstarter, as opposed to having anything to do with minimizing down-cascades?

Comment by jacobjacob on How can we respond to info-cascades? [Info-cascade series] · 2019-08-04T13:37:10.822Z · score: 2 (1 votes) · LW · GW

That's a really interesting effect, thanks for linking. I have two questions:

1) I'm confused about what the mechanism that produces the Bullwhip effect is.

One video suggested the following: as demand rapidly increases during time_step_1, suppliers aren't able to fully adapt and meet it, which causes an even larger shortage during time_step_2 and hence even larger demand; and somehow these effects compound down the supply chain.

Another mechanism is just that the demand signal is noisy, and so it's variance will increase as one moves down the supply chain. But I'm confused why this causes everything to blow up (as opposed to, say, different sub-supplieres making errors in different directions, which sorta cancel out, even though the larger variance at the end-supplier causes some volatility. That is, it's just as likely that they underestimate demand as it is that they overestimate it.)


changing viewpoints and using systems thinking

Exactly what does this imply that I, as a middle-manager somewhere in the supply chain, observing a noisy demand signal, should do? How does this concretely change my order decision from my supplier, in a way which improves things?

Comment by jacobjacob on Formalising continuous info cascades? [Info-cascade series] · 2019-08-04T12:50:56.981Z · score: 2 (1 votes) · LW · GW

We haven't, but there's some interesting stuff in the economics literature.

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-08-02T12:10:16.753Z · score: 10 (5 votes) · LW · GW

There are different kinds of information instutitions.

  • Info-generating (e.g. Science, ...)
  • Info-capturing (e.g. short-form? prediction markets?)
  • Info-sharing (e.g. postal services, newspapers, social platforms, language regulators, ...)
  • Info-preserving (e.g. libraries, archives, religions, Chesterton's fence-norms, ...)
  • Info-aggregating (e.g. prediction markets,

I hadn't considered the use of info-capturing institutions until recently, and in particular how prediction sites might help with this. If you have an insight or an update, there is cheap and standardised way to make it part of the world's share knowledge.

The cheapness means you might actually do it. And the standardisation means it will interface more easily with the other parts of the info pipeline (easier to share, easier to preserve, easier to aggregate into established wisdom...)

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-08-02T11:50:06.984Z · score: 6 (3 votes) · LW · GW

No, it's just me being surprised about how much information conversation can contain (I expected the bandwidth to be far lower, and the benefits mostly to be due to quick feedback loops). Combine that with personal experience of attending bad conferences and lame conversations, and things seem sad. But that's not the key point.

Really like your post. (Also it's crazy if true that the conversational upper bound decreases by 1 if people are gossiping, since they have to model the gossip subject.)

Comment by jacobjacob on Conversation on forecasting with Vaniver and Ozzie Gooen · 2019-08-01T11:20:40.691Z · score: 21 (6 votes) · LW · GW

Glad that you found the write-up useful!

I might disagree with your other points. There's the idea that forecasting is only valuable if it's decision-relevant, or action-guiding, and so far no forecasting org has solved this problem. But I think this is the wrong bar to beat. Making something action-guiding is really hard -- and lots of things which we do think of as important don't meet this bar.

For example, think of research. Most people at e.g. FHI don't set out to write documents that will change how Bostrom takes decisions. Rather, they seek out something that they're curious about, or that seems interesting, or just generally important... and mostly just try to have true beliefs, more than having impactful actions. They're doing research, not decisions.

Most people think essays are important and enabling people to do essays better has high impact. But if you pick a random curated LW post and ask what decision was improved as a result, I think you'll be disappointed (though not as disappointed as with forecasting questions). And this is fine. Decision-making takes in a large number of inputs, considerations, emotions, etc. which influence it in strange, non-linear ways. Its mostly just a fact about human decision-making being complex, rather than a fact about essays being useless.

So I'm thinking that the evidence that should suggest to us that forecasting is valuable is not hearing an impactful person say "I saw forecast X which caused me to change decision Y", but rather "I saw forecast X which changed my mind about topic Y". Then, downstream, there might be all sorts of actions which changed as a result, and the forecast-induced mind-change might be one out of a hundred counterfactually important inputs. Yet we shouldn't propose an isolated demand for rigor that forecasting do the credit assignment problem any better than the other 99 inputs.

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-08-01T10:57:47.325Z · score: 22 (8 votes) · LW · GW

I edited a transcript of a 1.5h conversation, and it was 10.000 words. That's roughly 1/8 of a book. Realising that a day of conversations can have more verbal content than half a book(!) seems to say something about how useless many conferences are, and how incredibly useful they could be.

Comment by jacobjacob on Conversation on forecasting with Vaniver and Ozzie Gooen · 2019-07-31T09:40:11.006Z · score: 10 (3 votes) · LW · GW

Factual correction:

Those people now... I think basically work as professional forecasters.

I don't think any of the superforecasters are full-time on forecasting, instead doing it as a contractor gig; mostly due to lack of demand for the services.

Vague impression that they mostly focus on geopolitics stuff?

Yes, initial tournament were sponsored by IARPA who cared about that, and Tetlock's earlier work in the 90's and 00's also considered expert political forecasting.

Comment by jacobjacob on Prediction as coordination · 2019-07-25T02:44:18.423Z · score: 2 (1 votes) · LW · GW

Good pointer! The idea here would be to predict the outcomes of that survey, to solve the problem of coordinating people over time, in addition to the problem of coordinating people at one point in time.

Also, Critch pointed to this in the context of surveys of the broader AI research community (in which case it might appear more dark artsy), I was pointing to use within the narrower x-risk/AI safety communities.

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-07-23T04:23:53.478Z · score: 2 (1 votes) · LW · GW

If you're asking whether your paraphrase actually captures my model, it doesn't. If you're making a point, I'm afraid I don't get it.

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-07-23T03:16:50.351Z · score: 13 (3 votes) · LW · GW

I'm confused about the effects of the internet on social groups. [1]

On the one hand... the internet enables much larger social groups (e.g. reddit communities with tens of thousands of members) and much larger circles of social influence (e.g. instagram celebrities having millions of followers). Both of these will tend to display network effects. However, A) social status is zero-sum, and B) there are diminishing returns to the health benefits of social status [2]. This suggests that on the margin moving from a very large number of small communities (e.g. bowling clubs) to a smaller number of larger communities (e.g. bowling YouTubers) will be net negative, because the benefits of status gains at the top will diminish faster than the harms from losses at the median.

On the other hand... the internet enables much more niched social groups. Ceteris paribus, this suggests it should enable more groups and higher quality groups.

I don't know how to weigh these effects, but currenly expect the former to be a fair bit larger.

[1] In addition to being confused I'm also uncertain, due to lack of data I could obtain in a few hours, probably.

[2] In contrast to the psychological benefits of status, I think social capital can sometimes have increasing returns. One mechanism: if you grow your network, the number of connections between units in your network grows even faster.

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-07-23T03:00:29.143Z · score: 16 (5 votes) · LW · GW

Had a good conversation with elityre today. Two nuggets of insight:

  • If you want to level up, find people who seem mysteriously better than you at something, and try to learn from them.
  • Work on projects which are plausibly crucial. That is, projects such that if we look back from a win scenario, it's not too implausible we'd say they were part of the reason.
Comment by jacobjacob on When does adding more people reliably make a system better? · 2019-07-19T18:56:11.775Z · score: 4 (2 votes) · LW · GW

I'm not confident that was actually the decline and shouldn't have sounded so confident in my post.

Though your explanation is confusing to me, because it doesn't explain the data-point that LW ended up having a lot of bad content and discussion, rather than no content and discussion.

Anyhow, I believe this discussion should be had in the meta section of the site, and that we should focus more on the object-level of the question here.

Comment by jacobjacob on When does adding more people reliably make a system better? · 2019-07-19T18:52:01.302Z · score: 2 (1 votes) · LW · GW

My claim is definitely not about the global financial system. It's about single financial markets becoming more accurate at tracking the metrics they're measuring as more people join, by default.

If I became convinced that the proper analogy is that companies by default become better at optimising profit as more employees join, I'd change my mind about the importance of prediction/financial markets. But I'd bet strongly that that is not the case.

Comment by jacobjacob on When does adding more people reliably make a system better? · 2019-07-19T18:49:12.538Z · score: 3 (2 votes) · LW · GW

Don't think I disagree, I've made a very similar point to yours in a previous LW thread here.

Also, my point is not that the gains from being a correct contrarian in the financial market always outweigh the social punishment for contrarianism, or that you can always trade between the two currencies. But despite being frozen out of investing, Michael Burry is still a multi-millionaire. That is an interesting observation. It's related to why I think Robin Hanson is excited about prediction markets -- they present a remarkable degree of robustness to social status games and manipulability.


Also I'm very curious about the outcome of Taleb's investments (some people say they're going awfully, which is why he's selling books...), so please share any links.

Comment by jacobjacob on A Key Power of the President is to Coordinate the Execution of Existing Concrete Plans · 2019-07-19T03:38:19.276Z · score: 7 (4 votes) · LW · GW

I'd like to coin a new term for that thing which the US President has a lot of: coordination capital.

This seems to require some combination of:

  • trust
  • long-term stability
  • Schelling point-y-ness
  • personal connections


Some properties

  • Coordination capital is depreciated as it is used

Consider the priest Kalil mentions. He's able to declare people married because people think he is. It's the equilibrium, and everyone benefits from maintaining it. But if he tests his powers and start declaring strange marriages not endorsed by the local social norm, the equilibrium might shift. Similarly, if the president tries to rally companies around a stag hunt, but does so poorly and some choose rabbit, they're all more likley to choose rabbit in future.

  • There are returns-to-scale to coordination capital

The more plan executions you successfully coordinate, the more willing future projects will be to approach you with their plans.

  • There is an upper bound to the amount of coordination capital

If you have a Schelling coordination point, and someone finds it bad and declares they will build a new, better coordination point, there is risk that you'll end up not with two but with zero coordination points. Similarly, coordination capital is scarce and it can result in lock-in scenarios if held by the wrong entities.


Background and implications

Part of the reason I want a term for this thing is that I've been experiencing a lack of this thing when working on coordination infrastructure for the EA and x-risk communities. I'm trying to build a forecasting platform and community to (among other things) build common knowledge of some timelines considerations, to coordinate around them.

However, to get people to use it, I can't just call up Holden Karnofsky, Nick Bostrom, and Nate Soares in order to kickstart the thing and make it a de facto Schelling point. Rather, I have to do some amount of "hustling", and things that don't scale -- finding people in the community with natural interest in stuff, reaching out to them personally, putting in legwork here and there to keep discussions going and add a missing piece to a quantitative model... and try to do this enough to hit some kind of escape velocity.

I don't have enough coordination capital, so I try to compensate by other means. Another example is Uber -- they're trying to move riders and drivers to a new equilibrium, they didn't have much coordination capital initially, and this requires them to burn a lot cash/free energy.

Writing this I'm a bit worried that all the leaders of the EA /x-risk communities are leaders of particular organizations with an object-level mission. They're primarily incentivised to achieve the organisation's mission, and there is no one who, like the president, simply serves to coordinate the community around the execution of plans. This suggests this function might be underutilised on the margin.

Comment by jacobjacob on How can guesstimates work? · 2019-07-11T02:00:18.539Z · score: 4 (4 votes) · LW · GW

Nitpick: "We've gotten much better at making guesstimates" and "Guesstimates have become more effective" are quite different claims, and it's not clear which one(s) you disagree with.

Comment by jacobjacob on Eight Books To Read · 2019-05-16T20:49:18.161Z · score: 12 (7 votes) · LW · GW

[Epistemic status: this comment is much less clear in elucidating the inputs rather than outputs of my thinking than I would have preferred, but I share it written roughly rather than not at all.]

On priors, it would be incredibly surprising to me if the best introduction to learning how to think about society did not include any of the progress we've made in fields like microeconomics and statistics (which only reached maturity in the last 100 years or so), or even simply empiricism and quantitative thinking (which only reached maturity in the last 500 years or so).

I believe there has been an absolutely outstanding amount of genuine conceptual and distillation progress in understanding society since Ancient Greece.

Another part of my experience feeding into this prior is that my undergrad was in philosophy at Oxford, and some professors really liked deeply studying ancient originals and criticising translations. In my experience this mostly didn't correlate with a productive or healthy epistemic culture.

Comment by jacobjacob on Understanding information cascades · 2019-05-14T09:56:59.477Z · score: 6 (3 votes) · LW · GW

This is an update on the timeline for paying out the bounties on this question. They will be awarded for work done before May 13th, but we're delayed by another few weeks in deciding on the allocation. Apologies!

Comment by jacobjacob on Swarm AI (tool) · 2019-05-03T01:46:38.533Z · score: 4 (2 votes) · LW · GW

The fact that they're measuring accuracy in a pretty bad way is evidence against them having a good algorithm.

Here's Anthony Aguirre (Metaculus) and Julia Galef on Rationally Speaking.

Anthony: On the results side, there's now an accrued track record of a couple of hundred predictions that have been resolved, and you can just look at the numbers. So, that shows that it does work quite well.
Julia: Oh, how do you measure how well it works?
Anthony: There's a few ways — going from the bad but easy to explain, to the better but harder to explain…
Julia: That's a good progression.
Anthony: And there's the worst way, which I won't even use — which is just to give you some examples of great predictions that it made. This I hate, so I won't even do it.
Julia: Good for you for shunning that.
Anthony: So looking over sort of the last half year or so, since December 1st, for example… If you ask for how many predictions was Metaculus on the right side of 50% — above 50% if it happened or below 50% if it didn't happen — that happens 77 out of 81 times the question resolved, so that's quite good.
And some of the aficionados will know about Brier scores. That's sort of the fairly easy to understand way to do it, which is that you assign a zero if something doesn't happen, and a one if something does happen. Then you take the difference between the predicted probability and that number. So if you predict at 20% and it didn't happen, you'd take that as a .2, or if it's 80% and it does happen and that's also a .2, because it's a difference between the 80% and a one, and then you square that number.
So Brier scores can run from basically zero to one, where low numbers are good. And if you calculate that for that same set of 80 questions, it's .072, which is a pretty good score.

Comment by jacobjacob on What are CAIS' boldest near/medium-term predictions? · 2019-04-04T10:13:45.744Z · score: 2 (1 votes) · LW · GW
This is a prediction I make, with "general-seeming" replaced by "more general", and I think of this as a prediction inspired much more by CAIS than by EY/Bostrom.

I notice I'm confused. My model of CAIS predicts that there would be poor returns to building general services compared to specialised ones (though this might be more of a claim about economics than a claim about the nature of intelligence).

Comment by jacobjacob on What are CAIS' boldest near/medium-term predictions? · 2019-03-29T14:01:16.820Z · score: 12 (3 votes) · LW · GW

The following exchange is also relevant:

[-] Raiden 1y link 30

Robin, or anyone who agrees with Robin:

What evidence can you imagine would convince you that AGI would go FOOM?

Reply[-] jprwg 1y link 22

While I find Robin's model more convincing than Eliezer's, I'm still pretty uncertain.

That said, two pieces of evidence that would push me somewhat strongly towards the Yudkowskian view:

  • A fairly confident scientific consensus that the human brain is actually simple and homogeneous after all. This could perhaps be the full blank-slate version of Predictive Processing as Scott Alexander discussedrecently, or something along similar lines.
  • Long-run data showing AI systems gradually increasing in capability without any increase in complexity. The AGZ example here might be part of an overall trend in that direction, but as a single data point it really doesn't say much.

Reply[-] RobinHanson 1y link 23

This seems to me a reasonable statement of the kind of evidence that would be most relevant.