Posts

Running Effective Structured Forecasting Sessions 2019-09-06T21:30:25.829Z · score: 21 (5 votes)
How to write good AI forecasting questions + Question Database (Forecasting infrastructure, part 3) 2019-09-03T14:50:59.288Z · score: 30 (13 votes)
AI Forecasting Resolution Council (Forecasting infrastructure, part 2) 2019-08-29T17:35:26.962Z · score: 30 (12 votes)
Could we solve this email mess if we all moved to paid emails? 2019-08-11T16:31:10.698Z · score: 32 (15 votes)
AI Forecasting Dictionary (Forecasting infrastructure, part 1) 2019-08-08T16:10:51.516Z · score: 41 (22 votes)
Conversation on forecasting with Vaniver and Ozzie Gooen 2019-07-30T11:16:58.633Z · score: 41 (10 votes)
Does improved introspection cause rationalisation to become less noticeable? 2019-07-30T10:03:00.202Z · score: 28 (8 votes)
Prediction as coordination 2019-07-23T06:19:40.038Z · score: 46 (14 votes)
jacobjacob's Shortform Feed 2019-07-23T02:56:35.132Z · score: 15 (2 votes)
When does adding more people reliably make a system better? 2019-07-19T04:21:06.287Z · score: 35 (10 votes)
How can guesstimates work? 2019-07-10T19:33:46.002Z · score: 26 (8 votes)
Can we use ideas from ecosystem management to cultivate a healthy rationality memespace? 2019-06-13T12:38:42.809Z · score: 37 (7 votes)
AI Forecasting online workshop 2019-05-10T14:54:14.560Z · score: 32 (6 votes)
What are CAIS' boldest near/medium-term predictions? 2019-03-28T13:14:32.800Z · score: 32 (9 votes)
Formalising continuous info cascades? [Info-cascade series] 2019-03-13T10:55:46.133Z · score: 17 (4 votes)
How large is the harm from info-cascades? [Info-cascade series] 2019-03-13T10:55:38.872Z · score: 23 (4 votes)
How can we respond to info-cascades? [Info-cascade series] 2019-03-13T10:55:25.685Z · score: 15 (3 votes)
Distribution of info-cascades across fields? [Info-cascade series] 2019-03-13T10:55:17.194Z · score: 15 (3 votes)
Understanding information cascades 2019-03-13T10:55:05.932Z · score: 55 (19 votes)
Unconscious Economics 2019-02-27T12:58:50.320Z · score: 74 (30 votes)
How important is it that LW has an unlimited supply of karma? 2019-02-11T01:41:51.797Z · score: 30 (12 votes)
When should we expect the education bubble to pop? How can we short it? 2019-02-09T21:39:10.918Z · score: 41 (12 votes)
What is a reasonable outside view for the fate of social movements? 2019-01-04T00:21:20.603Z · score: 36 (12 votes)
List of previous prediction market projects 2018-10-22T00:45:01.425Z · score: 33 (9 votes)
Four kinds of problems 2018-08-21T23:01:51.339Z · score: 41 (19 votes)
Brains and backprop: a key timeline crux 2018-03-09T22:13:05.432Z · score: 88 (23 votes)
The Copernican Revolution from the Inside 2017-11-01T10:51:50.127Z · score: 142 (68 votes)

Comments

Comment by jacobjacob on The LessWrong 2018 Review · 2019-12-11T08:32:27.998Z · score: 4 (2 votes) · LW · GW

For people who are into forecasting, I made a Foretold notebook where you can predict which posts will end up in the final Best of 2018 book.

Comment by jacobjacob on ozziegooen's Shortform · 2019-12-11T08:22:34.845Z · score: 2 (1 votes) · LW · GW

I think this paper might be relevant: https://users.cs.duke.edu/~conitzer/predictionWINE09.pdf

Abstract. A potential downside of prediction markets is that they may incentivize agents to take undesirable actions in the real world. For example, a prediction market for whether a terrorist attack will happen may incentivize terrorism, and an in-house prediction market for whether a product will be successfully released may incentivize sabotage. In this paper, we study principal-aligned prediction mechanisms– mechanisms that do not incentivize undesirable actions. We characterize all principal-aligned proper scoring rules, and we show an “overpayment” result, which roughly states that with n agents, any prediction mechanism that is principal-aligned will, in the worst case, require the principal to pay Θ(n) times as much as a mechanism that is not. We extend our model to allow uncertainties about the principal’s utility and restrictions on agents’ actions, showing a richer characterization and a similar “overpayment” result.
Comment by jacobjacob on Seeking Power is Provably Instrumentally Convergent in MDPs · 2019-12-05T10:30:50.363Z · score: 7 (4 votes) · LW · GW

That paper seems quite different from this post in important ways.

In particular, the gist of the OP seems to be something like "showing that pre-formal intuitions about instrumental convergence persist under a certain natural class of formalisations". In particular, it does so using formalism closer to standard machine learning research.

The paper you linked seems to me to instead assume that this holds true, and then apply that insight in the context of military strategy. Without speculating about the merits of that, it seems like a different thing which will appeal to different readers, and if it is important, it will be important for somewhat different reasons.

Comment by jacobjacob on In which ways have you self-improved that made you feel bad for not having done it earlier? · 2019-12-04T18:48:02.867Z · score: 5 (3 votes) · LW · GW

Swedish program called learningtosleep.se

I think they just do whatever the standard sleep cbt thing is (at least that's what they say).

Comment by jacobjacob on In which ways have you self-improved that made you feel bad for not having done it earlier? · 2019-12-04T13:39:14.241Z · score: 14 (8 votes) · LW · GW

Currently undergoing a CBT program for improved sleeping. It is literally giving me 3-4 extra productive hours a day. (By cutting down roughly 1.5h of lying in bed not sleeping, and sleeping overall 1.5h-2.5 less.) That's roughly an extra two months every year.

It's the most surprising and effective productivity thing I've done in years, and costs only $25 a week + 1h of watching instruction videos, and some energy/mental effort of sticking to the routine.

Though note that I've only done it for 2.5 weeks and expect the final improvements to land on 0-2 extra hours per day.

Comment by jacobjacob on ozziegooen's Shortform · 2019-12-04T10:01:37.100Z · score: 2 (1 votes) · LW · GW
"I'd be willing to bet $1,000 with anyone that the eventual total error of my forecasts will be less than the 65th percentile of my specified predicted error."

I think this is equivalent to applying a non-linear transformation to your proper scoring rule. When things settle, you get paid S(p) both based on the outcome of your object-level prediction p, and your meta prediction q(S(p)).

Hence:

S(p)+B(q(S(p)))

where B is the "betting scoring function".

This means getting the scoring rules to work while preserving properness will be tricky (though not necessarily impossible).

One mechanism that might help is that if each player makes one object prediction p and one meta prediction q, but for resolution you randomly sample one and only one of the two to actually pay out.

Comment by jacobjacob on Being a Robust Agent · 2019-12-02T17:37:09.794Z · score: 10 (5 votes) · LW · GW

Bumping this up to two nominations not because I think it needs a review, but because I like it and it captures an important insight that I've not seen written up like this elsewhere.

In my own life, these insights have led me to do/considering doing things like:

  • not sharing private information even with my closest friends -- in order for them to know in future that I'm the kind of agent who can keep important information (notice that there is the counterincentive that, in the moment, sharing secrets makes you feel like you have a stronger bond with someone -- even though in the long-run it is evidence to them that you are less trustworthy)
  • building robustness between past and future selves (e.g. if I was excited about and had planned for having a rest day, but then started that day by work and being really excited by work, choosing to stop work and decide to rest such that different parts of me learn that I can make and keep inter-temporal deals (even if work seems higher ev in the moment))
  • being more angry with friends (on the margin) -- to demonstrate that I have values and principles and will defend those in a predictable way, making it easier to coordinate with and trust me in future (and making it easier for me to trust others, knowing I'm capable of acting robustly to defend my values)
  • thinking about, in various domains, "What would be my limit here? What could this person do such that I would stop trusting them? What could this organisation do such that I would think their work is net negative?" and then looking back at those principles to see how things turned out
  • not sharing passwords with close friends, even for one-off things -- not because I expect them to release or lose it, but simply because it would be a security flaw that makes them more vulnerable to anyone wanting to get to me. It's a very unlikely scenario, but I'm choosing to adopt a robust policy across cases, and it seems like useful practice
Comment by jacobjacob on Anti-social Punishment · 2019-12-02T17:27:22.194Z · score: 3 (2 votes) · LW · GW

Nominating to bump this up to 2 reviews.

The insight I took from this was to beware of a scary kind of norms: not just norms those inadvertently cause equilibria, but norms which serve to maintain the equilibrium itself.

For example, requiring a publication track record in academia is intended to ensure people produce sufficiently good research output, and as a side-effect causes p-hacking and similar. However, it seems to me (though it was a while since I read it) that anti-social punishment (and relatedly, punishment of non-punishers) mostly serves to enforce the validity of the current set of norms (as opposed to some other terminal goal).

This suggests that not all norms are equally important for the stickiness of equilibria (which is an important property to keep in mind when reasoning about coordination and equilibrium-shifting), and points to some gears for predicting (and creating!) stickiness.

Comment by jacobjacob on Strategies of Personal Growth · 2019-12-02T15:39:42.191Z · score: 2 (1 votes) · LW · GW

This has been useful for me in crystallising things that I had noticed but not made as clear.

In particular, it...

  • highlights the importance of practice, the value of which I think is neglected in the rationality community. (The thing that caused me to stop neglecting it was training to become a cfar instructor, and being put in the environment where it was expected that anything you taught should be grounded in actual, real examples; and that if you didn't have any you weren't ready to teach, and it would not be unreasonable to practice a technique for 20+ hours; and that techniques are learnable skills just like playing the piano or something; and that any person working at cfar, regardless of seniority, will attend classes and think anew about the content and how they can improve)
  • deconfuses several self-improvement concepts that are often batched together
Comment by jacobjacob on Functional Institutions are the Exception · 2019-12-02T15:30:46.264Z · score: 5 (3 votes) · LW · GW

This post makes many bold claims in rapid succession, and mostly supports them via anecdotes or simple statistical/microeconomic models. It also does so in a way which sounds ambitious and wise.

These are flags suggesting that it could use a review. However, the low karma and comments might make a review less necessary.

To be clear, though...

1) I personally liked the post and believe many of the claims in the post, e.g.

  • "in any given type of institution [...] some organizations that outperform the others by orders of magnitude"
  • "Institutions we do see are functional enough to persist because of selection effects, not because humans are particularly good at making them work"
  • "It is much more difficult to make a dysfunctional institution functional than to create a functional institution from scratch"

I probably agree with it somewhat strongly on the whole.

2) I think it's unreasonable to demand that every post justifies itself, and covers all corner cases, in a perfectly rigorous way. That'll reduce the amount of content by a lot. That's not what I suggest that this post does.

However, it's at least plausible to demand that every post be able to pass review -- that is, of being able to reasonably be turned into something that is sufficiently rigorous (even if the draft wasn't).

If someone writes in this style and reliable passes reviews, it seems fine for them to keep writing in this style.

Comment by jacobjacob on How did academia ensure papers were correct in the early 20th Century? · 2019-12-02T15:18:03.723Z · score: 13 (3 votes) · LW · GW

This post:

  • Tackles an important question. In particular, it seems quite valuable to me that someone who tries to build a platform for intellectual progress attempts to build their own concrete models of the domain and try to test those against history
  • It also has a spirit of empiricism and figuring things out yourself, rather than assuming that you can't learning anything from something that isn't an academic paper
  • Those are positive attributes and contribute to good epistemic norms on the margin. Yet at the same time, a culture of unchecked amateur research could end up in bad states, and reviews seem like a useful mechanism to protect against that

This makes this suitable for a nomination.

Comment by jacobjacob on Competitive Markets as Distributed Backprop · 2019-12-02T15:11:44.094Z · score: 9 (2 votes) · LW · GW

People sometimes make grand claims like "system X is basically just Y" (10-sec off-the-cuff examples: "the human brain is just a predictive processing engine", "evolution is just an optimisation algorithm", "community interaction is just a stag hunt", "economies in disequilibrium is just gradient descent"...)

When I first saw you mention this in a comment on one my posts, I felt a similar suspicion as I usually do toward such claims.

So, to preserve high standards in an epistemic community, I think it's very valuable to actually write up the concrete, gears-like models behind claims like that -- as well as reviewing those derivations at a later time to see if they actually support the bold claim.

___

Separately, I think it's good to write-up things which seem like "someone must have thought of this" -- because academia might have a "simplicity tax", where people avoiding clearly explaining such things since it might make them look naïve.

___

For the record, I have not engaged a lot with this post personally, and it has not affected my own thinking in a substantial way.

Comment by jacobjacob on Historical forecasting: Are there ways I can get lots of data, but only up to a certain date? · 2019-11-22T20:50:01.676Z · score: 8 (4 votes) · LW · GW

I think this is a challenging and non-trivial question, which I've considered before, but I'm less pessimistic than some other commenters.

I think what we really should do is to fund someone to research and build a rigorous training set along these lines, using some kind of bias avoiding methodology (eg clever pre-registration, systematic protocols for what data to include, etc ).

I find it conceivable but very implausible that doing this will make you worse, and can certainly imagine that doing it might make you a lot better. Though most plausibly it will have a small positive effect (though that might entirely be due to the benefits of just doing deliberate practice in thinking at all).

Also Tegan McCaslin did some work on this and at one point we ran a test workshop with some superforecasters trying to predict decades of steamship development in the 19ty century based on a dataset she'd made. Could did that out for you.

Comment by jacobjacob on Understanding information cascades · 2019-11-01T15:05:03.262Z · score: 2 (1 votes) · LW · GW

We (jacobjacob and Ben Pace) decided to award $200 (out of the total bounty of $800) to this answer (and the additional comment below).

It seems to offer a learnt summary of the relevance of network science (which offers a complementary perspective on the phenomenon to the microeconomic literature linked by other commenters), which not implausibly took Jan at least an order of magnitude less time to compile than it would have taken us. (For example, the seemingly simple fact of using a different Google scholar keyword than "information cascade" might have taken several hours to realise for a non-expert.)

It also attempts to apply these to the case of forecasting (despite Jan's limited knowledge of the domain), which is a task that would likely have been even harder to do without deep experience of the field.

I'll PM Jan about payment details.

Comment by jacobjacob on Understanding information cascades · 2019-11-01T15:04:01.522Z · score: 2 (1 votes) · LW · GW

We (jacobjacob and Ben Pace) decided to award $100 (out of the total bounty of $800) to this answer.

It compiles a useful summary of the literature (we learnt a lot from going through on of the papers linked), and it attaches handy links to everything, which is a task which is on the one hand very helpful to other people, and on the other tedious and without many marginal benefits for the writer, and so likely to be under-incentivised.

I'll PM you for payment details.

Comment by jacobjacob on Understanding information cascades · 2019-11-01T15:02:41.972Z · score: 2 (1 votes) · LW · GW

We (jacobjacob and Benito) decided to award $150 (out of the total bounty of $800) to this answer (and the additional points made in the discussion).

It offers relevant and robust evidence about the role of info-cascades in forecasting environments, together with a discussion of its interpretation.

I'll PM you about payment details.

Comment by jacobjacob on How can we respond to info-cascades? [Info-cascade series] · 2019-11-01T15:01:44.580Z · score: 2 (1 votes) · LW · GW

We (jacobjacob and Ben Pace) decided to award $250 (out of the total bounty of $800) to this answer. It does several important things.

  • It references existing (and novel) work in economics and mechanism design, which might have been time-consuming to discover otherwise
  • It distills a technical paper, which is a valuable service that is usually underfunded (academic institutions comparatively incentivise novel and surprising insights)
  • The insights provided are quite action-guiding, and caused me (jacobjacob) to have ideas for how one can experiment with new kinds of forecasting tournaments that use a threshold-mechanism to change participant incentives

I'll PM you for details about payment.

Comment by jacobjacob on Understanding information cascades · 2019-11-01T14:57:41.652Z · score: 12 (3 votes) · LW · GW

UPDATE.

We (jacobjacob and Ben Pace) have finally settled on the allocation of the $800 bounty for this question. All the motivations are summarised in this comment, together with links to the relevant prize-winning answer/comment.

We will also post individual notices with motivations next to each comment for ease of discussing them.

We'll PM all prize winners to sort out logistical details of payment.

Main post

David Manheim (answer and additional points made in discussion) $150

This answer offers relevant and robust evidence about the role of info-cascades in forecasting environments, together with a discussion of its interpretation.

Jan Kulveit (answer and additional comment below) $200

This answer seems to offer a learnt summary of the relevance of network science (which offers a complementary perspective on the phenomenon to the microeconomic literature linked by other commenters), which not implausibly took Jan at least an order of magnitude less time to compile than it would have taken us. (For example, the seemingly simple fact of using a different Google scholar keyword than "information cascade" might have taken several hours to realise for a non-expert.) It also attempts to apply these to the case of forecasting (despite Jan's limited knowledge of the domain), which is a task that would likely have been even harder to do without deep experience of the field.

Pablo (1 and 2) $100

These answers compile a useful summary of the literature (we learnt a lot from going through on of the papers linked), and it attaches handy links to everything, which is a task which is on the one hand very helpful to other people, and on the other tedious and without many marginal benefits for the writer, and so likely to be under-incentivised.

Michael McLaren $50

This answer:

  • It offers a novel mechanism which is relevant to the context of intellectual progress, and ties it in with literature cited in the OP
  • Rather than just linking the paper, it distills a technical paper, which is a valuable service that is usually underfunded (academic institutions comparatively incentivise novel and surprising insights)

Ways of responding

David Manheim $50

This answer offers a practical example of a cascade-like phenomenon, which is both generally applicable and has real economic consequences. Also, the fact that it comes with a game to understand and practice responding is rare and potentially quite valuable (I (jacobjacob) am of the opinion that deliberate practice is currently a neglected virtue in the rationality/EA spheres).

rossry $250

This answer does several important things.

  • It references existing (and novel) work in economics and mechanism design, which might have been time-consuming to discover otherwise
  • It distills a technical paper, which is a valuable service that is usually underfunded (academic institutions comparatively incentivise novel and surprising insights)
  • The insights provided are quite action-guiding, and caused me (jacobjacob) to have ideas for how one can experiment with new kinds of forecasting tournaments that use a threshold-mechanism to change participant incentives
Comment by jacobjacob on How can we respond to info-cascades? [Info-cascade series] · 2019-11-01T14:54:59.451Z · score: 4 (2 votes) · LW · GW

We (jacobjacob and Benito) decided to award $50 (out of the total bounty of $800) to this answer.

It offers a practical example of a cascade-like phenomenon, which is both generally applicable and has real economic consequences. Also, the fact that it comes with a came to understand and practice responding is rare and potentially quite valuable (I'm of the opinion that deliberate practice is currently a neglected virtue in the rationality/EA spheres).

Comment by jacobjacob on How can we respond to info-cascades? [Info-cascade series] · 2019-11-01T14:53:37.925Z · score: 2 (1 votes) · LW · GW

.

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-10-31T12:29:45.367Z · score: 4 (2 votes) · LW · GW

I am quite uncertain but have an intuition that there should be an expectation of more justification accompanying stronger negative aversions (and "hate" is about as strong as it gets).

(Naturally not everything has to be fully justified, that's an unbearable demand which will stifle lots of important discourse. This is rather a point about the degree to which different things should be, and how communities should make an unfortunate trade-off to avoid Moloch when communicating aversions.)

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-10-31T12:24:25.566Z · score: 6 (3 votes) · LW · GW

Ah! I read "it" as the comment. That does change my mind about how adversarial it was.

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-10-30T13:21:27.100Z · score: 4 (4 votes) · LW · GW

Well, I strong downvoted because adversarial tone, though I'd be pretty excited about fighting about this in the right kind of way.

Curious if you could introspect/tell me more about the aversion?

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-10-29T20:38:53.828Z · score: 14 (7 votes) · LW · GW

Something interesting happens when one draws on a whiteboard ✍️while talking.

Even drawing 🌀an arbitrary squiggle while making a point makes me more likely to remember it, whereas points made without squiggles are more easily forgotten.

This is a powerful observation.

We can chunk complex ideas into simple pointers.

This means I can use 2d surfaces as a thinking tool in a new way. I don't have to process content by extending strings over time, and forcibly feeding an exact trail of thought into my mind by navigating with my eyes. Instead I can distill the entire scenario into 🔭a single, manageable, overviewable whole -- and do so in a way which leaves room for my own trails and 🕸️networks of thought.

At a glance I remember what was said, without having to spend mental effort keeping track of that. This allows me to focus more fully on what's important.

In the same way, I've started to like using emojis in 😃📄essays and other documents. They feel like a spiritual counterpart of whiteboard squiggles.

I'm quite excited about this. In future I intend to 🧪experiment more with it.



Comment by jacobjacob on What are some unpopular (non-normative) opinions that you hold? · 2019-10-24T15:51:01.549Z · score: 2 (1 votes) · LW · GW

I downvoted this even though it followed instructions, because the final sentence has a scornful tone that does not seem conducive to good-faith intellectual discourse.

Comment by jacobjacob on Testing the Efficacy of Disagreement Resolution Techniques (and a Proposal for Testing Double Crux) · 2019-10-24T13:52:52.922Z · score: 3 (2 votes) · LW · GW

Problem

The space of possible DRT-induction method pairs is much larger than this would suggest.

I think the space of things you could try is quite large indeed, both when it comes to DRT-induction as well as what you choose to include in the control condition. I can also imagine this being a major point of contention/annoyance post-study (“This is nice, but for me to really change my mind I’d want you to have used this induction/control”).

Solution

Before the experiment, we have prediction markets/forecasting tournaments on the results of the pre-registered statistical tests, given a particular induction x control combination.

When the markets close, your experiment runs as planned -- but you only test the induction x control combinations that had the most disagreement/variance in their estimates.

Prediction market participants are then paid according to a proper scoring rule based on the outcome of the experiment.

So overall, even if you just test 1-3 experimental designs, we could have these markets on 10-20 designs, and get priors for all of them!

This is also a more transparent way of picking conditions to run for the experiment.

___

I've messaged you privately to discuss this further and organise eventual funding and operational support.

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-10-23T18:59:35.570Z · score: 17 (6 votes) · LW · GW

Blackberries and bananas

Here's a simple metaphor I've recently been using in some double-cruxes about intellectual infrastructure and tools, with Eli Tyre, Ozzie Gooen, and Richard Ngo.

Short people can pick blackberries. Tall people can pick both blackberries and bananas. We can give short people lots of training to make them able to pick blackberries faster, but no amount of blackberries can replace a banana if you're trying to make banana split.

Similarly, making progress in a pre-paradigmatic field might require x number of key insights. But are those insights bananas, which can only be picked by geniuses, whereas ordinary researchers can only get us blackberries?

Or, is this metaphor false, such that having a certain number of non-geniuses + excellent tools, we can actually replicate geniuses?

This has implications for the impact of rationality training and internet intellectual infrastructure, as well as what versions of those endeavours are most promising to focus on.

Comment by jacobjacob on What Comes After Epistemic Spot Checks? · 2019-10-23T17:31:57.891Z · score: 7 (4 votes) · LW · GW

This is a good point. I think the epistemic ability to predict and evaluate arguments independently of the truth of the conclusion is something we want to heavily select for and reward, see e.g. Eliezer's writing on that here.

If Elizabeth is interested, I'm definitely interested in funding and experimenting with prediction markets on argument validity for the next round of amplifying epistemic spot checks.

Comment by jacobjacob on I would like to try double crux. · 2019-10-14T10:48:52.401Z · score: 16 (7 votes) · LW · GW

Your cruxes are formulated as "Why do I believe what I believe?"

This is quite different from what seems to me important to get at in double crux -- "What would change my mind?"

For example, case 2 is only crux if it is true that: "Were you to believe that a person can not have contact with God through prayer, then you would change your mind and think there's no God".

Is that correct?


As a metaphor, think of a ceiling supported by a few walls. It's usually the case when constructing houses that not all walls are equally important to keeping the ceiling up. Some are merely decorative -- you can knock them over and put up a new one elsewhere, and the thing will be fine. But others are load-bearing -- if you take those walls down, the ceiling itself will come crashing in.

From experience, I find something similar happens with belief. For a given belief, I can often list many arguments supporting it. And those usually that take the form "I believe X". But it often turns out that most of them aren't actually the load-bearing reason I believe it. Because were I to knock them down and stop believing them, I still would not change my mind about X. To find the ones which are actually load-bearing, it's more useful to use as a search query "If false, would this change my mind?", rather than "Do I believe this?"

Comment by jacobjacob on Rationality and Levels of Intervention · 2019-10-12T10:55:22.782Z · score: 2 (1 votes) · LW · GW

Nitpick. Mildly triggered by:

These are posts about moving from thought A to thought B, and whether thought B is allowed given thought A.

“Allowed” is of course a very social term, and one that sounds a lot like “will my teacher accept it if I make this inference?”

Which is different from the mathematical mindset of what happens if I make that inference, and is that thing interesting/elegant/useful. What does it capture to have those kinds of inference rules, and does it capture the kind of process I want to run or not?

Moreover, when it comes to Bayesian reasoning and its various generalisations, the correct inference is _inevitable_, and not optional. There is one single credence which is correct to hold given your priors and the evidence you’ve observed. (Compare this to old school rationality, like Popper and Feynman, thought more in terms of you being “allowed” to hold a variety of beliefs as long as you hadn’t been refuted by experiment. I can’t find the reference post for this now, though.)

Comment by jacobjacob on Rationality and Levels of Intervention · 2019-10-12T10:54:12.031Z · score: 6 (3 votes) · LW · GW

Just riffing a bit on the same project you started :)

There’s integrity and accountability -- integrity (Level 3) as following a certain decision theory and making it common knowledge that you do, such that others can reliably simulate you, and coordinate and make trades with you; and accountability as choosing who you want to do your individual holistic regulation (Level 4).

On another note, predictions and calibration training is often pitched as a kind of Level 1/2 intervention, but I’m more bullish on it as a Level 2 intervention with important Level 5 consequences.

It’s certainly often helpful to quantify your beliefs, and to form an all-things-considered opinion as an ensemble model of all the things you might trust. But to restrict your trains-of-thought to always follow an all-things-considered view, never veering off into resonating with a single model or world-view, is, as you point out, not that great. However, spreading the meme of being able to zoom out to an all-things-considered, quantitative opinion when necessary, and engaging with that level regularly enough to build a track-record of being able to do that, seems like a core part of having a healthy Bayesian community, even if you actually use it quite infrequently compared to other modes of thinking (just like professional mathematicians riff on a post-rigorous level but can drop down to the rigorous level when need be). This is part of my current framing for the forecasting class I’m teaching at CFAR mainlines.

There’s also a long list of other CFAR techniques one could analyse.

Eliezer’s and Abram’s posts are interesting Level 1 interventions, but look at lot like improvements to your slow, deliberate, conscious thinking processes, perhaps eventually becoming ingrained in your S1. I’d compare that with TAPs, which seem to intervene quite directly at Level 2 (and probably with backchaining effects to Level 1): “what thoughts do I want to follow from other thoughts?” [1]

This also seems to me to be the core of what makes CBT therapy work, whereby you uncover unwanted trains (“Get invite to social event” → “Visualise public shame from making an embarrassing comment” → “Flinch away from invite”), and then intervene to change their trajectory.

This causes the question of whether there are any more direct interventions at Level 1. Interventions determining which thoughts, in and of themselves, are even desirable or not. I interpret Selective reporting and Lines of retreat as analysing such interventions. The former (a bit extrapolated) as noting that if there are some unitary thoughts we cannot think, regardless of whether we actually believe them, this can cause large mistakes elsewhere in our belief system. The latter tries to tackle the problem when the blocker is motivational rather than social, by embedding the thoughts in conditionals and building a backup plan before considering whether it has to be used.

Then there's goal factoring, closely related to separation of concerns. Don't take actions which confusedly optimise for orthogonal goals, separate out your desires and optimize them separately. This probably has implications at Levels 1 through 4.

I could go on through the CFAR techniques and might at a later point, but that will do for now.

[1] This looks more like “epistemic TAPs”, or “internal TAPs”, which haven’t yet become a standard part of the mainline curriculum, where TAPs are often more external, and for things like “Deciding to take the stairs instead of the elevator as soon as I come into the office and look at them”.

Comment by jacobjacob on Rationality and Levels of Intervention · 2019-10-11T14:36:42.679Z · score: 4 (2 votes) · LW · GW

As we’re thinking about _intervention_, we’re hoping to _change_ something, or accomplish some _effect_. And in this vein, it’s interesting to note how the levels aren’t that independent.

For example, incentives tend to backpropagate from one level to the other. I expect that if you regularly give someone negative reinforcement for expressing half-formed ideas (Level 3 intervention), they might not just stop expressing ideas, but also stop _having_ original ideas altogether (Level 1 / 2 effect).

Or if you establish a meme of sharing the causes of your beliefs (Level 3 intervention), your community as a whole will run into fewer info-cascades (Level 5 effect).

Some of the most powerful interventions are those which create loops between levels. Helping people become stronger rationalists (Level 1 / 2) will enable them to make important changes to their and their community’s environment (Level 4 / 5) which will then feedback into their ability to think true thoughts and enact further important changes.

Similarly, bad equilibria emerge when Level 5 interventions change the optimal strategy at Level 3, and people doubling down on that then further entrenches those Level 5 changes.

Comment by jacobjacob on Introduction to Introduction to Category Theory · 2019-10-07T00:50:08.333Z · score: 11 (6 votes) · LW · GW

This sounds like a very exciting project and a solution to an open exposition problem.

I look forward to reading the posts!

I also wonder if you know what kinds of things would motivate you to write them?

Interesting discussion? Getting data about your ability to tutor concepts? Money (maybe we could organize a Patreon with interested LW users if so)?

Comment by jacobjacob on Honoring Petrov Day on LessWrong, in 2019 · 2019-09-27T10:18:55.885Z · score: 2 (1 votes) · LW · GW

Ah. I thought it was the entire site. (Though it did say "Frontpage" in the post.)

Comment by jacobjacob on Honoring Petrov Day on LessWrong, in 2019 · 2019-09-27T09:54:18.650Z · score: 26 (11 votes) · LW · GW

All your reasons look like People Are Bad. I think it suffices that The World is Complex and Coordination is Hard.

Consider, for example:

  • Someone thinks Petrov day is not actually a good ritual and wants to make a statement about this
  • Someone thinks the reasoning exhibited in OP/comments is naïve and wouldn't stand up to the real test, and so wants to punish people/teach them a lesson about this
  • Someone comes up with a clever argument involving logical decision theories and Everett branches meaning they should push... but they made a mistake and the argument is wrong
  • Someone thinks promising but unstable person X is about to press the button, and that this would be really bad for X's community position, and so instead they take it upon themselves to press the button to enable the promising but unstable person to be redeemed and flourish
  • Someone accidentally gives away/loses their launch codes (e.g. just keeps their gmail inbox open at work)
  • A group of people tries to set up a scheme to reliable prevent a launch, however this grows increasingly hard and confusing and eventually escalates into one of the above failure modes
  • Several people try to set up precommitments that will guarantee a stable equilibrium; however, one of them makes a mistake and interpret the result as launching being the only way for them to follow-through on it
  • Someone who feels that "this is trivial if there's no incentive to press the button" tries to "make things more interesting" and sets off a chain of events that culmninates in a failure mode like the above
  • ...

Generating this list only took a few minutes and wasn't that high effort. Lots of the examples have a lot of ways of being realised.

So overall, adding karma bounty for launching could be cool, but I don't think it's as necessary as some people think.

Comment by jacobjacob on [Site Feature] Link Previews · 2019-09-19T13:29:38.898Z · score: 3 (2 votes) · LW · GW

I liked the little "LW".

Though why does it appear at the end of the link?

Since all links begin in the same place but end in different places there's an annoying micro-suspense before I reach the end of the link and figure out whether it's LW or not.

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-09-04T20:24:59.987Z · score: 19 (7 votes) · LW · GW

Someone tried to solve a big schlep of event organizing.

Through this app, you:

  • Pledge money when signing up to an event
  • Lose it if you don't attend
  • Get it back if you attend + a share of the money from all the no-shows

For some reason it uses crypto as the currency. I'm also not sure about the third clause, which seems to incentivise you to want others to no-show to get their deposits.

Anyway, I've heard people wanting something like this to exist and might try it myself at some future event I'll organize.

https://kickback.events/

H/T Vitalik Buterin's Twitter

Comment by jacobjacob on AI Forecasting Resolution Council (Forecasting infrastructure, part 2) · 2019-08-30T18:56:46.340Z · score: 2 (1 votes) · LW · GW

Thanks for pointing that out. I was aware of such superhuman programs, but the last sentence failed to make the self-play condition sufficiently clear. Have updated it now to reflect this.

Comment by jacobjacob on Epistemic Spot Check: The Fate of Rome (Kyle Harper) · 2019-08-25T18:01:26.324Z · score: 11 (10 votes) · LW · GW

I think of it like "95% confidence interval of where the mean of Elizabeth's estimate would land after 10 hours of further research".

I've found personally this format is often useful when giving quick probability estimates. Precision is costly.

Comment by jacobjacob on What experiments would demonstrate "upper limits of augmented working memory?" · 2019-08-23T09:34:45.754Z · score: 2 (1 votes) · LW · GW

I'm curious what software this was made with, and how habryka made it readable?

Comment by jacobjacob on jacobjacob's Shortform Feed · 2019-08-14T00:28:29.812Z · score: 19 (10 votes) · LW · GW

What important book that needs fact-checking is nobody fact-checking?

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-13T22:52:06.415Z · score: 2 (1 votes) · LW · GW

I also don't know if "this answers my objection" means "oh, then I'd use it" or "other problems still seem to big" (though I'd bet on the latter).

Comment by jacobjacob on Matthew Barnett's Shortform · 2019-08-13T18:01:57.861Z · score: 3 (2 votes) · LW · GW
  • The development of modern formal logic (predicate logic, modal logic, the equivalence of higher-order logics and set-theory, etc.), which is of course deeply related to Zermelo, Fraenkel, Turing and Church, but which involved philosophers like Quine, Putnam, Russell, Kripke, Lewis and others.
  • The model of scientific progress as proceeding via pre-paradigmatic, paradigmatic, and revolutionary stages (from Kuhn, who wrote as a philosopher, though trained as a physicist)
Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-13T17:29:45.533Z · score: 5 (2 votes) · LW · GW
As I understand it, the point of a costly signal is that it's supposed to be relatively more affordable if you actually have that quality.

Yeah. In hindsight the terminology of "costly signal" is a bit unfortunate, because the payment here would actually work a bit like Mario's jump or LessWrong karma -- it's a very simple mechanism which can co-opted to solve a large number of different problems. In particular, the money is not intended to be burnt (as would be the case with status signals, or proof-of-work as mentioned in some other comments), but actually paid to you.

Overall appreciate you writing up those points, they're pretty helpful in understanding how people might (and might not) use this.

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-13T09:38:25.915Z · score: 2 (1 votes) · LW · GW

That's a great point, I will do that.

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T10:43:48.652Z · score: 2 (3 votes) · LW · GW

This was crossposted to the EA forum replacing all mentions of "rationality" with "EA", mutatis mutandis.

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T10:25:21.150Z · score: 2 (1 votes) · LW · GW

Yup, I think one of the main use cases is to enable a way of contacting people much higher status/more busy than you that doesn't require a ton of networking or makes their lives terrible. (N.b. I have lots of uncertainty over whether there's actually demand for this among that cohort, which is partly why I'm writing this.)

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T10:21:38.373Z · score: 2 (1 votes) · LW · GW

Updated the OP to clarify this. Will hold off on replying until I know whether this changes Scott's mind or not!

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T10:14:19.161Z · score: 2 (1 votes) · LW · GW

Yeah, referrals are important.

They also have some problems:

  • Requires you to already know the right people
  • If the referer doesn't know how valuable the recipient will think this is, they might just avoid it as opposed to risking burning reputation (same problem as outlined above)
  • It's still costly for the recipient. E.g. it doesn't have any way of compensating the recipient for giving them the "reduce option value" vs "slightly harm relationship" trade-off

There are probably large differences between how important each of these problems are, though I'm moderately confident that at least the first presents are real and important user case. If the options are "Pay $50 to message X" or "Try to network with people who can then introduce you to X", the first might be better and save us some social games.

Comment by jacobjacob on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T09:54:08.788Z · score: 2 (1 votes) · LW · GW

Thanks, this is a good data-point.

Though I want to ask: if people know that you have this problem, as things stand currently, they might just avoid messaging you (since they don't have any way of compensating you for marginally making the burden on you worse)? Moreover, your time and energy are presumably exchangeable for money (though not indefinitely so)?

So it still seems paid emails might help with that?

(PS. I don't think it's only about information asymmetries and having too many emails, though I realise the OP quite strongly implies that. Might rewrite to reflect that.)