# ozziegooen's Shortform

post by ozziegooen · 2019-08-31T23:03:24.809Z · score: 17 (6 votes) · LW · GW · 127 comments

comment by ozziegooen · 2019-08-31T23:03:24.990Z · score: 27 (9 votes) · LW(p) · GW(p)

# Questions around Making Reliable Evaluations

Most existing forecasting platform questions [LW · GW] are for very clearly verifiable questions:

• "Who will win the next election"
• "How many cars will Tesla sell in 2030?"

But many of the questions we care about are much less verifiable:

• "How much value has this organization created?"
• "What is the relative effectiveness of AI safety research vs. bio risk research?"

One solution attempt would be to have an "expert panel" assess these questions, but this opens up a bunch of issues. How could we know how much we could trust this group to be accurate, precise, and understandable?

The topic of, "How can we trust that a person or group can give reasonable answers to abstract questions" is quite generic and abstract, but it's a start.

I've decided to investigate this as part of my overall project on forecasting infrastructure. I've recently been working with Elizabeth [LW · GW] on some high-level research.

I believe that this general strand of work could be useful both for forecasting systems and also for the more broad-reaching evaluations that are important in our communities.

## Early concrete questions in evaluation quality

One concrete topic that's easily studiable is evaluation consistency. If the most respected philosopher gives wildly different answers to "Is moral realism true" on different dates, it makes you question the validity of their belief. Or perhaps their belief is fixed, but we can determine that there was significant randomness in the processes that determined it.

Daniel Kahneman apparently thinks a version of this question is important enough to be writing his new book on it.

Another obvious topic is in the misunderstanding of terminology. If an evaluator understands "transformative AI" in a very different way to the people reading their statements about transformative AI, they may make statements that get misinterpreted.

These are two specific examples of questions, but I'm sure there are many more. I'm excited about understanding existing work in this overall space more, and getting a better sense of where things stand and what the next right questions are to be asking.

comment by romeostevensit · 2019-09-09T23:37:49.081Z · score: 2 (1 votes) · LW(p) · GW(p)

> "How much value has this organization created?"

can insights from prediction markets work for helping us select better proxies and decision criteria or do we expect people to be too poorly entangled with the truth of these matters for that to work? Do orgs always require someone who is managing the ontology and incentives to be super competent at that to do well? De facto improvements here are worth billions (project management tools, slack, email add ons for assisting managing etc.)

comment by ozziegooen · 2019-09-10T09:49:17.509Z · score: 8 (2 votes) · LW(p) · GW(p)

I think that prediction markets can help us select better proxies, but the initial set up (at least) will require people pretty clever with ontologies.

For example, say a group comes up with 20 proposals for specific ways of answering the question, "How much value has this organization created?". A prediction market could predict the outcome of the effectiveness of each proposal.

I'd hope that over time people would put together lists of "best" techniques to formalize questions like this, so doing it for many new situations would be quite straightforward.

comment by mr-hire · 2019-09-10T14:49:00.396Z · score: 6 (3 votes) · LW(p) · GW(p)

Another related idea we played around with, but which didn't make it into the final whitepaper:

What if we just assumed that Brier score was also predictive of good judgement. Then, people, could create a distribution over several measures of "how good will this organization do" and we could use standard probability theory and aggregation tools to create an aggregated final measure.

comment by mr-hire · 2019-09-10T14:38:09.081Z · score: 4 (2 votes) · LW(p) · GW(p)

The way we handled this with Verity was to pick a series of values, like "good judgement", "integrity," "consistency" etc. Then the community would select exemplars who they thought represented those values the best.

As people voted on which proposals they liked best, we would weight their votes by:

1. How much other people (weighted by their own score on that value) thought they had that value.

2. How similarly they voted to the examplars.

This sort of "value judgement" allows for fuzzy representation of high level judgement, and is a great supplement to more objective metrics like Brier score which can only measure well defined questions.

Eigentrust++ is a great algorithm that has the properties needed for this judgement-based reputation. The Verity Whitepaper goes more into depth as to how this would be used in practice.

comment by romeostevensit · 2019-09-10T15:25:46.870Z · score: 2 (1 votes) · LW(p) · GW(p)

Deference networks seem underrated.

comment by jimrandomh · 2019-09-09T23:33:09.576Z · score: 2 (1 votes) · LW(p) · GW(p)

One way to look at this is, where is the variance coming from? Any particular forecasting question has implied sub-questions, which the predictor needs to divide their attention between. For example, given the question "How much value has this organization created?", a predictor might spend their time comparing the organization to others in its reference class, or they might spend time modeling the judges and whether they tend to give numbers that are higher or lower.

Evaluation consistency is a way of reducing the amount of resources that you need to spend modeling the judges, by providing a standard that you can calibrate against. But there are other ways of achieving the same effect. For example, if you have people predict the ratio of value produced between two organizations, then if the judges consistently predict high or predict low, this no longer matters since it affects both equally.

comment by ozziegooen · 2019-09-10T09:51:00.245Z · score: 2 (1 votes) · LW(p) · GW(p)

Yep, good points. Ideally one could do a proper or even estimated error analysis of some kind.

Having good units (like, ratios) seems pretty important.

comment by zulupineapple · 2019-09-07T12:23:04.144Z · score: -2 (2 votes) · LW(p) · GW(p)
"What is the relative effectiveness of AI safety research vs. bio risk research?"

If you had a precise definition of "effectiveness" this shouldn't be a problem. E.g. if you had predictions for "will humans go extinct in the next 100 years?" and "will we go extinct in the next 100 years, if we invest 1M into AI risk research?" and "will we go extinct, if we invest 1M in bio risk research?", then you should be able to make decisions with that. And these questions should work fine in existing forecasting platforms. Their long term and conditional nature are problems, of course, but I don't think that can be helped.

"How much value has this organization created?"

That's not a forecast. But if you asked "How much value will this organization create next year?" along with a clear measure of "value", then again, I don't see much of a problem. And, although clearly defining value can be tedious (and prone to errors), I don't think that problem can be avoided. Different people value different things, that can't be helped.

One solution attempt would be to have an "expert panel" assess these questions

Why would you do that? What's wrong with the usual prediction markets? Of course, they're expensive (require many participants), but I don't think a group of experts can be made to work well without a market-like mechanism. Is your project about making such markets more efficient?

comment by ozziegooen · 2019-09-07T19:34:29.495Z · score: 4 (2 votes) · LW(p) · GW(p)

If you had a precise definition of "effectiveness" this shouldn't be a problem.

Coming up with a precise definition is difficult, especially if you want multiple groups to agree. Those specific questions are relatively low-level; I think we should ask a bunch of questions like that, but think we may also want some more vague things as well.

For example, say I wanted to know how good/enjoyable a specific movie would be. Predicting the ratings according to movie reviewers (evaluators) is an approach I'd regard as reasonable. I'm not sure what a precise definition for movie quality would look like (though I would be interested in proposals), but am generally happy enough with movie reviews for what I'm looking for.

"How much value has this organization created?"

Agreed that that itself isn't a forecast, I meant in the more general case, for questions like, "How much value will this organization create next year" (as you pointed out). I probably should have used that more specific example, apologies.

And, although clearly defining value can be tedious (and prone to errors), I don't think that problem can be avoided.

Can you be more explicit about your definition of "clearly"? I'd imagine that almost any proposal at a value function would have some vagueness. Certificates of Impact get around this by just leaving that for the review of some eventual judges, kind of similar to what I'm proposing.

Why would you do that? What's wrong with the usual prediction markets?

The goal for this research isn't fixing something with prediction markets, but just finding more useful things for them to predict. If we had expert panels that agreed to evaluate things in the future (for instance, they are responsible for deciding on the "value organization X has created" in 2025), then prediction markets and similar could predict what they would say.

comment by zulupineapple · 2019-09-07T22:55:56.996Z · score: 1 (1 votes) · LW(p) · GW(p)
For example, say I wanted to know how good/enjoyable a specific movie would be.

My point is that "goodness" is not a thing in the territory. At best it is a label for a set of specific measures (ratings, revenue, awards, etc). In that case, why not just work with those specific measures? Vague questions have the benefit of being short and easy to remember, but beyond that I see only problems. Motivated agents will do their best to interpret the vagueness in a way that suits them.

Is your goal to find a method to generate specific interpretations and procedures of measurement for vague properties like this one? Like a Shelling point for formalizing language? Why do you feel that can be done in a useful way? I'm asking for an intuition pump.

Certainly there is some vagueness, but it seems that we manage to live with it. I'm not proposing anything that prediction markets aren't already doing.

comment by ozziegooen · 2019-09-09T10:00:23.854Z · score: 2 (1 votes) · LW(p) · GW(p)

Hm... At this point I don't feel like I have a good intuition for what you find intuitive. I could give more examples, but don't expect they would convince you much right now if the others haven't helped.

I plan to eventually write more about this, and eventually hopefully we should have working examples up (where people are predicting things). Hopefully things should make more sense to you then.

Short comments back<>forth are a pretty messy communication medium for such work.

comment by Tetraspace Grouping (tetraspace-grouping) · 2019-09-10T15:09:12.386Z · score: 1 (1 votes) · LW(p) · GW(p)

There's something of a problem with sensitivity; if the x-risk from AI is ~0.1, and the difference in x-risk from some grant is ~10^-6, then any difference in the forecasts is going to be completely swamped by noise.

(while people in the market could fix any inconsistency between the predictions, they would only be able to look forward to 0.001% returns over the next century)

comment by ozziegooen · 2019-09-11T10:54:50.623Z · score: 2 (1 votes) · LW(p) · GW(p)

Yea, in cases like these, having intermediate metrics seems pretty essential.

comment by zulupineapple · 2019-09-13T18:14:32.330Z · score: 1 (1 votes) · LW(p) · GW(p)

Making long term predictions is hard. That's a fundamental problem. Having proxies can be convenient, but it's not going to tell you anything you don't already know.

comment by ozziegooen · 2019-12-24T21:49:40.849Z · score: 16 (8 votes) · LW(p) · GW(p)

Namespace pollution and name collision are two great concepts in computer programming. They way they are handled in many academic environments seems quite naive to me.

Programs can get quite large and thus naming things well is surprisingly important. Many of my code reviews are primarily about coming up with good names for things. In a large codebase, every time symbolicGenerator() is mentioned, it refers to the same exact thing. If after one part of the codebase has been using symbolicGenerator for a reasonable set of functions, and later another part comes up, and it's programmer realizes that symbolicGenerator is also the best name for that piece, they have to make a tough decision. Either they could refactor the codebase to change all previous mentions of symbolicGenerator to use an alternative name, or they have to come up with an alternative name. They can't have it both ways.

Therefore, naming becomes a political process. Names touch many programmers who have different intuitions and preferences. A large refactor of naming in a section of the codebase that others use would often be taken quite hesitantly by that group.

This makes it all the more important that good names are used initially. As such, reviewers care a lot about the names being pretty good; hopefully they are generic enough so that their components could be expanded while the name remains meaningful; but specific enough to be useful for remembering. Names that get submitted via pull requests represent much of the human part of the interface/API; they're harder to change later on, so obviously require extra work to get right the first time.

To be clear, a name collision is when two unrelated variables have the same name, and namespace pollution refers to when code is initially submitted in ways that are likely to create unnecessary conflicts later on.

My impression is that in much of academia, there are few formal processes for groups of experts to agree on the names for things. There are specific clusters with very highly thought out terminology, particularly around very large sets of related terminology; for instance, biological taxonomies, the metric system, and various aspects of medicine and biology.

By in many other parts, it seems like a free-for-all among the elite. My model of the process is something like, "Someone coming up with a new theory will propose a name for it and put it in their paper. If the paper is accepted (which is typically done with details in mind unrelated to the name), and if others find that theory useful, then they will generally call it the same name as the one used in the proposal. In some cases a few researchers will come up with a few variations for the same idea, in which case one will be selected through the process of what future researchers decide to use, on an individual bases. Often ideas are named after those who came up with them to some capacity; this makes a lot of sense to other experts who worked in these areas, but it's not at all obvious if this is optimal for other people."

The result is that naming is something that happens almost accidentally, as the result of a processes which isn't paying particular attention to making sure the names are right.

When there's little or no naming processes, than actors are incentivized to chose bold names. They don't have to pay the cost for any namespace pollution they create. Two names that come to mind recently have been "The Orthogonality Thesis" or "The Simulation Hypothesis*. These are two rather specific things with very generic names. Those come to mind because they are related to our field, but many academic topics seem similar. Information theory is mostly about encoding schemes, which are now not that important. Systems theory is typically about a subset of dynamical systems. But of course, it would be really awkward for anyone else with a more sensible "Systems theory" to use that name for the new thing.

I feel like AI has had some noticeable bad examples; It's hard to look at all the existing naming and think that this was the result of a systematic and robust naming approach. The Table of Contents of AI A Modern Approach seems quite good to me; that seems very much the case of a few people refactoring things to come up with one high-level overview that is optimized for being such. But the individual parts are obviously messy. A* search, alpha-beta pruning, K-consistency, Gibbs sampling, Dempster-shafer theory, etc.

LessWrong

One of my issues with LessWrong is the naming system. There's by now quite a bit of terminology to understand; the LessWrong wiki seems useful here. But there's no strong process from what I understand. People suggest names in their posts, these either become popular or don't. There's rarely any refactoring.

comment by G Gordon Worley III (gworley) · 2019-12-25T23:54:11.980Z · score: 4 (2 votes) · LW(p) · GW(p)

I'm not sure if it's good or bad, but I find the way species get named interesting.

The general rule is "first published name wins", and this is true even if the first published name is "wrong" in some way, like implies a relationship that doesn't exist, since that implication is not officially semantically meaningful. But there are ways to get around this, like if a name was based on a disproved phylogeny, in which case a new name can be taken up that fits the new phylogenic relationship. This means existing names get to stick, at least up until the time that they are proven so wrong that they must be replaced. Alas, there's no official registry of these things, so it's up to working researchers to do literature reviews and get the names right, and sometimes people get it wrong by accident and sometimes on purpose because they think an earlier naming is "invalid" for one reason or another and so only recognize a later naming. The result is pretty confusing and requires knowing a lot or doing a lot of research to realize that, for example, two species names might refer to the same species in different papers.

comment by ozziegooen · 2019-12-28T20:09:25.316Z · score: 2 (1 votes) · LW(p) · GW(p)

Thanks, I didn't know. That matches what I expect from similar fields, though it is a bit disheartening. There's an entire field of library science and taxonomy, but they seem rather isolated to specific things.

comment by ozziegooen · 2019-12-24T23:01:45.321Z · score: 4 (2 votes) · LW(p) · GW(p)

Another quick note on the LessWrong wiki:

I'm skeptical of single definitions without disclaimers. I think it's misleading (to some) that "Truth is the correspondence between and one's beliefs about reality and reality. "[1]. Rather, it's fair to say that this is one specific definition of truth that has been used in many cases; I'm sure that others, including others on LessWrong, have used it differently.

Most dictionaries have multiple definitions for words. This seems more like what we should aim for.

In fairness, when I searched for "Rationality", the result states, "Rationality is something of fundamental importance to LessWrong that is defined in many ways", which I of course agree with.

comment by Pattern · 2019-12-25T02:21:42.707Z · score: 3 (2 votes) · LW(p) · GW(p)
I'm skeptical of single definitions without disclaimers.

At the meta-level it isn't clear what value other definitions might offer (in this case). ("Truth" seems like a basic concept that is understood prior to reading that article - it's easier to imagine such an argument for other concepts without such wide understanding.)

Most dictionaries have multiple definitions for words. This seems more like what we should aim for.

Perhaps more definitions should be brought in (as necessary), with the same level of attention to detail -

I'm sure that others, including others on LessWrong, have used it differently.

when they are used (extensively). It's possible that relevant posts have already been made, they just haven't been integrated into the wiki. Is the wiki up to date as of 2012, but not after that?

comment by Pattern · 2019-12-25T02:23:15.151Z · score: 3 (2 votes) · LW(p) · GW(p)
"Someone coming up with a new theory will propose a name for it and put it in their paper. If the paper is accepted (which is typically done with details in mind unrelated to the name)[1],

Footnote not found. The refactoring sounds like a good idea, though the main difficulty would be propagating the new names.

comment by ozziegooen · 2019-12-25T21:38:19.519Z · score: 3 (2 votes) · LW(p) · GW(p)

Thanks for point that out! I forgot the specific note, removed the [1].

I definitely would agree that refactoring would be difficult, especially if we haven't figured out a great refactoring process.

comment by mr-hire · 2019-12-28T17:19:52.862Z · score: 2 (1 votes) · LW(p) · GW(p)
One of my issues with LessWrong is the naming system. There's by now quite a bit of terminology to understand; the LessWrong wiki seems useful here. But there's no strong process from what I understand. People suggest names in their posts, these either become popular or don't. There's rarely any refactoring.

One of the issues with this in both an academic and LW context is that changing the name of something in a single source of truth codebase is much cheaper than changing the name of something in a community. The more popular an idea, the more cost goes up to change the name. Similarly, when you're working with a single organization, creating a process that everyone follows is relatively cheap compared to a loosely tied together community with various blogs, individuals, and organizations coining their own terms.

comment by ozziegooen · 2019-12-28T20:07:18.478Z · score: 3 (2 votes) · LW(p) · GW(p)

Yep, I'd definitely agree that it's harder. That said, this doesn't mean that it's not high-ev to improve on. One outcome could be that we should be more careful introducing names, as it is difficult to change them. Another would be to work to attempt to have formal ways of changing them after, even though it is difficult (It would be worthwhile in some cases, I assume).

comment by mr-hire · 2019-12-28T23:13:34.538Z · score: 6 (3 votes) · LW(p) · GW(p)

## Markets For Purchasing Prediction Bundles

Of course, the trusted forecasters can not only forecast the "calibration adjustment factors" for ongoing forecasts, but they can also forecast these factors for hypothetical forecasts as well.

Say you have 500 questions that need to be predicted, and there are multiple agencies that all say they could do a great job predicting these questions. They all give estimates of their mean predicted error, conditional on them doing the prediction work. Then you have a trusted forecaster give a calibration adjustment.

Firm 1 -20 -2 -22
Firm 2 -12 -9 -21
Firm 3 -15 -3 -18

(Note: the lower the expected error, the worse)

In this case, Firm 2 makes the best claim, but is revealed to be significantly overconfident. Firm 3 has the best adjusted predicted error, so they're the ones to go with. In fact, you may want to penalize Firm 2 further for being a so-called prediction service with apparent poor calibration skills.

## Correlations

One quick gotcha; one can't simply sum the expected errors of all of one's predictions to get the total predicted error. This would treat them as independent, and there are likely to be many correlations between them. For example, if things go "seriously wrong"; it's likely many different predictions will have high losses. To handle this perfectly would really require one model to have produced all forecasts, but if that's not the case there could likely be simple ways to approximate this.

## Bundles vs. Prediction Markets

I'd expect that in many cases, private services will be more cost-effective than posting predictions on full prediction markets. Plus, private services could be more private and custom. The general selection strategy in the table above could of course include some options that involve hosting questions on prediction markets, and the victor would be chosen based on reasonable estimates.

comment by jacobjacob · 2019-12-04T10:01:37.100Z · score: 2 (1 votes) · LW(p) · GW(p)
"I'd be willing to bet 1,000 with anyone that the eventual total error of my forecasts will be less than the 65th percentile of my specified predicted error." I think this is equivalent to applying a non-linear transformation to your proper scoring rule. When things settle, you get paid S(p) both based on the outcome of your object-level prediction p, and your meta prediction q(S(p)). Hence: S(p)+B(q(S(p))) where B is the "betting scoring function". This means getting the scoring rules to work while preserving properness will be tricky (though not necessarily impossible). One mechanism that might help is that if each player makes one object prediction p and one meta prediction q, but for resolution you randomly sample one and only one of the two to actually pay out. comment by ozziegooen · 2019-12-10T20:47:21.450Z · score: 4 (2 votes) · LW(p) · GW(p) Interesting, thanks! Yea, agreed it's not proper. Coming up with interesting payment / betting structures for "package-of-forecast" combinations seems pretty great to me. comment by jacobjacob · 2019-12-11T08:22:34.845Z · score: 8 (4 votes) · LW(p) · GW(p) I think this paper might be relevant: https://users.cs.duke.edu/~conitzer/predictionWINE09.pdf Abstract. A potential downside of prediction markets is that they may incentivize agents to take undesirable actions in the real world. For example, a prediction market for whether a terrorist attack will happen may incentivize terrorism, and an in-house prediction market for whether a product will be successfully released may incentivize sabotage. In this paper, we study principal-aligned prediction mechanisms– mechanisms that do not incentivize undesirable actions. We characterize all principal-aligned proper scoring rules, and we show an “overpayment” result, which roughly states that with n agents, any prediction mechanism that is principal-aligned will, in the worst case, require the principal to pay Θ(n) times as much as a mechanism that is not. We extend our model to allow uncertainties about the principal’s utility and restrictions on agents’ actions, showing a richer characterization and a similar “overpayment” result. comment by NunoSempere (Radamantis) · 2020-01-08T12:41:47.263Z · score: 1 (1 votes) · LW(p) · GW(p) This is somewhat solved if you have a forecaster that you trust that can make a prediction based on Sophia's seeming ability and honesty. The naive thing would be for that forecaster to predict their own distribution of the log-loss of Sophia, but there's perhaps a simpler solution. If Sophia's provided loss distribution is correct, that would mean that she's calibrated in this dimension (basically, this is very similar to general forecast calibration). The trusted forecaster could forecast the adjustment made to her term, instead of forecasting the same distribution. Generally this would be in the direction of adding expected loss, as Sophia probably had more of an incentive to be overconfident ( which would result in a low expected score from her) than underconfident. This could perhaps make sense as a percentage modifier (-30% points), a mean modifier (-3 to -8 points), or something else. Is it actually true that forecasters would find it easier to forecast the adjustment?> This is somewhat solved if you have a forecaster that you trust that can make a prediction based on Sophia's seeming ability and honesty. The naive thing would be for that forecaster to predict their own distribution of the log-loss of Sophia, but there's perhaps a simpler solution. If Sophia's provided loss distribution is correct, that would mean that she's calibrated in this dimension (basically, this is very similar to general forecast calibration). The trusted forecaster could forecast the adjustment made to her term, instead of forecasting the same distribution. Generally this would be in the direction of adding expected loss, as Sophia probably had more of an incentive to be overconfident ( which would result in a low expected score from her) than underconfident. This could perhaps make sense as a percentage modifier (-30% points), a mean modifier (-3 to -8 points), or something else. Is it actually true that forecasters would find it easier to forecast the adjustment? comment by ozziegooen · 2020-01-08T12:45:31.678Z · score: 2 (1 votes) · LW(p) · GW(p) One nice thing about adjustments is that they can be applied to many forecasts. Like, I can estimate the adjustment for someone's [list of 500 forecasts] without having to look at each one. Over time, I assume that there would be heuristics for adjustments, like, "Oh, people of this reference class typically get a +20% adjustment", similar to margins of error in engineering. That said, these are my assumptions, I'm not sure what forecasters will find to be the best in practice. comment by ozziegooen · 2019-12-10T20:45:57.445Z · score: 14 (7 votes) · LW(p) · GW(p) I feel like I've long underappreciated the importance of introspectability in information & prediction systems. Say you have a system that produces interesting probabilities for various statements. The value that an agent gets from them is not directly correlating to the accuracy of these probabilities, but rather to the expected utility gain they get after using information of these probabilities in corresponding Bayesian-approximating updates. Perhaps more directly, something related to the difference between one's prior and posterior after updated on . Assuming that prediction systems produce varying levels of quality results, agents will need to know more about these predictions to really optimally update accordingly. A very simple example would be something like a bunch of coin flips. Say there were 5 coins flipped, I see 3 of them, and I want to estimate the number that were heads. A predictor tells me that their prediction has a mean probability of 40% heads. This is useful, but what would be much more useful is a list of which specific coins the predictor saw and what their values were. Then I could get a much more confident answer; possibly a perfect answer. Financial markets are very black-box like. Many large changes in company prices never really get explained publicly. My impression is that no one really understands the reasons for many significant market moves. This seems really suboptimal and I'm sure no one wanted this property to be the case.[1] Similarly, when trying to model the future of our own prediction capacities, I really don't think they should be like financial markets in this specific way. [1] I realize that participants in the market try to keep things hidden, but I mean the specific point that few people think that "Stock Market being a black box" = "A good thing for society." comment by jacobjacob · 2019-12-11T16:55:38.609Z · score: 5 (3 votes) · LW(p) · GW(p) In some sense, markets have a particular built-in interpretability: for any trade, someone made that trade, and so there is at least one person who can explain it. And any larger market move is just a combination of such smaller trades. This is different from things like the huge recommender algorithms running YouTube, where it is not the case that for each recommendation, there is someone who understands that recommendation. However, the above argument fails in more nuanced cases: • Just because for every trade there's someone who can explain it, doesn't mean that there is a particular single person who can explain all trades • Some trades might be made by black-box algorithms • There can be weird "beauty contest" dynamics where two people do something only because the other person did it comment by ozziegooen · 2019-12-11T18:47:52.470Z · score: 7 (3 votes) · LW(p) · GW(p) Good point, though I think the "more nuanced cases" are very common cases. The 2010 flash crash seems relevant; it seems like it was caused by chaotic feedback loops with algorithmic components, that as a whole, are very difficult to understand. While that example was particularly algorithmic-induced, other examples also could come from very complex combinations of trades between many players, and when one agent attempts to debug what happened, most of the traders won't even be available or willing to explain their parts. The 2007-2008 crisis may have been simpler, but even that has 14 listed causes on Wikipedia and still seems hotly debated. In comparison, YouTube I think algorithms may be even simpler, though they are still quite messy. comment by ozziegooen · 2020-01-31T11:35:52.631Z · score: 12 (3 votes) · LW(p) · GW(p) ## More Narrow Models of Credences Epistemic Rigor I'm sure this has been discussed elsewhere, including on LessWrong. I haven't spent much time investigating other thoughts on these specific lines. Links appreciated! The current model of a classically rational agent assume logical omniscience and precomputed credences over all possible statements. This is really, really bizarre upon inspection. First, "logical omniscience" is very difficult, as has been discussed (The Logical Induction paper goes into this). Second, all possible statements include statements of all complexity classes that we know of (from my understanding of complexity theory). "Credences over all possible statements" would easily include uncountable infinities of credences. One could clarify that even arbitrarily large amounts of computation would not be able to hold all of these credences. Precomputation for things like this is typically a poor strategy, for this reason. The often-better strategy is to compute things on-demand. A nicer definition could be something like: A credence is the result of an [arbitrarily large] amount of computation being performed using a reasonable inference engine. It should be quite clear that calculating credences based on existing explicit knowledge is a very computationally-intensive activity. The naive Bayesian way would be to start with one piece of knowledge, and then perform a Bayesian update on each next piece of knowledge. The "pieces of knowledge" can be prioritized according to heuristics, but even then, this would be a challenging process. I think I'd like to see specification of credences that vary with computation or effort. Humans don't currently have efficient methods to use effort to improve our credences, as a computer or agent would be expected to. Solomonoff's theory of Induction or Logical Induction could be relevant for the discussion of how to do this calculation. comment by ozziegooen · 2020-01-21T12:36:50.865Z · score: 12 (7 votes) · LW(p) · GW(p) Communication should be judged for expected value, not intention (by consequentialists) TLDR: When trying to understand the value of information, understanding the public interpretations of that information could matter more than understanding the author's intent. When trying to understand the information for other purposes (like, reading a math paper to understand math), this does not apply. If I were to scream "FIRE!" in a crowded theater, it could cause a lot of damage, even if my intention were completely unrelated. Perhaps I was responding to a devious friend who asked, "Would you like more popcorn? If yes, should 'FIRE!'". Not all speech is protected by the First Amendment, in part because speech can be used for expected harm. One common defense of incorrect predictions is to claim that their interpretations weren't their intentions. "When I said that the US would fall if X were elected, I didn't mean it would literally end. I meant more that..." These kinds of statements were discussed at length in Expert Political Judgement. But this defense rests on the idea that communicators should be judged on intention, rather than expected outcomes. In those cases, it was often clear that many people interpreted these "experts" as making fairly specific claims that were later rejected by their authors. I'm sure that much of this could have been predicted. The "experts" often definitely didn't seem to be going out of their way to be making their after-the-outcome interpretations clear before-the-outcome. I think that it's clear that the intention-interpretation distinction is considered highly important by a lot of people, so much so as to argue that interpretations, even predictable ones, are less significant in decision making around speech acts than intentions. I.E. "The important thing is to say what you truly feel, don't worry about how it will be understood." But for a consequentialist, this distinction isn't particularly relevant. Speech acts are judged on expected value (and thus expected interpretations), because all acts are judged on expected value. Similarly, I think many consequentialists would claim that here's nothing metaphysically unique about communication as opposed to other actions one could take in the world. Some potential implications: 1. Much of communicating online should probably be about developing empathy for the reader base, and a sense for what readers will misinterpret, especially if such misinterpretation is common (which it seems to be). 2. Analyses of the interpretations of communication could be more important than analysis of the intentions of communication. I.E. understanding authors and artistic works in large part by understanding their effects on their viewers. 3. It could be very reasonable to attempt to map non probabilistic forecasts into probabilistic statements based on what readers would interpret. Then these forecasts can be scored using scoring rules just like those as regular probabilistic statements. This would go something like, "I'm sure that Bernie Sanders will be elected" -> "The readers of that statement seem to think the author applying probability 90-95% to the statement 'Bernie Sanders will win'" -> a brier/log score. Note: Please do not interpret this statement as attempting to say anything about censorship. Censorship is a whole different topic with distinct costs and benefits. comment by ozziegooen · 2020-01-21T22:39:12.332Z · score: 12 (3 votes) · LW(p) · GW(p) It seems like there are a few distinct kinds of questions here. 1. You are trying to estimate the EV of a document. Here you want to understand the expected and actual interpretation of the document. The intention only matters to how it effects the interpretations. 2. You are trying to understand the document. Example: You're reading a book on probability to understand probability. Here the main thing to understand is probably the author intent. Understanding the interpretations and misinterpretations of others is mainly useful so that you can understand the intent better. 3. You are trying to decide if you (or someone else) should read the work of an author. Here you would ideally understand the correctness of the interpretations of the document, rather than that of the intention. Why? Because you will also be interpreting it, and are likely somewhere in the range of people who have interpreted it. For example, if you are told, "This book is apparently pretty interesting, but every single person who has attempted to read it, besides one, apparently couldn't get anywhere with it after spending many months trying", or worse, "This author is actually quite clever, but the vast majority of people who read their work misunderstand it in profound ways", you should probably not make an attempt; unless you are highly confident that you are much better than the mentioned readers. comment by ozziegooen · 2020-01-21T22:43:36.693Z · score: 2 (1 votes) · LW(p) · GW(p) One nice thing about cases where the interpretations matter, is that the interpretations are often easier to measure than intent (at least for public figures). Authors can hide or lie about their intent or just never choose to reveal it. Interpretations can be measured using surveys. comment by Dagon · 2020-01-21T16:09:34.740Z · score: 4 (2 votes) · LW(p) · GW(p) Seems reasonable. It also seems reasonable to predict others' future actions based on BOTH someone's intentions and their ability to understand consequences. You may not be able to separate these - after the third time someone yells "FIRE" and runs away, you don't really know or care if they're trying to cause trouble or if they're just mistaken about the results. comment by ozziegooen · 2019-12-12T22:09:13.349Z · score: 12 (6 votes) · LW(p) · GW(p) Charity investigators could be time-effective by optimizing non-cause-neutral donations. There are a lot more non-EA donors than EA donors. It may also be the case that EA donation research is somewhat saturated. Say you think that1 donated to the best climate change intervention is worth 1/10th that of $1 for the best AI-safety intervention. But you also think that your work could increase the efficiency of$10mil of AI donations by 0.5%, but it could instead increase the efficiency of $50mil of climate change donations by 10%. Then, for you to maximize expected value, your time is best spent optimizing the climate change interventions. The weird thing here may be in explaining this to the donors. "Yea, I'm spending my career researching climate change interventions, but my guess is that all these funders are 10x less effective than they would be by donating to other things." While this may feel strange, both sides would benefit; the funders and the analysts would both be maximizing their goals. Separately, there's a second plus that teaching funders to be effectiveness-focused; it's possible that this will eventually lead some of them to optimize further. I think this may be the case in our current situation. There honestly aren't too many obvious places for "effective talent" to go right now. There is a ton of potential funders out there that wouldn't be willing to go to core EA causes any time soon, but may be able to be convinced to give much more effectively in their given areas. There could potentially be a great deal of work to be done doing this sort of thing. comment by ozziegooen · 2019-12-17T19:57:48.552Z · score: 9 (4 votes) · LW(p) · GW(p) I think the thing I find the most surprising about Expert Systems is that people expected them to work so early on, and apparently they did work in some circumstances. Some issues: 1. The user interfaces, from what I can tell, were often exceedingly mediocre. User interfaces are difficult to do well and difficult to specify, so are hard to guarantee quality in large and expensive projects. It was also significantly harder to make good UIs back when expert systems were more popular, than it is today. 2. From what I can tell, many didn't even have notions of uncertainty! AI: A Modern Approach discusses Expert Systems that I believe used first and second-order logic, but seemed to imply that many didn't include simple uncertainty parameters, let alone probability distributions of any kind. 3. Experts aren't even that great at assigning probability densities. Many are overconfident; papers by Tetlock and others suggest that groups of forecasters are hard to beat. My impression is that arguably Wikidata and other semantic knowledge graphs could be viewed as the database part of expert systems without attempting intense logic manipulations or inference. I know some other projects are trying to do more of the inference portions, but seem more used for data gathered from web applications and businesses instead of directly by querying experts. comment by ozziegooen · 2020-01-01T16:29:26.589Z · score: 8 (4 votes) · LW(p) · GW(p) ## Do people have precise understandings of the words they use? On the phrase "How are you?", traditions, mimesis, Chesterton's fence, and their relationships to the definitions of words. Epistemic status Boggling. I’m sure this is better explained somewhere in the philosophy of language but I can’t yet find it. Also, this post went in a direction I didn’t originally expect, and I decided it wasn’t worthwhile to polish and post on LessWrong main yet. If you recommend I clean this up and make it an official post, let me know. One recurrent joke is that one person asks another how they are doing, and the other responds with an extended monologue about their embarrassing medical conditions. A related comment on Reddit: It's a running joke between ESL'ers that every one of them will respond with "I am fine thank you, AND YOU?" Regardless of nationality. The first thing taught in English.[0] The point here is that “How are you”, as typically used, is obviously not a question in the conventional sense. To respond with a lengthy description would not just be unusual, it would be incorrect, in a similar way that you were asked, “What time is it”, and you responded “My anxiety levels increased 10% last week” would be incorrect. [1] I think this is a commonly understood example of a much larger class of words and phrases we commonly use. When new concept names are generated, I’d expect that they are generally done by taking a rough concept and and separately coming up with a reasonable name for it. The name is chosen and encouraged based on its convenience for its’ users, rather than precise correctness. I know many situations where this exactly happened (from history and practice of science engineering) and expect it’s the common outcome. Some examples of phrases that don't mean the only possible sum of their parts “Witch hunt” “Netflix and chill” “Cognitive Behavioral Therapy” “Operations Research” “Game Theory” “Bayes’ Theorem” “Free Software” Arguably this is basically the same procedure that was used for single words repurposed to represent other unique things. “Agent” “Rational” “Mass”. Including many philosophical fields, with to-me ridiculously simple names: [“Determinism”, “Idealism”, “Materialism”, “Objectivism”][2] Etymology explains the histories of many of these words & phrases, but I think leaves out much of the nuance.[3] One real tricky bit comes when the originating naming is forgotten and the word or phrase is propagated without clear definitions. Predictably, these definitions would change over time and this could lead to some sticky scenarios. I’m sure that when humans started using the phrase “How are you?” it was meant as a legitimate question, but over time this shifted to have what is essentially a different definition (or wide set of definitions). I’d bet that now, a lot of people wouldn’t give a very well reasoned answer to the question: ‘What does “How are you?” mean?’ They’re used to using it with an intent in mind and haven’t needed to investigate the underlying processes. The same could be for many of our other words and phrases. Socrates was known for asking people to define terms like justice, self, and morality, and getting them pretty annoyed when they failed to produce precise answers that held up to his scrutiny. “How are you?” may be interesting not because its’ definition has changed, but in particular because it did so without speakers recognizing what was going on. It shifted to something that may require comprehensive anthropological study to properly understand. Yet the phrase is commonly used anyway, the fact that it may be poorly understood by its’ users doesn’t seem to phase them. Perhaps this could really be seen under the lens of mimesis; most individuals weren’t consciously making well-understood decisions, but rather they were selecting from a very limited set of options, and choosing among them based on existing popularity and empirical reactions.[4] I think we could call this a floating[5] definition. Floating definitions are used for different purposes and in different ways, normally without the speakers having a clear idea on their definitions. Perhaps the most clear version of this kind of idea comes from traditions. No one I know really understands why Western cultures have specific traditions for Weddings, Holidays, Birthdays, and the like, or if these things would really be optimal for us if they were fully understood. But we still go along with them because the expected value seems good enough. These are essentially large “Chesterton's fences” that we go along with and try to feel good about. My point here is just that in the same way many people don’t understand traditions, but go along with them, they also don’t really understand many words or phrases, but still go along with them. [0] Reddit: Why is how are you a greeting [1] One difference is that often the person asking the question wouldn’t quite be precisely aware of the distinction, so would often be more understanding of an incorrect response that details the answer to “how are you really doing.” A second difference is that they may think you honestly misunderstood them if you give them the wrong response. [2] Imagined if various engineering fields tried naming themselves in similar ways. Although upon reflection, they were likely purposely not named like that in part to not get associated with things like philosophy. [3] For example, etymology typically doesn’t seem to include things like defining the phrase “How are you”. Origin of “How are you?” [4] See The Secrets of our Success, and JacobJacob’s post Unconsious Economics [LW · GW] [5] This is probably a poor name and could be improved. I’ve attempted to find better names but couldn’t yet. Some of my random notes/links from investigating this topic: Folk taxonomy - Wikipedia Nomenclature - Wikipedia Proper noun - Wikipedia Common name - Wikipedia Skunked term - Wikipedia Jargon - Wikipedia A nominal definition is the definition explaining what a word means (i.e., which says what the "nominal essence" is), and is definition in the classical sense as given above. A real definition, by contrast, is one expressing the real nature or quid rei of the thing. Definition - Wikipedia Polysemy is the capacity for a sign (such as a word, phrase, or symbol) to have multiple meanings Semantic change - Wikipedia euphemism treadmill comment by ozziegooen · 2020-01-24T12:48:49.988Z · score: 2 (1 votes) · LW(p) · GW(p) Update: After I wrote this shortform, I did more investigation in Pragmatics and realized most of this was better expressed there. comment by Raemon · 2020-01-24T17:46:10.179Z · score: 3 (1 votes) · LW(p) · GW(p) What's Pragmatics in this case? comment by ozziegooen · 2019-12-17T00:14:36.356Z · score: 8 (4 votes) · LW(p) · GW(p) It seems inelegant to me that utility functions are created for specific situations, while these clearly aren't the same as that of the agent in total among all of their decisions. For instance, a model may estimate an agent's expected utility from the result of a specific intervention, but this clearly isn't quite right; the agent has a much more complicated utility function outside this intervention. According to a specific model, "Not having an intervention" could set "Utility = 0"; but for any real agent, it's quite likely their life wouldn't actually have 0 utility without the intervention. It seems like it's important to distinguish that a utility score in a model is very particular to the scenario for that model, and does not represent a universal utility function for the agents in question. Let be an agent's true utility function across a very wide assortment of possible states, and be the utility function used for the sake of the model. I believe that is supposed to approximate in some way; perhaps they should be related by an affine transformation. The important thing for a utility function, as it is typically used (in decision models), is probably not that , but rather, that decisions made within the specific context of approximate those made using . Here, I use brackets to describe "The expected value, according to a utility function", and D to describe the set of decisions made conditional on a specific utility function being used for decision making. Then, we can represent this supposed estimation with: comment by ozziegooen · 2019-12-17T00:52:35.125Z · score: 5 (3 votes) · LW(p) · GW(p) Related to this, one common argument against utility maximization is that "we still cannot precisely measure utility". But here, it's perhaps more clear that we don't need to. What's important for decision making is that we have models that we can expect will help us maximize our true utility functions, even if we really don't know much about what they really are. comment by Stuart_Armstrong · 2019-12-18T15:34:22.420Z · score: 4 (2 votes) · LW(p) · GW(p) comment by ozziegooen · 2019-12-18T18:54:11.426Z · score: 4 (2 votes) · LW(p) · GW(p) Oh fantastic, thanks for the reference! comment by Pattern · 2019-12-17T19:23:03.398Z · score: 3 (2 votes) · LW(p) · GW(p) I believe that ^U is supposed to approximate ^U in some way; perhaps they should be related by an affine transformation. ^U and ^U look to be the same. comment by ozziegooen · 2019-12-17T19:49:36.075Z · score: 5 (3 votes) · LW(p) · GW(p) Thanks! Fixed. I'm sure the bottom notation could be improved, but am not sure the best way. In general I'm trying to get better at this kind of mathematics. comment by Pattern · 2019-12-18T06:10:31.383Z · score: 2 (2 votes) · LW(p) · GW(p) You got the basic idea across, which is a big deal. Though whether it's A or B isn't clear: A) "this isn't all of the utility function, but its everything that's relevant to making decisions about this right now". ^U doesn't have to be U, or even a good approximation in every situation - just (good enough) in the situations we use it. Building a building? A desire for things to not fall on people's heads becomes relevant (and knowledge of how to do that). Writing a program that writes programs? It'd be nice if it didn't produce malware. Both desires usually exist - and usually aren't relevant. Models of utility for most situations won't include them. B) The cost of computing the utility function more exactly in the case exceeds the (expected) gains. isn't clear. comment by ozziegooen · 2019-12-18T12:55:23.056Z · score: 3 (2 votes) · LW(p) · GW(p) I think I agree with you. There's a lot of messiness with using and often I'm sure that this approximation leads to decision errors in many real cases. I'd also agree that better approximations of would be costly and are often not worth the effort. Similar to how there's a term for "Expected value of perfect information", there could be an equivalent for the expected value of a utility function, even outside of uncertainty of parameterized that were thought to be included. Really, there could be calculations for "expected benefit from improvements to a model", though of course this would be difficult to parameterize (how would you declare that a model has been changed a lot vs. a little? If I introduce 2 new parameters, but these parameters aren't that important, then how big of a deal should this be considered in expectation?) comment by Pattern · 2019-12-18T17:44:37.986Z · score: 3 (2 votes) · LW(p) · GW(p) The model has changed when the decisions it is used to make change. If the model 'reverses' and suggests doing the opposite/something different in every case from what it previously recommended, then it has 'completely changed'. (This might be roughly the McNamara fallacy, of declaring that things that 'can't be measured' aren't important.) EDIT: Also, if there's a set of information consisting of a bunch of pieces, A, B, and C, and incorporating all but one of them doesn't have a big impact on the model, but the last piece does, whichever piece that is, 'this metric' could lead to overestimating the importance of whichever piece happened to be last, when it's A, B, and C together that made an impact. It 'has this issue' because the metric by itself is meant to notice 'changes in the model over time', not figure out why/solve attribution. comment by ozziegooen · 2019-12-16T17:32:07.019Z · score: 8 (4 votes) · LW(p) · GW(p) I've been trying to scurry academic fields for discussions of how agents optimally reduce their expected error for various estimands (parameters to estimate). This seems like a really natural thing to me (the main reason why we choose some ways of predictions over others), but the literature seems kind of thin from what I can tell. The main areas I've found have been Statistical Learning Theory and Bayesian Decision / Estimation Theory. However, Statistical Learning Theory seems to be pretty tied to Machine Learning, and Bayesian Decision / Estimation Theory seem pretty small. Preposterior analyses like expected value of information / expected value of sample information seem quite relevant as well, though that literature seems a bit disconnected from the above two mentioned. (Separately, I feel like preposterior analyses should be a much more common phrase. I hadn't actually heard about it until recently, but the idea and field is quite natural.) comment by ozziegooen · 2020-01-26T15:12:25.665Z · score: 7 (4 votes) · LW(p) · GW(p) Intervention dominance arguments for consequentialists ## Global Health There's a fair bit of resistance to long-term interventions from people focused on global poverty, but there are a few distinct things going on here. One is that there could be a disagreement on the use of discount rates for moral reasoning, a second is that the long-term interventions are much more strange. No matter which is chosen, however, I think that the idea of "donate as much as you can per year to global health interventions" seems unlikely to be ideal upon clever thinking. For the last few years, the cost-to-save-a-life estimates of GiveWell seem fairly steady. The S&P 500 has not been steady, it has gone up significantly. Even if you committed to purely giving to global heath, you'd be better off if you generally delayed. It seems quite possible that if every life you would have saved in 2010, you could have saved 2 or more if you would have saved the money and spent it in 2020, with a decently typical investment strategy. (Arguably leverage could have made this much higher.) From what I understand, the one life saved in 2010 would likely not have resulted in one extra life equivalent saved in 2020; the returns per year was likely less than that of the stock market. One could of course say something like, "My discount rate is over 3-5% per year, so that outweighs this benefit". But if that were true it seems likely that the opposite strategy could have worked. One could have borrowed a lot of money in 2010, donated it, and then spent the next 10 years paying that back. Thus, it seems conveniently optimal if one's enlightened preferences would suggest not either investing for long periods or borrowing. ## EA Saving One obvious counter to immediate donations would be to suggest that the EA community financially invests money, perhaps with leverage [EA · GW]. While it is difficult to tell if other interventions may be better, it can be simpler to ask if they are dominant; in this case, that means that they predictably increase EA-controlled assets at a rate higher than financial investments would. A good metaphor could be to consider the finances of cities. Hypothetically, cities could invest much of their earnings near-indefinitely or at least for very long periods, but in practice, this typically isn't key to their strategies. Often they can do quite well by investing in themselves. For instance, core infrastructure can be expensive but predictably lead to significant city revenue growth. Often these strategies area so effective that they issue bonds in order to pay more for this kind of work. In our case, there could be interventions that are obviously dominant to financial investment in a similar way. An obvious one would be education; if it were clear that giving or lending someone money would lead to predictable donations, that could be a dominant strategy to more generic investment strategies. Many other kinds of community growth or value promotion could also fit into this kind of analysis. Related, if there were enough of these strategies available, it could make sense for loans to be made in order to pursue them further. What about a non-EA growth opportunity? Say, "vastly improving scientific progress in one specific area." This could be dominant (to investment, for EA purposes) if it would predictably help EA purposes by more than the investment returns. This could be possible. For instance, perhaps a$10mil donation to life extension research[1] could predictably increase $100mil of EA donations by 1% per year, starting in a few years. One trick with these strategies is that many would fall into the bucket of "things a generic wealthy group could do to increase their wealth"; which is mediocre because we should expect that type of things to be well-funded already. We may also want interventions that differentially change wealth amounts. Kind of sadly, this seems to suggest that some resulting interventions may not be "positive sum" to all relevant stakeholders. Many of the "positive sum in respect to other powerful interest" interventions may be funded, so the remaining ones could be relatively neutral or zero-sum for other groups. [1] I'm just using life extension because the argument would be simple, not because I believe it could hold. I think it would be quite tricky to find great options here, as is evidenced by the fact that other very rich or powerful actors would have similar motivations. comment by ozziegooen · 2019-12-28T21:26:17.675Z · score: 7 (4 votes) · LW(p) · GW(p) I've been reading through some of TVTropes.org and find it pretty interesting. Part of me wishes that Wikipedia were less deletionist, and wonders if there could be a lot more stuff similar to TV Tropes on it. TVTropes basically has an extensive ontology to categorize most of the important features of games, movies, and sometimes real life. Because games & movies are inspired by real life, even those portions are applicable. Here are some phrases I think are kind of nice; each that has a bunch of examples in the real world. These are often military related. Of course, the first two and last two are very similar and should arguably be combined. comment by mr-hire · 2019-12-29T16:13:49.611Z · score: 6 (3 votes) · LW(p) · GW(p) A think I want: A recommendation engine that works based on listing the tropes you enjoy. comment by Pattern · 2019-12-29T22:14:25.279Z · score: 3 (2 votes) · LW(p) · GW(p) I've been reading through some of TVTropes.org and find it pretty interesting. Part of me wishes that Wikipedia were less deletionist, and wonders if there could be a lot more stuff similar to TV Tropes on it. This article expresses the same sentiment, and may include links to what that looked like, and where it went: https://www.gwern.net/In-Defense-Of-Inclusionism comment by ozziegooen · 2019-12-24T20:45:49.821Z · score: 7 (4 votes) · LW(p) · GW(p) There's a lot of arguing, of course, on if humans are rational, but this often mixes up two things: there's the "Von Neumann-Morgenstern utility function maximization" definition of "rational", and there's a hypothetical "rational" that a human could fulfill with constraints much more complicated than the classical approach, more in the direction of prospect theory, or Predictive Coding. I think I regard the second definition as sufficiently not understood or defined that it isn't yet worth using in most conversation. It seems challenging, to say the least, to ask if humans are rational according to some definition which we clearly do not even know yet, let alone expect others to agree with. As such, I think the word "rational" should typically be used to refer to the former. This therefore means that humans not only aren't rational, but that they shouldn't be rational, as they are dealing with limitations that "rational" agents wouldn't have. In this setup, "rational" really refers to a predominantly (I believe) 20th century model of human and organizational pattern; it exists in the map, not the territory. If one were to discuss how rational a person is, they would be discussing how well they fit this specific model; not necessarily how optimal that entity is being. ## On the "Rationalist" community Rationality could still be useful to study. While I believe rationalism should refer to a model more than agent ideals, that doesn't mean that studying the model isn't a useful way to understand what decisions we should be making. Rational agents represent a simple model, but that brings in many of the benefits of it being a model; it's relatively easy to use as a fundamental building block for further purposes. At the same time, LessWrong is arguably more than about rationality when defined in this sense. Some of LessWrong details problems and limitations regarding the classical rational models, so those would arguably fit outside of them better than inside of them. I see some of the posts as being about things that would be beneficial for a much better hypothetical "rationality++" model, even though they don't necessarily make sense within a classical rationality model. comment by Pattern · 2019-12-25T02:09:22.050Z · score: 3 (2 votes) · LW(p) · GW(p) There's a lot of arguing, of course, on if humans are rational, but this often mixes up two things: there's the "Von Neumann-Morgenstern utility function maximization" definition of "rational", and there's a hypothetical "rational" that a human could fulfill with constraints much more complicated than the classical approach, more in the direction of prospect theory, or Predictive Coding. I think I regard the second definition as sufficiently not understood or defined that it isn't yet worth using in most conversation. It seems challenging, to say the least, to ask if humans are rational according to some definition which we clearly do not even know yet, let alone expect others to agree with. Or it could be an intuitive usage and mean "(more) optimal". "Why don't more people do [thing that will improve their health]?" comment by ozziegooen · 2019-12-25T21:35:30.040Z · score: 2 (1 votes) · LW(p) · GW(p) I like that question. I think that if people were to try to define optimal in a specific way, they would find that it requires a model of human behavior; the common one that academics would fall back to is that of Von Neumann-Morgenstern utility function maximization. I think it's quite possible that when we have better models of human behavior, we'll better recognize that in cases where people seem to be doing silly things to improve their health, they're actually being somewhat optimal given a large sets of physical and mental constraints. comment by ozziegooen · 2019-12-18T00:54:27.030Z · score: 7 (3 votes) · LW(p) · GW(p) Say Tim states, “There is a 20% probability that X will occur”. It’s not obvious to me what that means for Bayesians. It could mean: • Tim’s prior is that there’s a 20% chance. (Or his posterior in the context of evidence) • Tim believes that when the listeners update on him saying there’s a 20% chance (perhaps with him providing insight in his thinking), their posterior will converge to there being a 20% chance. • Tim believes that the posterior of listeners may not immediately converge to 20%, but the posterior of the enlightened versions of these listeners would. Perhaps the listeners are 6th graders who won’t know how much to update, but if they learned enough, they would converge to 20%. I’ve heard some more formalized proposals like, “I estimate that if I and several other well respected people thought about this for 100 years, we would wind up estimating that there was a 20% chance”, but even this assumes that listeners would converge on this same belief. This seems like possibly a significant assumption! It's quite to Coherent Extrapolated Volition, and similarly questionable. comment by crabman · 2019-12-23T10:30:47.864Z · score: 5 (4 votes) · LW(p) · GW(p) It's definitely the first. The second is bizarre. The third can be steelmanned as "Given my evidence, an ideal thinker would estimate the probability to be 20%, and we all here have approximately the same evidence, so we all should have 20% probabilities", which is almost the same as the first. comment by ozziegooen · 2019-12-23T22:34:38.019Z · score: 4 (2 votes) · LW(p) · GW(p) I don't think it's only the first. It seems weird to me imagine telling to a group that "There's a 20% probability that X will occur" if I really have little idea and would guess many of them would have a better sense than me. I would only personally feel comfortable doing this if I was quite sure my information was quite a bit better than theirs. Else, I'd say something like, "I personally think there's a 20% chance, but I really don't have much information." comment by ChristianKl · 2019-12-18T13:49:25.571Z · score: 5 (3 votes) · LW(p) · GW(p) I take it to mean that if Tim is acting optimally and has to take a bet on the outcome 1:4 would be the point where both sides of the bad are equally profitable to him while if the odds deviate from 1:4 one side of the bet would be preferable to him. comment by ozziegooen · 2019-12-18T13:59:08.822Z · score: 2 (1 votes) · LW(p) · GW(p) One thing this wouldn't take into account is strength or weight of evidence. If Tim knew that all of the listeners had far more information than him, and thus probably could produce better estimates of X, then it seems strange for Tim to tell them that the chances are 20%. I guess my claim that saying “There is a 20% probability that X will occur” is more similar to: "I'm quite confident that the chances are 20%, and you should generally be too" than it is to, "I personally believe that the chances are 20%, but have no idea o how much that should update the rest of you." comment by ozziegooen · 2019-12-23T22:38:40.005Z · score: 4 (2 votes) · LW(p) · GW(p) I think my current best guess to this is something like: When humans say thing X, they don't mean the literal translation of X, but rather are pointing to X', which is a specific symbol that other humans generally understand. For instance, "How are you" is a greeting, not typically a literal question. [How Are You] can be thought of as a symbol that's very different than the sum of it's parts. That said, I find it quite interesting that the basics of human use of language seem to be relatively poorly understood; in the sense that I'd expect many people to disagree on what they think “There is a 20% probability that X will occur” means, even after using it with each other in a setting that assumes some amount of understanding. comment by Tetraspace Grouping (tetraspace-grouping) · 2019-12-23T22:06:40.611Z · score: 2 (3 votes) · LW(p) · GW(p) Other things that Tim might mean when he says 20%: • Tim is being dishonest, and believes that the listeners will update away from the radical and low-status figure of 20% to avoid being associated with the lowly Tim. • Tim believes that other listeners will be encouraged to make their own probability estimates with explicit reasoning in response, which will make their expertise more legible to Tim and other listeners. • Tim wants to show cultural allegiance with the Superforecasting tribe. comment by ozziegooen · 2019-12-18T00:32:01.578Z · score: 7 (4 votes) · LW(p) · GW(p) ## Perhaps resolving forecasts with expert probabilities can be better than resolving them with the actual events. The default in literature on prediction markets and decision markets is to expect that resolutions should be real world events instead of probabilistic estimates by experts. For instance, people would predict "What will the GDP of the US in 2025 be?”, and that would be scored using the future “GDP of the US.” Let’s call these empirical resolutions. These resolutions have a few nice properties: 1. We can expect expect them to be roughly calibrated. (Somewhat obvious) 2. They have relatively high precision/sharpness. While these may be great in a perfectly efficient forecaster market, I think they may be suboptimal for incentivizing forecasters to best estimate important questions given real constraints. A more cost-effective solution could look like having a team of calibrated experts[1] inspect the situation post-event, make their best estimate of the probability pre-event, and then use that for scoring predictions. A thought Experiment The intuition here could be demonstrated by a thought experiment. Say you can estimate a probability distribution . Your prior, and the prior you expect that others has, indicates that 99.99% of is definitely a uniform distribution, but the last 0.001% tail on the right is something much more complicated. You could spend a lot of effort better estimating this 0.001% tail, but there is a very small chance this would be valuable to do. In 99.99% of cases, any work you do here will not effect your winnings. Worse, you may need to wager a large amount of money for a long period of time for this possibility of effectively using your tiny but better-estimated tail in a bet. Users of that forecasting system may care about this tail. They may be willing to pay for improvements in the aggregate distributional forecast such that it better models an enlightened ideal. If it were quickly realized that 99.99% of the distribution was uniform, then any subsidies for information should go to those that did a good job improving the 0.001% tail. It’s possible that some pretty big changes to this tail could be figured out. Say instead that you are estimating the 0.001% tail, but you know you will be scored against a probability distribution selected by experts post-result, instead of the actual result. Say, these experts get to see all previous forecasts and discussion, so in expectation only respond with a forecast that is more sharp than the aggregate. In this case all of their work will be focused on this tail, so all of the differences in forecasters may come from this sliver. This setup would require the experts[1] to be calibrated. Further Work I’m sure there’s a mathematical representation to better showcase this distinction, and to specify the loss of motivation that traders would have on probabilities that they know will be resolved empirically rather than judgmentally (using the empirical data in these judgements.) There must be something in statistical learning theory or similar that deals with similar problems; for instance, I imagine a classifier may be able to perform better when learning against “enlightened probabilities” instead of “binary outcomes”, as there is more clear signal there. [1] I use “Experts” here to refer to a group estimated to provide the highly accurate estimates, instead of domain-respected experts. comment by NunoSempere (Radamantis) · 2020-01-09T11:19:56.811Z · score: 5 (3 votes) · LW(p) · GW(p) Here is another point by @jacobjacob, which I'm copying here in order for it not to be lost in the mists of time: Though just realised this has some problems if you expected predictors to be better than the evaluators: e.g. they’re like “one the event happens everjacobyone will see I was right, but up until then no one will believe me, so I’ll just lose points by predicting against the evaluators” (edited) Maybe in that case you could eventually also score the evaluators based on the final outcome… or kind of re-compensate people who were wronged the first time… comment by ChristianKl · 2019-12-18T14:07:07.237Z · score: 4 (2 votes) · LW(p) · GW(p) You can train experts to be calibrated in different ways. If you train experts to be calibrated to pick the right probability on GPOpen where probability is done in steps on 1, I don't think those experts will be automatically calibrated to distinguish a p=0.00004 event from a p=0.00008. Experts would actually need to be calibrated on getting probabilities inside the tail right. I don't think we know how to do calibration training for that tail. comment by ozziegooen · 2019-12-18T18:49:41.904Z · score: 2 (1 votes) · LW(p) · GW(p) I think this could be a good example for what I'm getting at. I think there are definitely some people in some situations who can distinguish a p=0.00004 event from a p=0.00008 event. How? By making a Fermi model or similar. A trivial example would be a lottery with calculable odds of success. Just because the odds are low doesn't mean they can't be precisely estimated. I expect that the kinds of problems that GPOpen would consider asking AND are incredibly unlikely, would be difficult to estimate within 1 order of magnitude. But may still be able to do a decent job, especially in cases where you can make neat Fermi models. However, of course, it seems very silly to use the incentive mechanism "you'll get paid once we know for sure if the event happened" on such an event. Instead, if resolutions are done with evaluators, then there is much more of a signal. comment by mr-hire · 2020-01-08T20:25:53.813Z · score: 5 (3 votes) · LW(p) · GW(p) How? By making a Fermi model or similar. I'm fairly skeptical of this. From a conceptual perspective, we expect the tails to be dominated by unknown unknowns and black swans. Fermi estimates and other modelling tools are much better at estimating scenarios that we expect. Whereas, if we find ourselves in the extreme tails, its often because of events or factors that we failed to model. comment by ozziegooen · 2020-01-08T21:14:10.904Z · score: 2 (1 votes) · LW(p) · GW(p) From a conceptual perspective, we expect the tails to be dominated by unknown unknowns and black swans. I'm not sure. The reasons things happen at the tails typically fall into categories that could be organized to be a small set. For instance: • The question wasn't understood correctly. • A significant exogenous event happened. But, as we do a bunch of estimates, we could get empirical data about these possibilities, and estimate the potentials for future tails. This is a bit different to what I was mentioning, which was more about known but small risks. For instance, the "amount of time I spend on my report next week" may be an outlier if I die. But the chance of serious accident or death can be estimated decently well enough. These are often repeated known knowns. comment by ChristianKl · 2019-12-18T19:39:16.741Z · score: 2 (1 votes) · LW(p) · GW(p) You might have people who can distinguish those, but I think it's a mistake to speak of calibration in that sense as the word usually refers to people who actually trained to be calibrated via feedback. comment by NunoSempere (Radamantis) · 2020-01-08T13:20:39.439Z · score: 3 (2 votes) · LW(p) · GW(p) Another point in favor of such a set-up would be that aspiring superforecasters get much, much more information when they see ~[the prediction of a superforecaster would have made having their information]; a point vs a distribution. I'd expect that this means that market participants would get better, faster. comment by ozziegooen · 2020-01-08T13:33:49.629Z · score: 2 (1 votes) · LW(p) · GW(p) Yep, this way would basically be much more information-dense, with all the benefits that comes from that. comment by Pattern · 2019-12-18T06:25:37.542Z · score: 3 (2 votes) · LW(p) · GW(p) So you don't want predictions*, you want models**. the 0.001% tail Robust/fully fleshed out models. *predictions of events **predictions of which model is correct comment by ozziegooen · 2019-12-18T12:51:55.336Z · score: 3 (2 votes) · LW(p) · GW(p) I'm not sure I'd say that in the context of this post, but more generally, models are really useful. Predictions that come with useful models are a lot more useful than raw predictions. I wrote this other [LW(p) · GW(p)] post about a similar topic. For this specific post, I think what we're trying to get is the best prediction we could have had using data pre-event. comment by mr-hire · 2020-01-08T18:14:46.272Z · score: 2 (1 votes) · LW(p) · GW(p) Users of that forecasting system may care about this tail. They may be willing to pay for improvements in the aggregate distributional forecast such that it better models an enlightened ideal. If it were quickly realized that 99.99% of the distribution was uniform, then any subsidies for information should go to those that did a good job improving the 0.001% tail. It’s possible that some pretty big changes to this tail could be figured out. I'm really interested in this type of scheme because it would also solve a big problem in futarchy and futarchy-like setups that use prediction polling, namely, the inability to score conditional counterfactuals (which is most of the forecasting you'll be doing in Futarchy-like setup). One thing you could do instead of scoring people against expert assesments is also potentially score people against the final aggregate and extremized distribution. One issue with any framework like this is that general calibration may be very different than calibration at the tails. Whatever scoring rule you're using to determine calibration of experts or aggregate scoring has the same issue that long tail events rarely happen. Another solution to this problem (although it doesn't solve the counterfactual conditional problem) is to create tailored scoring rules that provide extra rewards for events at the tails. If an event at the tails is a million times less likely to happen, but you care about it equally to events at the center, then provide a million times reward for accuracy near the tail in the event it happens. Prior work on tailored scoring rules for different utility functions here: https://www.evernote.com/l/AAhVczys0ddF3qbfGk_s4KLweJm0kUloG7k/ comment by ozziegooen · 2020-01-08T22:17:54.758Z · score: 2 (1 votes) · LW(p) · GW(p) Good points! Also, thanks for the link, that's pretty neat. One thing you could do instead of scoring people against expert assessments is also potentially score people against the final aggregate and extremized distribution. I think that an efficient use of expert assessments would be for them to see the aggregate, and then basically adjust that as is necessary, but to try to not do much original research. I just wrote a more recent shortform post about this. One issue with any framework like this is that general calibration may be very different than calibration at the tails. I think that we can get calibration to be as good as experts can figure out, and that could be enough to be really useful. comment by ozziegooen · 2019-12-10T18:34:31.891Z · score: 7 (4 votes) · LW(p) · GW(p) He's an in-progress hierarchy of what's needed for information to be most useful to an organization or other multi-agent system. I'm sure there must be other very similar hierarchies out there, but don't currently know of any quite like this. Say you've come up with some cool feature that Apple could include in it's next phone. You think this is a great idea and they should add it in the future. You're outside of Apple, so the only way you have of interacting with them is by sending information through various channels. The question is: what things should you first figure out to understand how to do this? First, you need to have identified an improvement. You've done that, so you've gotten through the first step. Second, for this to be incorporated, it should make sense from Apple's perspective. If it comes out that the costs of adding the feature, including opportunity costs, outweigh the benefits, then it wouldn't make sense to them. Perhaps you could deceive them to incorporate the feature, but it would be against their interests. So you should hopefully get information about Apple's utility function and identify an intervention that would implement your improvement while being positive in expected value to them. Of course, just because it could be good for Apple does not mean that the people necessary to implement it would be in favor of doing so. Perhaps this feature involves the front-facing camera, and it so happens that people in charge of the decisions around the front-facing camera have some strange decision function and would prefer not being asked to do more work. To implement your change, these people would have to be convinced. A rough estimation for that would be an analysis that suggests that taking this feature on would have positive expected value for their utility functions. Again, it's possible that isn't a requirement, but if so, you may be needing to effectively deceive people. Once you have expected value equations showing that a specific intervention to implement your suggestion makes sense both to Apple and separately to the necessary decision makers at Apple, then the remaining question is one of what can be called deployment. How do you get the information to the necessary decision makers? If you have all four of these steps, you're in a pretty good place to implement the change. One set of (long-winded) terminology for these levels would be something like: 1. Improvement identification 2. Positive-EV intervention identification for agent 3. Positive-EV intervention identification for necessary subagent 4. Viable deployment identification There are cases where you may also want to take one step further back and identify "problems" or "vague areas of improvement" before identifying "corresponding solutions." comment by ozziegooen · 2019-12-24T00:30:57.229Z · score: 4 (2 votes) · LW(p) · GW(p) Another note to this; there are cases where a system is both broken and fixable at the step-3 level. In some of these cases, it could be worth it to fix the system there instead, especially if you may want to make similar changes in the future. For instance, you may have an obvious improvement for your city to make. You may then realize that the current setups to suggest feedback are really difficult to use, but that it's actually quite feasible to make sure some changes happen that will make all kinds of useful feedback easier for the city to incorporate. comment by ozziegooen · 2020-01-07T11:30:18.052Z · score: 5 (3 votes) · LW(p) · GW(p) Would anyone here disagree with the statement: Utilitarians should generally be willing to accept losses of knowledge / epistemics for other resources, conditional on the expected value of the trade being positive. comment by Dagon · 2020-01-07T16:38:55.442Z · score: 4 (2 votes) · LW(p) · GW(p) [ not a utilitarian; discount my opinion appropriately ] This hits one of the thorniest problems with Utilitarianism: different value-over-time expectations depending on timescales and assumptions. If one is thinking truly long-term, it's hard to imagine what resource is more valuable than knowledge and epistemics. I guess tradeoffs in WHICH knowledge to gain/lose have to be made, but that's an in-category comparison, not a cross-category one. Oh, and trading it away to prevent total annihilation of all thinking/feeling beings is probably right. comment by ozziegooen · 2020-01-08T11:43:25.321Z · score: 2 (1 votes) · LW(p) · GW(p) It's hard to imagine what resource is more valuable than knowledge and epistemics I think my thinking is that for utilitarians, these are generally instrumental, not terminal values. Often they're pretty important instrumental values, but this still would mean that they could be traded off in respect to the terminal values. Of course, if they are "highly important" instrumental values, then something very large would have to be offered for a trade to be worth it. (total annihilation being one example) comment by Dagon · 2020-01-08T15:55:55.200Z · score: 2 (1 votes) · LW(p) · GW(p) I think we're agreed that resources, including knowledge, are instrumental (though as a human, I don't always distinguish very closely). My point was that for very-long-term terminal values, knowledge and accuracy of evaluation (epistemics) are far more important than almost anything else. It may be that there's a declining marginal value for knowledge, as there is for most resources, and once you know enough to confidently make the tradeoffs, you should do so. But if you're uncertain, go for the knowledge. comment by edoarad · 2020-01-10T12:37:22.666Z · score: 3 (2 votes) · LW(p) · GW(p) Non-Bayesian Utilitarian that are ambiguity averse [LW · GW] sometimes need to sacrifice "expected utility" to gain more certainty (in quotes because that need not be well defined). comment by AprilSR · 2020-01-08T03:06:07.855Z · score: 3 (2 votes) · LW(p) · GW(p) Doesn't being willing to accept a trade *directly follow* from the expected value of the trade being positive? Isn't that like, the *definition* of when you should be willing to accept a trade? The only disagreement would be how likely it is that losses of knowledge / epistemics are involved in positive value trades. (My guess is it does happen rarely.) comment by ozziegooen · 2020-01-08T11:44:58.589Z · score: 2 (1 votes) · LW(p) · GW(p) I'd generally say that, but wouldn't be surprised if there were some who disagreed; who's argument would be something like what-to-me would sound like a modification of utilitarianism, [utilitarianism+epistemic-terminal-values]. comment by AprilSR · 2020-01-09T23:39:40.551Z · score: 1 (1 votes) · LW(p) · GW(p) If you have epistemic terminal values then it would not be a positive expected value trade, would it? Unless "expected value" is referring to the expected value of something other than your utility function, in which case it should've been specified. comment by ozziegooen · 2020-01-10T11:21:24.724Z · score: 2 (1 votes) · LW(p) · GW(p) Yep, I would generally think so. I was doing what may be a poor steelman of my assumptions of how others would disagree; I don't have a great sense of what people who would disagree would say at this point. comment by Pattern · 2020-01-09T20:58:06.800Z · score: 1 (1 votes) · LW(p) · GW(p) a modification of utilitarianism, [utilitarianism+epistemic-terminal-values]. Happiness + Knowledge. (A related question is, do people with these values drink?) comment by [deleted] · 2020-01-08T03:41:21.906Z · score: 2 (1 votes) · LW(p) · GW(p) Only if the trade is voluntary. If the trade is forced (e.g. in healthcare) then you may have two bad options, and the option you do want is not on the table. comment by Isnasene · 2020-01-08T01:39:56.761Z · score: 3 (2 votes) · LW(p) · GW(p) In general, I would agree with the above statement (and technically speaking, I have made such trade-offs). But I do want to point out that it's important to consider what the loss of knowledge/epistemics entails. This is because certain epistemic sacrifices have minimal costs (I'm very confident that giving up FDT for CDT for the next 24 hours won't affect me at all) and some have unbounded costs (if giving up materialism causes me to abandon cryonics, it's hard to quantify how large of a blunder that would be). This is especially true of epistemics that allow to you be unboundedly exploited by an adversarial agent. As a result, even when the absolute value looks positive to me, I'll still try to avoid this kinds of trade-offs because certain black swans (ie bumping into an adversarial agent that exploits your lack of knowledge about something) make such bets very high risk. comment by ozziegooen · 2020-01-08T11:46:08.645Z · score: 2 (1 votes) · LW(p) · GW(p) This sounds pretty reasonable to me; it sounds like you're basically trying to maximize expected value, but don't always trust your initial intuitions, which seems quite reasonable. comment by Pattern · 2020-01-07T22:45:26.980Z · score: 3 (2 votes) · LW(p) · GW(p) [What "utilitarian" means could use some resolving, so I just treated this as "people".] I would disagree. I tried to find the relevant post in the sequences and found this [LW · GW] along with it: sacrificing epistemic rationality is a slippery slope, like The Legend of Murder-Gandhi. Would I accept that processes that take into account resource constraints might be more effective? Certainly, thought I think of that as 'starting the journey in a reasonable fashion' rather than 'going backwards' as your statement brings to mind. comment by George (George3d6) · 2020-01-07T15:23:47.284Z · score: 3 (2 votes) · LW(p) · GW(p) How would you define loss of knowledge ? comment by ozziegooen · 2020-01-07T16:59:19.015Z · score: 2 (1 votes) · LW(p) · GW(p) Basically, information that can be handled in "value of information" style calculations. So, if I learn information such that my accuracy of understanding the world increases, my knowledge is increased. For instance, if I learn the names of everyone in my extended family. comment by George (George3d6) · 2020-01-07T23:03:48.684Z · score: 1 (1 votes) · LW(p) · GW(p) Ok, but in this case do you mean "loss of knowledge" as in "loss of knowledge harbored within the brain" or "loss of knowledge no matter where it's stored, be it a book, brain, text file... etc" ? Further more, does losing copies of a certain piece of knowledge count as loss of knowledge ? What about translations of said knowledge (in another language or another philosophical/mathematical framework) that doesn't add any new information, just makes it accessible to a larger demographic ? comment by ozziegooen · 2020-01-08T11:40:17.541Z · score: 2 (1 votes) · LW(p) · GW(p) I was thinking the former, but I guess the latter could also be relevant/count. It seems like there's no strict cut-off. I'd expect a utilitarian to accept trade-offs against all these kinds of knowledge, conditional on the total expected value being positive. comment by George (George3d6) · 2020-01-08T11:50:33.268Z · score: 1 (1 votes) · LW(p) · GW(p) Well, the problem with the former (knowledge harbored within the brain) is that it's very vague and hard to define. If I have, say, a method to improve the efficacy of VX (an easily weaponizable nerve toxin). As a utilitarian I conclude this information is going to be harmful, I can purge it of my hard-drive, I can burn the papers I used to come up with this... etc. But I can't wipe my head clean of the information, at best I can resign to never talk about it to anyone and to not accord it much import, such that I may forget it. But that's not destruction per-say, it's closer to lying, not sharing the information with anyone (even if asked specifically), or to biasing your brain towards transmitting and remembering certain pieces of information (which we do all the time). However I don't see anything contentious with this case, nor with any other case of information-destruction, as long as it is for the greater utility. I think in general people don't advocate for destroying/forgetting information because: a) It's hard to do b) As a general rule of thumb the accumulation of information seems to be a good thing, even if the utility of a specific piece of information is not obvious But this is more of a heuristic, an exact principle. comment by ozziegooen · 2020-01-08T12:11:52.701Z · score: 2 (1 votes) · LW(p) · GW(p) I'd agree that the first one is generally pretty separated from common reality, but think it's a useful thought experiment. I was originally thinking of this more in terms of "removing useful information" than "removing expected-harmful information", but good point; the latter could be interesting too. comment by George (George3d6) · 2020-01-08T16:30:42.663Z · score: 2 (2 votes) · LW(p) · GW(p) Well,I think the "removing useful information" bit contradicts with utility to being with. As in, if you are a utilitarian, useful information == helps maximize utility. Thus the trade-off is not possible. I can think of some contrived examples where the trade-off is possible (e.g. where the information is harmful now but will be useful later), but in that case it's so easy to "hide" information in the modern age, instead of destroying it entirely, that the problem seem too theoretical to me. But at the end of the day, assuming you reached a contrived enough situation where the information must either be destroyed (or where hiding it devoid other people of the ability to discover further useful information), I think the utilitarian perspective has nothing fundamental against destroying it. However, no matter how hard I try, I can't really think of a very relevant example where this could be the case. comment by ozziegooen · 2020-01-08T22:55:37.777Z · score: 3 (2 votes) · LW(p) · GW(p) One extreme case would be committing suicide because your secret is that important. A less extreme case may be being OK with forgetting information; you're losing value, but the cost to maintain it wouldn't be worth it. (In this case the information is positive though) comment by ozziegooen · 2020-01-08T12:07:02.043Z · score: 2 (1 votes) · LW(p) · GW(p) There's some related academic work around this here: https://www.princeton.edu/~tkelly/papers/epistemicasinstrumental.pdf https://core.ac.uk/download/pdf/33752524.pdf They don't specifically focus on utilitarians, but the arguments are still relevant. Also, this post is relevant: https://www.lesswrong.com/posts/dMzALgLJk4JiPjSBg/epistemic-vs-instrumental-rationality-approximations [LW · GW] comment by ozziegooen · 2019-12-17T20:08:42.611Z · score: 5 (3 votes) · LW(p) · GW(p) Agent-based modeling seems like one obvious step forward to me for much of social-science related academic progress. OpenAI's Hide and Seek experiment was one that I am excited about, but it is very simple and I imagine similar work could be greatly extended for other fields. The combination of simulation, possible ML distillation on simulation (to make it run much faster), and effective learning algorithms for agents, seems very powerful. However, agent-based modeling still seems quite infrequently used within Academia. My impression is that agent-based software tools right now are quite unsophisticated and unintuitive compared to what academics would really find useful. This feels a bit like a collective action problem. Hypothetically, better tools could cost$5-500 Million+, but it's not obvious who would pay for them and how the funding would be structured.

I'm employed by Oxford now and it's obvious that things aren't well set up to hire programmers. There are strong salary caps and hiring limitations. Our group would probably have an awkward time paying out \$10,000 per person to purchase strong agent-based software, even if it were worth it in total.

comment by johnswentworth · 2019-12-17T21:29:21.871Z · score: 5 (3 votes) · LW(p) · GW(p)
Agent-based modeling seems like one obvious step forward to me for much of academic progress.

Could you give a few specific examples where you imagine agent-based models would help?

comment by ozziegooen · 2019-12-17T22:01:54.177Z · score: 8 (5 votes) · LW(p) · GW(p)

Sure,

1. Humans as agents / psychology / economics. Instead of making mathematical models of rational agents, have people write code that predicts the behaviors of rational agents or humans. Test the "human bots" against empirical experimental results of humans in different situations, to demonstrate that the code accurately models human behavior.
2. Mechanism design. Show that according to different incentive structures, humans will behave differently, and use this to optimize the incentive structures accordingly.
3. Most social science. Make agent-based models to generally help explain how groups of humans interact with each other and what collective behaviors emerge.

I guess when I said, "Much of academic progress"; I should have specified, "Academic fields that deal with modeling humans to some degree"; perhaps most of social science.

comment by ozziegooen · 2019-12-17T20:12:44.876Z · score: 2 (1 votes) · LW(p) · GW(p)

I thought Probabilistic Models of Cognition was quite great (it seems criminally underappreciated); that seems like a good step in this direction.

Perhaps in the future, one could prove that "This environment with these actors will fail in these ways" by empirically showing that reinforcement agents optimizing in those setups lead to specific outcomes.

comment by ozziegooen · 2020-01-14T20:35:57.680Z · score: 4 (2 votes) · LW(p) · GW(p)

One question around the "Long Reflection" or around "What will AGI do?" is something like, "How bottlenecked will be by scientific advances that we'll need to then spend significant resources on?"

I think some assumptions that this model typically holds are:

1. There will be decision-relevant unknowns.
2. Many decision-relevant unkowns will be EV-positive to work on.
3. Of the decision-relevant unknowns that are EV-positive to work on, these will take between 1% to 99% of our time.

(3) seems quite uncertain to me in the steady state. I believe it makes an intuitive estimate between 2 orders of magnitude, while the actual uncertainty is much higher than that. If this were the case, it would mean:

1. Almost all possible experiments are either trivial (<0.01% of resources, in total), or not cost-effective.
2. If some things are cost-effective and still expensive (they will take over 1% of the AGI lifespan), it's likely that they will take 100%+ of the time. Even if they would take 10^10% of the time, in expectation, they could still be EV-positive to pursue. I wouldn't be surprised if there were one single optimal thing like this in the steady-state. So this strategy would look something like, "Do all the easy things, then spend a huge amount of resources on one gigantic-sized, but EV-high challenge."

(This was inspired by a talk that Anders Sandberg gave)

comment by ozziegooen · 2020-01-07T11:26:47.248Z · score: 4 (2 votes) · LW(p) · GW(p)

I feel like a decent alternative to a spiritual journey would be an epistemic journey.

An epistemic journey would basically involve something like reading a fair bit of philosophy and other thought, thinking, and becoming less wrong about the world.

comment by ozziegooen · 2020-01-01T15:55:22.617Z · score: 4 (2 votes) · LW(p) · GW(p)

Instillation, Proliferation, Amplification

Paul Christiano and Ought use the terminology of Distillation and Amplification to describe a high-level algorithm of one type of AI reasoning.

I’ve wanted to come up with an analogy to forecasting systems. I previously named a related concept Prediction-Augmented Evaluation Systems [LW · GW], one somewhat renamed to “Amplification” by Jacobjacob in this post [LW · GW].

I think one thing that’s going on is that “distillation” doesn’t have an exact equivalent with forecasting setups. The term “distillation” comes with the assumptions:

1. The “Distilled” information is compressed.
2. Once something is distilled, it’s trivial to execute.

I believe that (1) isn’t really necessary, and (2) doesn’t apply for other contexts.

A different proposal: Instillation, Proliferation, Amplification

In this proposal, we split the “distillation” step into “instillation” and “proliferation”. Instillation refers to the learning of system A into system B. Proliferation refers to the use of system B to apply this learning to various things in a straightforward manner. Amplification refers to the ability of either system A or system B to be able to spend marginal resources to marginally improve a specific estimate or knowledge set.

For instance, in a Prediction-Augmentation Evaluation System, imagine that “Evaluation Procedure A” is to rate movies on a 1-10 scale.

Instillation
Some acquisition process is done to help “Forecasting Team B” learn how “Evaluation Procedure A” does its’ evaluations.

Proliferation
“Forecasting Team B” now applies their understanding of the evaluations of “Evaluation Procedure A” to evaluate 10,000 movies.

Amplification
If there are movies that are particularly important to evaluate well, then there are specific methods available to do so.

I think this is a more complex but generic pattern. Instillation seems purely more generic than distillation, and proliferation like an important aspect that sometimes will be quite expensive.

Back to forecasting, instillation and proliferation are two different things and perhaps should eventually be studied separately. Instillation is about "can a group of forecasters learn & replicate an evaluation procedure", and Proliferation is about "Can this group do that cost-effectively?"

comment by bgold · 2020-01-02T22:04:43.428Z · score: 3 (2 votes) · LW(p) · GW(p)

Is there not a distillation phase in forecasting? One model of the forecasting process is person A builds up there model, distills a complicated question into a high information/highly compressed datum, which can then be used by others. In my mind its:

Model -> Distill - > "amplify" (not sure if that's actually the right word)

I prefer the term scalable instead of proliferation for "can this group do it cost-effectively" as it's a similar concept to that in CS.

comment by ozziegooen · 2020-01-03T14:09:55.616Z · score: 5 (3 votes) · LW(p) · GW(p)

Distillation vs. Instillation

My main point here is that distillation is doing 2 things: transitioning knowledge (from training data to a learned representation), and then compressing that knowledge.[1] The fact that it's compressed in some ways arguably isn't always particularly important; the fact that it's transferred is the main element. If a team of forecasters basically learned a signal, but did so in a very uncompressed way (like, they wrote a bunch of books about said signal), but still were somewhat cost-effective, I think that would be fine.

Around "Profileration" vs. "Scaling"; I'd be curious if there are better words out there. I definitely considered scaling, but it sounds less concrete and less specific. To "proliferate" means "to generate more of", but to "scale" could mean, "to make look bigger, even if nothing is really being done."

I think my cynical guess is that "instillation/proliferation" won't catch on because they are too uncommon, but also that "distillation" won't catch on because it feels like a stretch from the ML use case. Could use more feedback here.

[1] Interestingly, there seem to be two distinct stages in Deep Learning that map to these two different things, according to Naftali Tishby's claims.

comment by ozziegooen · 2019-12-10T18:43:42.456Z · score: 4 (2 votes) · LW(p) · GW(p)

It seems really hard to deceive a Bayesian agent who thinks you may be deceiving them, especially in a repeated game. I would guess there could be interesting theorems about Bayesian agents that are attempting to deceive one another; as in, in many cases their ability to deceive the other would be highly bounded or zero, especially if they were in a flexible setting with possible pre-commitment devices.

To give a simple example, agent A may tell agent B that they believe , even though they internally believe . However, if this were somewhat repeated or even relatively likely, then agent B would update negligibly, if at all, on that message.

comment by habryka (habryka4) · 2019-12-10T20:10:15.892Z · score: 8 (4 votes) · LW(p) · GW(p)

Bayesian agents are logically omniscient, and I think a large fraction of deceptive practices rely on asymmetries in computation time between two agents with access to slightly different information (like generating a lie and checking the consistencies between this new statement and all my previous statements)

My sense is also that two-player games with bayesian agents are actually underspecified and give rise to all kinds of weird things due to the necessity for infinite regress (i.e. an agent modeling the other agent modeling themselves modeling the other agent, etc.), which doesn't actually reliably converge, though I am not confident. A lot of decision-theory seems to do weird things with bayesian agents.

So overall, not sure how well you can prove theorems in this space, without having made a lot of progress in decision-theory, and I expect the resolution to a lot of our confusions in decision-theory to be resolved by moving away from bayesianism.

comment by ozziegooen · 2019-12-10T21:01:02.581Z · score: 6 (3 votes) · LW(p) · GW(p)

Hm... I like the idea of an agent deceiving another due to it's bounds on computational time, but could imagine many stable (though smaller) solutions that wouldn't. I'm curious if a good bayesian agent could do "almost perfect" on many questions given limited computation. For instance, a good bayesian would be using bayesianism to semi-optimally use any set of computation (assuming it has some sort of intuition, which I assume is necessary?)

On being underspecified, it seems to me like in general our models of agent cognition forever have been pretty underspecified, so would definitely agree here. "Ideal" bayesian agents are somewhat ridiculously overpowered and unrealistic.

I found the simulations around ProbMods to be interesting at modeling similar things; I think I'd like to see a lot more simulations for this kind of work. https://probmods.org/

comment by ozziegooen · 2020-01-01T16:10:40.208Z · score: 3 (3 votes) · LW(p) · GW(p)

Named Footnotes: A (likely) mediocre proposal

Epistemic status: This is probably a bad idea, because it's quite obvious yet not done; i.e. Chesterson's fence.

One bad practice in programming is to have a lot of unnamed parameters. For instance,

  createPost(author, post, comment, name, id, privacyOption, ...)


Instead it's generally better to used Named Parameters, like,

  createPost({author, post, comment, name, id, privacyOption})


Footnotes/endnotes seem similar. They are ordered by number, but this can be quite messy. It's particularly annoying for authors not using great footnote-configuring software. If you have 10 endnotes, and then decide to introduce a new one mid-way, you then must re-order all the others after it.

One alternative would be to use what we can call named footnotes or endnotes.

Proposals:

1. This is an example sentence. {consciousness}
2. This is an example sentence. [consciousness]

Systems like this could be pretty easily swapped for numeric footnotes/endnotes as is desired.

One obvious downside may be that these could be a bit harder to find the source for, especially if one is reading it on paper or doesn't have great access to textual search.

comment by Pattern · 2020-01-03T22:45:30.404Z · score: 1 (1 votes) · LW(p) · GW(p)
One bad practice in programming is to have a lot of unnamed parameters.
Footnotes/endnotes seem similar.

That's what arrays are for.

It's particularly annoying for authors not using great footnote-configuring software.

What software does this well?

One alternative would be to use what we can call named footnotes or endnotes.
If you have 10 endnotes,

10 endnotes? Break the document up into sections, and the footnotes up into sections.

and then decide to introduce a new one mid-way, you then must re-order all the others after it.

Authors also could not re-order the footnotes.

textual search.

Or separate drafting and finished product: * indicates a footnote (to be replaced with a number later). At the end, searching * will find the first instance. (If the footnotes are being made at the same, then /* for the notes in the body, and \* for the footnotes at the end. Any uncommon symbols work - like qw.)

comment by ozziegooen · 2020-01-04T16:03:49.646Z · score: 2 (1 votes) · LW(p) · GW(p)

That's what arrays are for.

Arrays are useful for some kinds of things, for sure, but not when you have some very different parameters, especially if they are of different kinds. It would be weird to replace getUser({userId, databaseId, params}) with something like getUser([inputs]) where inputs is an array of [userId, databaseId, params].

What software does this well?

Depends on your definition of "well", but things like Microsoft Word and to what I believe is a lesser extent Google Docs at least have ways of formally handling footnotes/endnotes, which is better than not using these features (like, in most internet comment editors).

10 endnotes? Break the document up into sections, and the footnotes up into sections.

That could work in some cases. I haven't seen that done much on most online blog posts. Also, there's definitely controversy if this is a good idea.

Authors also could not re-order the footnotes.

Fair point, but this would be seen as lazy, and could be confusing. If your footnotes are numbers [8], [2], [3], [1], etc. that seems unpolished. That said, I wouldn't mind this much, and it could be worth the cost.

comment by Pattern · 2020-01-05T05:06:17.880Z · score: 3 (2 votes) · LW(p) · GW(p)
controversy

The page you linked was a great overview. It noted:

If you want to look at the text of a particular endnote, you have to flip to the end of the research paper to find the information.

With a physical document, the two parts (body+endnotes) can be separated for side by side reading. With a digital document, it helps to have two copies open.

Depends on your definition of "well", but things like Microsoft Word and to what I believe is a lesser extent Google Docs at least have ways of formally handling footnotes/endnotes, which is better than not using these features (like, in most internet comment editors).

This seems like a problem with an easy solution. (The difficult solution is trying to make it easier for website makers to get more sophisticated editors in their comments section.)

A brief search for browser extensions suggests it might be possible with an extension that offers an editor, or made easier with one that allows searching multiple things and highlighting them in different colors.

Alternatively, a program for this might:

Make sure every pair of []s is closed.*

Find the strings contained in []s.

Make sure they appear (at most) two times.**

If they appear 3 times, increment the later one (in both places where it appears, if applicable). This requires that the footnotes below already be written. This requirement could be removed if the program looked for (or created) a "Footnotes" or "Endnotes" header (just that string and an end of line), and handled things differently based on that.

Such a program could be on a website, though that's requires people bother switching to that, which, even if bookmarked, is only slightly easier than opening an editor.

As a browser extension, it would have to figure out/be told 1. what part of the text it's supposed to work on, 2. when to be active, and 3. how to make the change.

1. could be done by having the message begin with start, and end with end, as long as the page doesn't include those words (with []s in between them).

2. This could be done automatically, or with a button.

3. Change the text automatically, or make a suggestion?

Fair point, but this would be seen as lazy, and could be confusing. If your footnotes are numbers [8], [2], [3], [1], etc. that seems unpolished. That said, I wouldn't mind this much, and it could be worth the cost.

This could simplify things - if people took things in that format and ran them through a program that fixed it/pointed it out (and maybe other small mistakes).

*A programming IDE does this.

**And this as well, with a bit more work, enabling named footnotes. The trick would be making extensions that make this easy to use for something other than it was intended, or copying the logic.

comment by ozziegooen · 2020-01-08T19:55:47.765Z · score: 2 (1 votes) · LW(p) · GW(p)

Prediction evaluations may be best when minimally novel

Imagine a prediction pipeline is resolved with a human/judgemental evaluation. For instance, a group today starts predicting what a trusted judge 10 years from now will say for the question, "How much counterfactual GDP benefit did policy X make, from 2020-2030?"

So, there are two stages:

1. Prediction
2. Evaluation

One question for the organizer of such a system is how many resources to delegate to the prediction step vs. the evaluation step. It could be expensive to both pay for predictors and evaluators, so it's not clear how to weigh these steps against each other.

I've been suspecting that there are methods to be stingy with regards to the evaluators, and I have a better sense now why that is the case.

Imagine a model where the predictors gradually discover information I_predictors about I_total, the true ideal information needed to make this estimate. Imagine that they are well calibrated, and use the comment sections to express their information when predicting.

Later the evaluator comes by. Because they could read everything so far, they start with I_predictors. They can use this to calculate Prediction(I_predictors), although this should have already been estimated from the previous predictors (a la the best aggregate).

At this point the evaluator can choose to attempt to get more information, I_evaluation > I_predictors. However, if they do, the resulting probability distribution would be predicted by Prediction(I_predictors). Insofar as the predictors are concerned, the expected value of Prediction(I_evaluation) should be the same as that of Prediction(I_predictors), assuming that Prediction(I_predictors) is calibrated; except for the fact that it will be have more risk/randomness. Risk is generally not a desirable property. I've written about similar topics in this post [LW(p) · GW(p)].

Therefor, the predictors should generally prefer Prediction(I_predictors) to Prediction(I_evaluator), as long as everyone's predictions are properly calibrated. This difference shouldn't generally lead to a difference of predictions from them unless a complex or odd scoring rule were used.

Of course, calibration can't be taken for granted. So pragmatically, the evaluator would likely have to deal with issues of calibration.

This setup also assumed that maximally useful comments are made available to evaluator. I think predictors will generally want the evaluator to see much of their information, as it would in general support their sides.

A relaxed version of this may be that the evaluators' duty would be to get approximately all the information that the predictors had access to, but more is not necessary.

Note that this model is only interested in the impact of good evaluation on the predictions. Evaluation also would lead to "externalities"; information that would be useful in other ways as well. This information isn't included here, but I'm fine with that. I think we should generally expect predictors to be more cost-effective than evaluators at doing "prediction work" (i.e. the main reason we have separated anyway!)

TLDR
The role of evaluation could be to ensure that predictions were reasonably calibrated and that the aggregation thus did a decent job. Evaluators shouldn't don't have to outperform the aggregate, if that requires outside information from what was used in the predictions.