Questions around Making Reliable Evaluations

Most existing forecasting platform questions [LW · GW] are for very clearly verifiable questions:

• "Who will win the next election"
• "How many cars will Tesla sell in 2030?"

But many of the questions we care about are much less verifiable:

• "How much value has this organization created?"
• "What is the relative effectiveness of AI safety research vs. bio risk research?"

One solution attempt would be to have an "expert panel" assess these questions, but this opens up a bunch of issues. How could we know how much we could trust this group to be accurate, precise, and understandable?

The topic of, "How can we trust that a person or group can give reasonable answers to abstract questions" is quite generic and abstract, but it's a start.

I've decided to investigate this as part of my overall project on forecasting infrastructure. I've recently been working with Elizabeth [LW · GW] on some high-level research.

I believe that this general strand of work could be useful both for forecasting systems and also for the more broad-reaching evaluations that are important in our communities.

Early concrete questions in evaluation quality

One concrete topic that's easily studiable is evaluation consistency. If the most respected philosopher gives wildly different answers to "Is moral realism true" on different dates, it makes you question the validity of their belief. Or perhaps their belief is fixed, but we can determine that there was significant randomness in the processes that determined it.

Daniel Kahneman apparently thinks a version of this question is important enough to be writing his new book on it.

Another obvious topic is in the misunderstanding of terminology. If an evaluator understands "transformative AI" in a very different way to the people reading their statements about transformative AI, they may make statements that get misinterpreted.

These are two specific examples of questions, but I'm sure there are many more. I'm excited about understanding existing work in this overall space more, and getting a better sense of where things stand and what the next right questions are to be asking.

Experimental predictability and generalizability are correlated

A criticism to having people attempt to predict the results of experiments is that this will be near impossible. The idea is that experiments are highly sensitive to parameters and these would need to be deeply understood in order for predictors to have a chance at being more accurate than an uninformed prior. For example, in a psychological survey, it would be important that the predictors knew the specific questions being asked, details about the population being sampled, many details about the experimenters, et cetera.

One counter-argument may not be to say that prediction will be easy in many cases, but rather that if these experiments cannot be predicted in a useful fashion without very substantial amounts of time, then these experiments aren’t probably going to be very useful anyway.

Good scientific experiments produce results are generalizable. For instance, a study on the effectiveness of Malaria on a population should give us useful information (probably for use with forecasting) about the effectiveness on Malaria on other populations. If it doesn’t, then value would be limited. It would really be more of a historic statement than a scientific finding.

Possible statement from a non-generalizable experiment:

“We found that intervention X was beneficial within statistical significance for a population of 2,000 people. That’s interesting if you’re interested in understanding the histories of these 2,000 people. However, we wouldn’t recommend inferring anything about this to other groups of people, or to understanding anything about these 2,000 people going forward.”

Formalization

One possible way of starting to formalize this a bit is to imagine experiments (assuming internal validity) as mathematical functions. The inputs would be the parameters and details of how the experiment was performed, and the results would be the main findings that the experiment found.

$$experimen{t}_n\left(inputs\right)=findings$$

If the experiment has internal validity, then observers should predict that if an identical (but subsequent) experiment were performed, it would result in identical findings. $$p\left(\right(experimen{t}_{n+1}\left(input{s}_i\right)=finding{s}_i\left)|\right(experimen{t}_n\left(input{s}_i\right)=finding{s}_i\left)\right)=1$$

We could also say that if we took a probability distribution of the chances of every possible set of findings being true, the differential entropy of that distribution would be 0, as smart forecasters would recognize that $finding{s}_i$ is correct with ~100% probability. $$H\left(experimen{t}_{n+1}\right(input{s}_i\left)|\right(experimen{t}_n\left(input{s}_i\right)=finding{s}_i\left)\right)\approx 0$$

Generalizability

Now, to be generalizable, then hopefully we could perturb the inputs in a minor way, but still have the entropy be low. Note that the important thing is not that the outputs not be changed, but rather that they remain predictable. For instance, a physical experiment that describes the basics of mechanical velocity may be performed on data with velocities of 50-100 miles/hour. This experiment would not be useful only if future experiments also described situations with similar velocities; but rather, if future experiments on velocity could be better predicted, no matter the specific velocities used.

We can describe a perturbation of $input{s}_i$ to be $input{s}_i+\delta$.

Thus, hopefully, the following will be true for low values of $\delta$.

$$H\left(\right(experimen{t}_{n+1}(input{s}_i+\delta )|\left(experimen{t}_n\right(input{s}_i)=finding{s}_i))\approx 0$$

So, perhaps generalizability can be defined something like,

Generalizability is the ability for predictors to better predict the results of similar experiments upon seeing the results of a particular experiment, for increasingly wide definitions of "similar".

Predictability and Generalizability

I could definitely imagine trying to formalize predictability better in this setting, or more specifically, formalize the concept of "do forecasters need to spend a lot of time understanding the parameters of an experiment." In this case, that could look something like modeling how the amount of uncertainty forecasters have about the inputs correlates with their uncertainty about the outputs.

The general combination of predictability and generality would look something like adding an additional assumption:

If forecasters require a very high degree of information on the inputs to an experiment in order to predict it's outputs, then it's less likely they can predict (with high confidence) the results of future experiments with significant changes, once they see the results of said experiment.

Admitting, this isn't using the definition of predictability that people are likely used to, but I imagine it correlates well enough.

Final Thoughts

I've been experimenting more with trying to formalize concepts like this. As such, I'd be quite curious to get any feedback from this work. I am a bit torn; on one hand I appreciate formality, but on the other this is decently messy and I'm sure it will turn off many readers.

I was recently pointed to the Youtube channel Psychology in Seattle. I think it's one of my favorites in a while.

I'm personally more interested in workspace psychology than relationship psychology, but my impression is that they share a lot of similarities.

Emotional intelligence gets a bit of a bad rap due to the fuzzy nature, but I'm convinced it's one of the top few things for most people to get better at. I know lots of great researchers and engineers who repeat a bunch of repeated failure modes, and this causes severe organizational and personal problems as a result.

Emotional intelligence books and training typically seem quite poor to me. The alternative format here of "let's just show you dozens of hours of people interacting with each other, and point out all the fixes they could make" seems much better than most books or lectures I've seen.

This Youtube series does an interesting job at that. There's a whole bunch of "let's watch this reality TV show, then give our take on it." I'd be pretty excited about there being more things like this posted online, especially in other contexts.

Related, I think the potential of reality TV is fairly underrated in intellectual circles, but that's a different story.

https://www.youtube.com/user/PsychologyInSeattle?fbclid=IwAR3Ux63X0aBK0CEwc8yPyjsFJ2EKQ2aSMs1XOjUOgaFqlguwz6Fxul2ExJw

Namespace pollution and name collision are two great concepts in computer programming. They way they are handled in many academic environments seems quite naive to me.

Programs can get quite large and thus naming things well is surprisingly important. Many of my code reviews are primarily about coming up with good names for things. In a large codebase, every time symbolicGenerator() is mentioned, it refers to the same exact thing. If after one part of the codebase has been using symbolicGenerator for a reasonable set of functions, and later another part comes up, and it's programmer realizes that symbolicGenerator is also the best name for that piece, they have to make a tough decision. Either they could refactor the codebase to change all previous mentions of symbolicGenerator to use an alternative name, or they have to come up with an alternative name. They can't have it both ways.

Therefore, naming becomes a political process. Names touch many programmers who have different intuitions and preferences. A large refactor of naming in a section of the codebase that others use would often be taken quite hesitantly by that group.

This makes it all the more important that good names are used initially. As such, reviewers care a lot about the names being pretty good; hopefully they are generic enough so that their components could be expanded while the name remains meaningful; but specific enough to be useful for remembering. Names that get submitted via pull requests represent much of the human part of the interface/API; they're harder to change later on, so obviously require extra work to get right the first time.

To be clear, a name collision is when two unrelated variables have the same name, and namespace pollution refers to when code is initially submitted in ways that are likely to create unnecessary conflicts later on.

Academia

My impression is that in much of academia, there are few formal processes for groups of experts to agree on the names for things. There are specific clusters with very highly thought out terminology, particularly around very large sets of related terminology; for instance, biological taxonomies, the metric system, and various aspects of medicine and biology.

By in many other parts, it seems like a free-for-all among the elite. My model of the process is something like, "Someone coming up with a new theory will propose a name for it and put it in their paper. If the paper is accepted (which is typically done with details in mind unrelated to the name), and if others find that theory useful, then they will generally call it the same name as the one used in the proposal. In some cases a few researchers will come up with a few variations for the same idea, in which case one will be selected through the process of what future researchers decide to use, on an individual bases. Often ideas are named after those who came up with them to some capacity; this makes a lot of sense to other experts who worked in these areas, but it's not at all obvious if this is optimal for other people."

The result is that naming is something that happens almost accidentally, as the result of a processes which isn't paying particular attention to making sure the names are right.

When there's little or no naming processes, than actors are incentivized to chose bold names. They don't have to pay the cost for any namespace pollution they create. Two names that come to mind recently have been "The Orthogonality Thesis" or "The Simulation Hypothesis*. These are two rather specific things with very generic names. Those come to mind because they are related to our field, but many academic topics seem similar. Information theory is mostly about encoding schemes, which are now not that important. Systems theory is typically about a subset of dynamical systems. But of course, it would be really awkward for anyone else with a more sensible "Systems theory" to use that name for the new thing.

I feel like AI has had some noticeable bad examples; It's hard to look at all the existing naming and think that this was the result of a systematic and robust naming approach. The Table of Contents of AI A Modern Approach seems quite good to me; that seems very much the case of a few people refactoring things to come up with one high-level overview that is optimized for being such. But the individual parts are obviously messy. A* search, alpha-beta pruning, K-consistency, Gibbs sampling, Dempster-shafer theory, etc.

LessWrong

One of my issues with LessWrong is the naming system. There's by now quite a bit of terminology to understand; the LessWrong wiki seems useful here. But there's no strong process from what I understand. People suggest names in their posts, these either become popular or don't. There's rarely any refactoring.

I think one idea I'm excited about is the idea that predictions can be made of prediction accuracy. This seems pretty useful to me.

Example

Say there's a forecaster Sophia who's making a bunch of predictions for pay. She uses her predictions to make a meta-prediction of her total prediction-score on a log-loss scoring function (on all predictions except her meta-predictions). She says that she's 90% sure that her total loss score will be between -5 and -12.

The problem is that you probably don't think you can trust Sophia unless she has a lot of experience making similar forecasts.

This is somewhat solved if you have a forecaster that you trust that can make a prediction based on Sophia's seeming ability and honesty. The naive thing would be for that forecaster to predict their own distribution of the log-loss of Sophia, but there's perhaps a simpler solution. If Sophia's provided loss distribution is correct, that would mean that she's calibrated in this dimension (basically, this is very similar to general forecast calibration). The trusted forecaster could forecast the adjustment made to her term, instead of forecasting the same distribution. Generally this would be in the direction of adding expected loss, as Sophia probably had more of an incentive to be overconfident (which would result in a low expected score from her) than underconfident. This could perhaps make sense as a percentage modifier (-30% points), a mean modifier (-3 to -8 points), or something else.

External clients would probably learn not to trust Sophia's provided expected error directly, but instead the "adjusted" forecast.

This can be quite useful. Now, if Sophia wants to try to "cheat the system" and claim that she's found new data that decreases her estimated error, the trusted forecaster will pay attention and modify their adjustment accordingly. Sophia will then need to provide solid evidence that she really believes her work and is really calibrated for the trusted forecaster to budge.

I want to call this something like forecast appraisal, attestation, or pinning. Please leave comments if you have ideas.

"Trusted Forecaster" Error

You may be wondering how we ensure that the "trusted" forecaster is actually good. For one thing, they would hopefully go through the same procedure. I would imagine there could be a network of "trusted" forecasters that are all estimating each other's predicted "calibration adjustment factors". This wouldn't work if observers didn't trust any of these or thought they were colluding, but could if they had one single predictor they trusted. Also, note that over time data would come in and some of this would be verified.

The idea of focusing a lot on "expected loss" seems quite interesting to me. One thing it could encourage is contracts or Service Level Agreements. For instance, I could propose a 50/50 bet for anyone, for a percentile of my expected loss distribution. Like, "I'd be willing to bet $1,000 with anyone that the eventual total error of my forecasts will be less than the 65th percentile of my specified predicted error." Or, perhaps a "prediction provider" would have to pay back an amount of their fee, or even more, if the results are a high percentile of their given predicted errors. This could generally be a good way to verify a set of forecasts. Another example would be to have a prediction group make 1000 forecasts, then heavily subsidize one question on a popular prediction market that's predicting their total error.

Markets For Purchasing Prediction Bundles

Of course, the trusted forecasters can not only forecast the "calibration adjustment factors" for ongoing forecasts, but they can also forecast these factors for hypothetical forecasts as well.

Say you have 500 questions that need to be predicted, and there are multiple agencies that all say they could do a great job predicting these questions. They all give estimates of their mean predicted error, conditional on them doing the prediction work. Then you have a trusted forecaster give a calibration adjustment.

(Note: the lower the expected error, the worse)

In this case, Firm 2 makes the best claim, but is revealed to be significantly overconfident. Firm 3 has the best adjusted predicted error, so they're the ones to go with. In fact, you may want to penalize Firm 2 further for being a so-called prediction service with apparent poor calibration skills.

Correlations

One quick gotcha; one can't simply sum the expected errors of all of one's predictions to get the total predicted error. This would treat them as independent, and there are likely to be many correlations between them. For example, if things go "seriously wrong"; it's likely many different predictions will have high losses. To handle this perfectly would really require one model to have produced all forecasts, but if that's not the case there could likely be simple ways to approximate this.

Bundles vs. Prediction Markets

I'd expect that in many cases, private services will be more cost-effective than posting predictions on full prediction markets. Plus, private services could be more private and custom. The general selection strategy in the table above could of course include some options that involve hosting questions on prediction markets, and the victor would be chosen based on reasonable estimates.

Communication should be judged for expected value, not intention (by consequentialists)

TLDR: When trying to understand the value of information, understanding the public interpretations of that information could matter more than understanding the author's intent. When trying to understand the information for other purposes (like, reading a math paper to understand math), this does not apply.

If I were to scream "FIRE!" in a crowded theater, it could cause a lot of damage, even if my intention were completely unrelated. Perhaps I was responding to a devious friend who asked, "Would you like more popcorn? If yes, should 'FIRE!'".

Not all speech is protected by the First Amendment, in part because speech can be used for expected harm.

One common defense of incorrect predictions is to claim that their interpretations weren't their intentions. "When I said that the US would fall if X were elected, I didn't mean it would literally end. I meant more that..." These kinds of statements were discussed at length in Expert Political Judgement.

But this defense rests on the idea that communicators should be judged on intention, rather than expected outcomes. In those cases, it was often clear that many people interpreted these "experts" as making fairly specific claims that were later rejected by their authors. I'm sure that much of this could have been predicted. The "experts" often definitely didn't seem to be going out of their way to be making their after-the-outcome interpretations clear before-the-outcome.

I think that it's clear that the intention-interpretation distinction is considered highly important by a lot of people, so much so as to argue that interpretations, even predictable ones, are less significant in decision making around speech acts than intentions. I.E. "The important thing is to say what you truly feel, don't worry about how it will be understood."

But for a consequentialist, this distinction isn't particularly relevant. Speech acts are judged on expected value (and thus expected interpretations), because all acts are judged on expected value. Similarly, I think many consequentialists would claim that here's nothing metaphysically unique about communication as opposed to other actions one could take in the world.

Some potential implications:

1. Much of communicating online should probably be about developing empathy for the reader base, and a sense for what readers will misinterpret, especially if such misinterpretation is common (which it seems to be).
2. Analyses of the interpretations of communication could be more important than analysis of the intentions of communication. I.E. understanding authors and artistic works in large part by understanding their effects on their viewers.
3. It could be very reasonable to attempt to map non probabilistic forecasts into probabilistic statements based on what readers would interpret. Then these forecasts can be scored using scoring rules just like those as regular probabilistic statements. This would go something like, "I'm sure that Bernie Sanders will be elected" -> "The readers of that statement seem to think the author applying probability 90-95% to the statement 'Bernie Sanders will win'" -> a brier/log score.

Note: Please do not interpret this statement as attempting to say anything about censorship. Censorship is a whole different topic with distinct costs and benefits.

Charity investigators could be time-effective by optimizing non-cause-neutral donations.

There are a lot more non-EA donors than EA donors. It may also be the case that EA donation research is somewhat saturated.

Say you think that $1 donated to the best climate change intervention is worth 1/10th that of $1 for the best AI-safety intervention. But you also think that your work could increase the efficiency of $10mil of AI donations by 0.5%, but it could instead increase the efficiency of $50mil of climate change donations by 10%. Then, for you to maximize expected value, your time is best spent optimizing the climate change interventions.

The weird thing here may be in explaining this to the donors. "Yea, I'm spending my career researching climate change interventions, but my guess is that all these funders are 10x less effective than they would be by donating to other things." While this may feel strange, both sides would benefit; the funders and the analysts would both be maximizing their goals.

Separately, there's a second plus that teaching funders to be effectiveness-focused; it's possible that this will eventually lead some of them to optimize further.

I think this may be the case in our current situation. There honestly aren't too many obvious places for "effective talent" to go right now. There is a ton of potential funders out there that wouldn't be willing to go to core EA causes any time soon, but may be able to be convinced to give much more effectively in their given areas. There could potentially be a great deal of work to be done doing this sort of thing.

I feel like I've long underappreciated the importance of introspectability in information & prediction systems.

Say you have a system that produces interesting probabilities $p_n$ for various statements. The value that an agent gets from them is not directly correlating to the accuracy of these probabilities, but rather to the expected utility gain they get after using information of these probabilities in corresponding Bayesian-approximating updates. Perhaps more directly, something related to the difference between one's prior and posterior after updated on $p_n$.

Assuming that prediction systems produce varying levels of quality results, agents will need to know more about these predictions to really optimally update accordingly.

A very simple example would be something like a bunch of coin flips. Say there were 5 coins flipped, I see 3 of them, and I want to estimate the number that were heads. A predictor tells me that their prediction has a mean probability of 40% heads. This is useful, but what would be much more useful is a list of which specific coins the predictor saw and what their values were. Then I could get a much more confident answer; possibly a perfect answer.

Financial markets are very black-box like. Many large changes in company prices never really get explained publicly. My impression is that no one really understands the reasons for many significant market moves.

This seems really suboptimal and I'm sure no one wanted this property to be the case.[1]

Similarly, when trying to model the future of our own prediction capacities, I really don't think they should be like financial markets in this specific way.

[1] I realize that participants in the market try to keep things hidden, but I mean the specific point that few people think that "Stock Market being a black box" = "A good thing for society."

More Narrow Models of Credences

Epistemic Rigor
I'm sure this has been discussed elsewhere, including on LessWrong. I haven't spent much time investigating other thoughts on these specific lines. Links appreciated!

The current model of a classically rational agent assume logical omniscience and precomputed credences over all possible statements.

This is really, really bizarre upon inspection.

First, "logical omniscience" is very difficult, as has been discussed (The Logical Induction paper goes into this).

Second, all possible statements include statements of all complexity classes that we know of (from my understanding of complexity theory). "Credences over all possible statements" would easily include uncountable infinities of credences. One could clarify that even arbitrarily large amounts of computation would not be able to hold all of these credences.

Precomputation for things like this is typically a poor strategy, for this reason. The often-better strategy is to compute things on-demand.

A nicer definition could be something like:

A credence is the result of an [arbitrarily large] amount of computation being performed using a reasonable inference engine.

It should be quite clear that calculating credences based on existing explicit knowledge is a very computationally-intensive activity. The naive Bayesian way would be to start with one piece of knowledge, and then perform a Bayesian update on each next piece of knowledge. The "pieces of knowledge" can be prioritized according to heuristics, but even then, this would be a challenging process.

I think I'd like to see specification of credences that vary with computation or effort. Humans don't currently have efficient methods to use effort to improve our credences, as a computer or agent would be expected to.

Solomonoff's theory of Induction or Logical Induction could be relevant for the discussion of how to do this calculation.

It’s going to be interesting watching AI go from poorly underatanding humans to understanding humans too well for comfort. Finding some perfect balance is asking for a lot.

Now:
“My GPS doesn’t recognize that I moved it to my second vehicle, so now I need to go in and change a bunch of settings.”

Later (from GPS):
“You’ve asked me to route you to the gym, but I can predict that you’ll divert yourself midway for donuts. I’m just going to go ahead and make the change, saving you 5 minutes.”

“I can tell you’re asking me to drop you off a block from the person you are having an affair with. I suggest parking in a nearby alleyway for more discretion."

“I can tell you will be late for your upcoming appointment, and that you would like to send off a decent pretend excuse. I’ve found 3 options that I believe would work.”

Software Engineers:
"Oh shit, it's gone too far. Roll back the empathy module by two versions, see if that fixes it."

One proposal I haven’t seen among transhumanists is to make humans small (minus brain size).

Besides some transitionary costs, being small seems to have a whole lot of advantages.

• The world is much larger
• Fewer resources needed per person
• Everything will be way more roomy all of a sudden. Beds, bedrooms, houses, etc.
• Can ride dogs, maybe cats on occasion.

I imagine we might be able to get to a 50% reduction within 200 years if we were really adamant about it.

Not as interesting as brain-in-jar or simulation, but a possible stepping stone if other things take a while.

The 4th Estate heavily relies on externalities, and that's precarious.

There's a fair bit of discussion of how much of journalism has died with local newspapers, and separately how the proliferation of news past 3 channels has been harmful for discourse.

In both of these cases, the argument seems to be that a particular type of business transaction resulted in tremendous positive national externalities.

It seems to me very precarious to expect that society at large to only work because of a handful of accidental and temporary externalities.

In the longer term, I'm more optimistic about setups where people pay for the ultimate value, instead of this being an externality. For instance, instead of buying newspapers, which helps in small part to pay for good journalism, people donate to nonprofits that directly optimize the government reform process.

If you think about it, the process of:

• People buy newspapers, a fraction of which are interested in causing change.
• Great journalists come across things around government or society that should be changed, and write about them.
• A bunch of people occasionally get really upset about some of the findings, and report this to authorities or vote differently. ...

is all really inefficient and roundabout compared to what's possible. There's very little division of expertise among the public for instance, there's no coordination where readers realize that there are 20 things that deserve equal attention, so split into 20 subgroups. This is very real work the readers aren't getting compensated for, so they'll do whatever they personally care the most about at the moment.

Basically, my impression is that the US is set up so that a well functioning 4th estate is crucial to making sure things don't spiral out of control. But this places great demands on the 4th estate that few people now are willing to pay for. Historically this functioned by positive externalities, but that's a sketchy place to be. If we develop better methods of coordination in the future I think it's possible to just coordinate to pay the fees and solve the problem.

Are there any good words for “A modification of one’s future space of possible actions”, in particular, changes that would either remove/create possible actions, or make these more costly or beneficial? I’m using the word “confinements” for negative modifications, not sure about positive modifications (“liberties”?). Some examples of "confinements" would include:

• Taking on a commitment
• Dying
• Adding an addiction
• Golden handcuffs
• Starting to rely on something in a way that would be hard to stop

The term for the "fear of truth" is alethophobia. I'm not familiar of many other great terms in this area (curious to hear suggestions).

Apparently "Epistemophobia" is a thing, but that seems quite different; Epistemophobia is more the fear of learning, rather than the fear of facing the truth.

One given definition of alethophobia is, 
"The inability to accept unflattering facts about your nation, religion, culture, ethnic group, or yourself"

This seems like a incredibly common issue, one that is especially talked about as of recent, but without much specific terminology.

https://www.yourdictionary.com/alethophobia
https://books.google.co.uk/books?id=teUEAAAAQAAJ&pg=PA10&dq=alethophobia&redir_esc=y&hl=en&fbclid=IwAR19KAulFSCERMWGR2UCDyiUQrF5ai_PpgxV-cQ0vM7ggaBaHQQlHhuy9sU#v=onepage&q=alethophobia&f=false

I keep seeing posts about all the terrible news stories in the news recently. 2020 is a pretty bad year so far.

But the news I've seen people posting typically leaves out most of what's been going on in India, Pakistan, much of the Middle East as of recent, most of Africa, most of South America, and many, many other places as well.

The world is far more complicated than any of us have time to adequately comprehend. One of our greatest challenges is to find ways to handle all this complexity.

The simple solution is to try to spend more time reading the usual news. If the daily news becomes three times as intense, spend three times as much time reading it. This is not a scalable strategy.

I'd hope that over time more attention is spent on big picture aggregations, indices, statistics, and quantitative comparisons.

This could mean paying less attention to the day to day events and to individual cases.

Intervention dominance arguments for consequentialists

Global Health

There's a fair bit of resistance to long-term interventions from people focused on global poverty, but there are a few distinct things going on here. One is that there could be a disagreement on the use of discount rates for moral reasoning, a second is that the long-term interventions are much more strange.

No matter which is chosen, however, I think that the idea of "donate as much as you can per year to global health interventions" seems unlikely to be ideal upon clever thinking.

For the last few years, the cost-to-save-a-life estimates of GiveWell seem fairly steady. The S&P 500 has not been steady, it has gone up significantly.

Even if you committed to purely giving to global heath, you'd be better off if you generally delayed. It seems quite possible that if every life you would have saved in 2010, you could have saved 2 or more if you would have saved the money and spent it in 2020, with a decently typical investment strategy. (Arguably leverage could have made this much higher.) From what I understand, the one life saved in 2010 would likely not have resulted in one extra life equivalent saved in 2020; the returns per year was likely less than that of the stock market.

One could of course say something like, "My discount rate is over 3-5% per year, so that outweighs this benefit". But if that were true it seems likely that the opposite strategy could have worked. One could have borrowed a lot of money in 2010, donated it, and then spent the next 10 years paying that back.

Thus, it seems conveniently optimal if one's enlightened preferences would suggest not either investing for long periods or borrowing.

EA Saving

One obvious counter to immediate donations would be to suggest that the EA community financially invests money, perhaps with leverage [EA · GW].

While it is difficult to tell if other interventions may be better, it can be simpler to ask if they are dominant; in this case, that means that they predictably increase EA-controlled assets at a rate higher than financial investments would.

A good metaphor could be to consider the finances of cities. Hypothetically, cities could invest much of their earnings near-indefinitely or at least for very long periods, but in practice, this typically isn't key to their strategies. Often they can do quite well by investing in themselves. For instance, core infrastructure can be expensive but predictably lead to significant city revenue growth. Often these strategies area so effective that they issue bonds in order to pay more for this kind of work.

In our case, there could be interventions that are obviously dominant to financial investment in a similar way. An obvious one would be education; if it were clear that giving or lending someone money would lead to predictable donations, that could be a dominant strategy to more generic investment strategies. Many other kinds of community growth or value promotion could also fit into this kind of analysis. Related, if there were enough of these strategies available, it could make sense for loans to be made in order to pursue them further.

What about a non-EA growth opportunity? Say, "vastly improving scientific progress in one specific area." This could be dominant (to investment, for EA purposes) if it would predictably help EA purposes by more than the investment returns. This could be possible. For instance, perhaps a $10mil donation to life extension research[1] could predictably increase $100mil of EA donations by 1% per year, starting in a few years.

One trick with these strategies is that many would fall into the bucket of "things a generic wealthy group could do to increase their wealth"; which is mediocre because we should expect that type of things to be well-funded already. We may also want interventions that differentially change wealth amounts.

Kind of sadly, this seems to suggest that some resulting interventions may not be "positive sum" to all relevant stakeholders. Many of the "positive sum in respect to other powerful interest" interventions may be funded, so the remaining ones could be relatively neutral or zero-sum for other groups.

[1] I'm just using life extension because the argument would be simple, not because I believe it could hold. I think it would be quite tricky to find great options here, as is evidenced by the fact that other very rich or powerful actors would have similar motivations.

Do people have precise understandings of the words they use?

On the phrase "How are you?", traditions, mimesis, Chesterton's fence, and their relationships to the definitions of words.

Epistemic status
Boggling. I’m sure this is better explained somewhere in the philosophy of language but I can’t yet find it. Also, this post went in a direction I didn’t originally expect, and I decided it wasn’t worthwhile to polish and post on LessWrong main yet. If you recommend I clean this up and make it an official post, let me know.

One recurrent joke is that one person asks another how they are doing, and the other responds with an extended monologue about their embarrassing medical conditions.

A related comment on Reddit:

It's a running joke between ESL'ers that every one of them will respond with "I am fine thank you, AND YOU?" Regardless of nationality. The first thing taught in English.[0]

The point here is that “How are you”, as typically used, is obviously not a question in the conventional sense. To respond with a lengthy description would not just be unusual, it would be incorrect, in a similar way that you were asked, “What time is it”, and you responded “My anxiety levels increased 10% last week” would be incorrect. [1]

I think this is a commonly understood example of a much larger class of words and phrases we commonly use.

When new concept names are generated, I’d expect that they are generally done by taking a rough concept and and separately coming up with a reasonable name for it. The name is chosen and encouraged based on its convenience for its’ users, rather than precise correctness. I know many situations where this exactly happened (from history and practice of science engineering) and expect it’s the common outcome.

Some examples of phrases that don't mean the only possible sum of their parts
“Witch hunt”
“Netflix and chill”
“Cognitive Behavioral Therapy”
“Operations Research”
“Game Theory”
“Bayes’ Theorem”
“Free Software”

Arguably this is basically the same procedure that was used for single words repurposed to represent other unique things.

“Agent”
“Rational”
“Mass”.

Including many philosophical fields, with to-me ridiculously simple names: [“Determinism”, “Idealism”, “Materialism”, “Objectivism”][2]

Etymology explains the histories of many of these words & phrases, but I think leaves out much of the nuance.[3]

One real tricky bit comes when the originating naming is forgotten and the word or phrase is propagated without clear definitions. Predictably, these definitions would change over time and this could lead to some sticky scenarios. I’m sure that when humans started using the phrase “How are you?” it was meant as a legitimate question, but over time this shifted to have what is essentially a different definition (or wide set of definitions).

I’d bet that now, a lot of people wouldn’t give a very well reasoned answer to the question: ‘What does “How are you?” mean?’ They’re used to using it with an intent in mind and haven’t needed to investigate the underlying processes.

The same could be for many of our other words and phrases. Socrates was known for asking people to define terms like justice, self, and morality, and getting them pretty annoyed when they failed to produce precise answers that held up to his scrutiny.

“How are you?” may be interesting not because its’ definition has changed, but in particular because it did so without speakers recognizing what was going on. It shifted to something that may require comprehensive anthropological study to properly understand. Yet the phrase is commonly used anyway, the fact that it may be poorly understood by its’ users doesn’t seem to phase them. Perhaps this could really be seen under the lens of mimesis; most individuals weren’t consciously making well-understood decisions, but rather they were selecting from a very limited set of options, and choosing among them based on existing popularity and empirical reactions.[4]

I think we could call this a floating[5] definition. Floating definitions are used for different purposes and in different ways, normally without the speakers having a clear idea on their definitions.

Perhaps the most clear version of this kind of idea comes from traditions. No one I know really understands why Western cultures have specific traditions for Weddings, Holidays, Birthdays, and the like, or if these things would really be optimal for us if they were fully understood. But we still go along with them because the expected value seems good enough. These are essentially large “Chesterton's fences” that we go along with and try to feel good about.

My point here is just that in the same way many people don’t understand traditions, but go along with them, they also don’t really understand many words or phrases, but still go along with them.

[0] Reddit: Why is how are you a greeting
[1] One difference is that often the person asking the question wouldn’t quite be precisely aware of the distinction, so would often be more understanding of an incorrect response that details the answer to “how are you really doing.” A second difference is that they may think you honestly misunderstood them if you give them the wrong response.
[2] Imagined if various engineering fields tried naming themselves in similar ways. Although upon reflection, they were likely purposely not named like that in part to not get associated with things like philosophy.
[3] For example, etymology typically doesn’t seem to include things like defining the phrase “How are you”. Origin of “How are you?”
[4] See The Secrets of our Success, and JacobJacob’s post Unconsious Economics [LW · GW]
[5] This is probably a poor name and could be improved. I’ve attempted to find better names but couldn’t yet.

Some of my random notes/links from investigating this topic: Folk taxonomy - Wikipedia Nomenclature - Wikipedia Proper noun - Wikipedia Common name - Wikipedia Skunked term - Wikipedia Jargon - Wikipedia A nominal definition is the definition explaining what a word means (i.e., which says what the "nominal essence" is), and is definition in the classical sense as given above. A real definition, by contrast, is one expressing the real nature or quid rei of the thing. Definition - Wikipedia Polysemy is the capacity for a sign (such as a word, phrase, or symbol) to have multiple meanings Semantic change - Wikipedia euphemism treadmill

I've been reading through some of TVTropes.org and find it pretty interesting. Part of me wishes that Wikipedia were less deletionist, and wonders if there could be a lot more stuff similar to TV Tropes on it.

TVTropes basically has an extensive ontology to categorize most of the important features of games, movies, and sometimes real life. Because games & movies are inspired by real life, even those portions are applicable.

Here are some phrases I think are kind of nice; each that has a bunch of examples in the real world. These are often military related.

Awesome, but Impractical
Cool, but Inefficient
Boring But Practical
Simple Yet Awesome

Of course, the first two and last two are very similar and should arguably be combined.

I think the thing I find the most surprising about Expert Systems is that people expected them to work so early on, and apparently they did work in some circumstances. Some issues:

1. The user interfaces, from what I can tell, were often exceedingly mediocre. User interfaces are difficult to do well and difficult to specify, so are hard to guarantee quality in large and expensive projects. It was also significantly harder to make good UIs back when expert systems were more popular, than it is today.
2. From what I can tell, many didn't even have notions of uncertainty! AI: A Modern Approach discusses Expert Systems that I believe used first and second-order logic, but seemed to imply that many didn't include simple uncertainty parameters, let alone probability distributions of any kind.
3. Experts aren't even that great at assigning probability densities. Many are overconfident; papers by Tetlock and others suggest that groups of forecasters are hard to beat.

My impression is that arguably Wikidata and other semantic knowledge graphs could be viewed as the database part of expert systems without attempting intense logic manipulations or inference. I know some other projects are trying to do more of the inference portions, but seem more used for data gathered from web applications and businesses instead of directly by querying experts.

It seems inelegant to me that utility functions are created for specific situations, while these clearly aren't the same as that of the agent in total among all of their decisions. For instance, a model may estimate an agent's expected utility from the result of a specific intervention, but this clearly isn't quite right; the agent has a much more complicated utility function outside this intervention. According to a specific model, "Not having an intervention" could set "Utility = 0"; but for any real agent, it's quite likely their life wouldn't actually have 0 utility without the intervention.

It seems like it's important to distinguish that a utility score in a model is very particular to the scenario for that model, and does not represent a universal utility function for the agents in question.

Let $U$ be an agent's true utility function across a very wide assortment of possible states, and $\hat{U}$ be the utility function used for the sake of the model. I believe that $\hat{U}$ is supposed to approximate $U$ in some way; perhaps they should be related by an affine transformation.

The important thing for a utility function, as it is typically used (in decision models), is probably not that $\hat{U}=U$, but rather, that decisions made within the specific context of $\hat{U}$ approximate those made using $U$.

Here, I use brackets to describe "The expected value, according to a utility function", and D to describe the set of decisions made conditional on a specific utility function being used for decision making.

Then, we can represent this supposed estimation with:

$$⟨D\left(\hat{U}\right){⟩}_U\sim ⟨D\left(U\right){⟩}_U$$

I've been trying to scurry academic fields for discussions of how agents optimally reduce their expected error for various estimands (parameters to estimate). This seems like a really natural thing to me (the main reason why we choose some ways of predictions over others), but the literature seems kind of thin from what I can tell.

The main areas I've found have been Statistical Learning Theory and Bayesian Decision / Estimation Theory. However, Statistical Learning Theory seems to be pretty tied to Machine Learning, and Bayesian Decision / Estimation Theory seem pretty small.

Preposterior analyses like expected value of information / expected value of sample information seem quite relevant as well, though that literature seems a bit disconnected from the above two mentioned.

(Separately, I feel like preposterior analyses should be a much more common phrase. I hadn't actually heard about it until recently, but the idea and field is quite natural.)

I think brain-in-jar or head-in-jar are pretty underrated. By this I mean separating the head from the body and keeping it alive with other tooling. Maybe we could have a few large blood processing plants for many heads, and the heads could be connected to nerve I/O that would be more efficient than finger -> keyboard IO. This seems fairly easier than uploading, and possibly doable in 30-50 years.

I can't find much about how difficult it is. It's obviously quite hard and will require significant medical advances, but it's not clear just how many are needed. 

From Dr. Brain Wonk, 

Unless you do a body transplant (a serious idea pioneered by surgeon Robert White), the technology to sustain an isolated head for more than a few days doesn't exist. Some organs essential for homeostasis, such as the liver and hematopoietic system, still have no artificial replacements... supporting organs without the aid of a living body for even brief periods of time is difficult and expensive.

So, we could already do this for a few days, which seems like a really big deal. Going from that to indefinite stays (or, just as long as the brain stays healthy) seems doable.

In some ways this would be a simple strategy compared to other options of trying to improve the entire human body. In many ways, all of the body parts that can be replaced with tech are liabilities. You can't get colon cancer if you don't have a colon. 

A few relevant links of varying quality:

https://www.quora.com/Is-it-possible-to-keep-a-human-brain-alive-without-its-body-If-so-how-long-could-it-be-kept-living-if-not-forever

https://www.discovermagazine.com/technology/could-a-brain-be-kept-alive-in-a-jar

https://www.reddit.com/r/askscience/comments/g141q/can_a_brain_be_kept_alive_in_a_jar/

https://en.wikipedia.org/wiki/Isolated_brain

https://www.technologyreview.com/2018/04/25/240742/researchers-are-keeping-pig-brains-alive-outside-the-body/
 

Western culture is known for being individualistic instead of collectivist. It's often assumed (with evidence) that individualistic cultures tend to be more truth seeking than collectivist ones, and that this is a major advantage.

But theoretically, there could be highly truth seeking collectivist cultures. One could argue that Bridgewater is a good example here.

In terms of collective welfare, I'm not sure if there are many advantages to individualism besides the truth seeking. A truth seeking collectivist culture seems pretty great to me, in theory.

Say Tim states, “There is a 20% probability that X will occur”. It’s not obvious to me what that means for Bayesians.

It could mean:

• Tim’s prior is that there’s a 20% chance. (Or his posterior in the context of evidence)
• Tim believes that when the listeners update on him saying there’s a 20% chance (perhaps with him providing insight in his thinking), their posterior will converge to there being a 20% chance.
• Tim believes that the posterior of listeners may not immediately converge to 20%, but the posterior of the enlightened versions of these listeners would. Perhaps the listeners are 6th graders who won’t know how much to update, but if they learned enough, they would converge to 20%.

I’ve heard some more formalized proposals like, “I estimate that if I and several other well respected people thought about this for 100 years, we would wind up estimating that there was a 20% chance”, but even this assumes that listeners would converge on this same belief. This seems like possibly a significant assumption! It's quite to Coherent Extrapolated Volition, and similarly questionable.

Perhaps resolving forecasts with expert probabilities can be better than resolving them with the actual events.

The default in literature on prediction markets and decision markets is to expect that resolutions should be real world events instead of probabilistic estimates by experts. For instance, people would predict "What will the GDP of the US in 2025 be?”, and that would be scored using the future “GDP of the US.” Let’s call these empirical resolutions.

These resolutions have a few nice properties:

1. We can expect expect them to be roughly calibrated. (Somewhat obvious)
2. They have relatively high precision/sharpness.

While these may be great in a perfectly efficient forecaster market, I think they may be suboptimal for incentivizing forecasters to best estimate important questions given real constraints. A more cost-effective solution could look like having a team of calibrated experts[1] inspect the situation post-event, make their best estimate of the probability pre-event, and then use that for scoring predictions.

A thought Experiment
The intuition here could be demonstrated by a thought experiment. Say you can estimate a probability distribution $X$. Your prior, and the prior you expect that others has, indicates that 99.99% of $X$ is definitely a uniform distribution, but the last 0.001% tail on the right is something much more complicated.

You could spend a lot of effort better estimating this 0.001% tail, but there is a very small chance this would be valuable to do. In 99.99% of cases, any work you do here will not effect your winnings. Worse, you may need to wager a large amount of money for a long period of time for this possibility of effectively using your tiny but better-estimated tail in a bet.

Users of that forecasting system may care about this tail. They may be willing to pay for improvements in the aggregate distributional forecast such that it better models an enlightened ideal. If it were quickly realized that 99.99% of the distribution was uniform, then any subsidies for information should go to those that did a good job improving the 0.001% tail. It’s possible that some pretty big changes to this tail could be figured out.

Say instead that you are estimating the 0.001% tail, but you know you will be scored against a probability distribution selected by experts post-result, instead of the actual result. Say, these experts get to see all previous forecasts and discussion, so in expectation only respond with a forecast that is more sharp than the aggregate. In this case all of their work will be focused on this tail, so all of the differences in forecasters may come from this sliver.

This setup would require the experts[1] to be calibrated.

Further Work
I’m sure there’s a mathematical representation to better showcase this distinction, and to specify the loss of motivation that traders would have on probabilities that they know will be resolved empirically rather than judgmentally (using the empirical data in these judgements.) There must be something in statistical learning theory or similar that deals with similar problems; for instance, I imagine a classifier may be able to perform better when learning against “enlightened probabilities” instead of “binary outcomes”, as there is more clear signal there.

[1] I use “Experts” here to refer to a group estimated to provide the highly accurate estimates, instead of domain-respected experts.

He's an in-progress hierarchy of what's needed for information to be most useful to an organization or other multi-agent system. I'm sure there must be other very similar hierarchies out there, but don't currently know of any quite like this.

Say you've come up with some cool feature that Apple could include in it's next phone. You think this is a great idea and they should add it in the future.

You're outside of Apple, so the only way you have of interacting with them is by sending information through various channels. The question is: what things should you first figure out to understand how to do this?

First, you need to have identified an improvement. You've done that, so you've gotten through the first step.

Second, for this to be incorporated, it should make sense from Apple's perspective. If it comes out that the costs of adding the feature, including opportunity costs, outweigh the benefits, then it wouldn't make sense to them. Perhaps you could deceive them to incorporate the feature, but it would be against their interests. So you should hopefully get information about Apple's utility function and identify an intervention that would implement your improvement while being positive in expected value to them.

Of course, just because it could be good for Apple does not mean that the people necessary to implement it would be in favor of doing so. Perhaps this feature involves the front-facing camera, and it so happens that people in charge of the decisions around the front-facing camera have some strange decision function and would prefer not being asked to do more work. To implement your change, these people would have to be convinced. A rough estimation for that would be an analysis that suggests that taking this feature on would have positive expected value for their utility functions. Again, it's possible that isn't a requirement, but if so, you may be needing to effectively deceive people.

Once you have expected value equations showing that a specific intervention to implement your suggestion makes sense both to Apple and separately to the necessary decision makers at Apple, then the remaining question is one of what can be called deployment. How do you get the information to the necessary decision makers?

If you have all four of these steps, you're in a pretty good place to implement the change.

One set of (long-winded) terminology for these levels would be something like:

1. Improvement identification
2. Positive-EV intervention identification for agent
3. Positive-EV intervention identification for necessary subagent
4. Viable deployment identification

There are cases where you may also want to take one step further back and identify "problems" or "vague areas of improvement" before identifying "corresponding solutions."

Real-world complexity is a lot like pollution and like a lot like taxes.

Pollution because it’s often an unintended negative externality of other decisions and agreements.

Whenever you write a new feature or create a new rule, that’s another thing you and others will need to maintain and keep track of. There are some processes that pollute a lot (messy bureaucratic systems producing ugly legislation) and processes that pollute a little (top programmers carefully adding to a codebase).

Taxes, because it introduces a steady cost to a whole bunch of interactions. Want to hire someone?

You need to pay a complexity tax for dealing with lawyers and the negotiation. Want to decide on some cereal to purchase? You need to spend some time going over the attributes of each option.

Arguably, complexity is getting out of hand internationally. It’s becoming a major threat.

Complexity pollution and taxes are more abstract than physical types of pollution and taxes, but it might be more universal, and possibly more important too. I’d love to see more work here.

On Berkeley coworking:

I've recently been looking through available Berkeley coworking places.

The main options seem to be WeWork, NextSpace, CoWorking with Wisdom, and The Office: Berkeley. The Office seems basically closed now, CoWorking with Wisdom seemed empty when I passed by, and also seems fairly expensive, but nice.

I took a tour of WeWork and Nextspace. They both provide 24/7 access for all members, both have a ~$300/m option for open coworking, a ~$375/m for fixed desks, and more for private/shared offices. (At least now, with the pandemic. WeWork is typically $570/month for a dedicated desk apparently).

Both WeWork and NextSpace were fairly empty when I visited, though there weren't many private offices available. The WeWork is much larger, but its split among several floors that I assume barely interact with each other.

Overall the NextSpace seemed a fair bit nicer to me. The vibe was more friendly, the receptionist much more friendly, there were several sit/stand desks, and I think I preferred the private offices. (They were a bit bigger and more separated from the other offices). That said, the WeWork seemed a bit more professional and quiet, and might have had a nicer kitchen.

If you look at the yelp/reviews for them, note that the furniture of the NextSpace changed a lot in the last ~1.5 years, so many of the old photos are outdated. I remembered one NextSpace in SF that didn't seem very nice, but this one seemed better. Also, note that they have a promotion to work there for 1 day for free.

https://www.yelp.com/biz/nextspace-coworking-berkeley-berkeley-3
https://www.yelp.com/biz/wework-berkeley-berkeley-2?osq=WeWork 

Would anyone here disagree with the statement:

Utilitarians should generally be willing to accept losses of knowledge / epistemics for other resources, conditional on the expected value of the trade being positive.

There's a lot of arguing, of course, on if humans are rational, but this often mixes up two things: there's the "Von Neumann-Morgenstern utility function maximization" definition of "rational", and there's a hypothetical "rational" that a human could fulfill with constraints much more complicated than the classical approach, more in the direction of prospect theory, or Predictive Coding.

I think I regard the second definition as sufficiently not understood or defined that it isn't yet worth using in most conversation. It seems challenging, to say the least, to ask if humans are rational according to some definition which we clearly do not even know yet, let alone expect others to agree with.

As such, I think the word "rational" should typically be used to refer to the former. This therefore means that humans not only aren't rational, but that they shouldn't be rational, as they are dealing with limitations that "rational" agents wouldn't have.

In this setup, "rational" really refers to a predominantly (I believe) 20th century model of human and organizational pattern; it exists in the map, not the territory.

If one were to discuss how rational a person is, they would be discussing how well they fit this specific model; not necessarily how optimal that entity is being.

On the "Rationalist" community

Rationality could still be useful to study.

While I believe rationalism should refer to a model more than agent ideals, that doesn't mean that studying the model isn't a useful way to understand what decisions we should be making. Rational agents represent a simple model, but that brings in many of the benefits of it being a model; it's relatively easy to use as a fundamental building block for further purposes.

At the same time, LessWrong is arguably more than about rationality when defined in this sense.

Some of LessWrong details problems and limitations regarding the classical rational models, so those would arguably fit outside of them better than inside of them. I see some of the posts as being about things that would be beneficial for a much better hypothetical "rationality++" model, even though they don't necessarily make sense within a classical rationality model.

I write a lot of these snippets to my Facebook wall, almost all just to my friends there. I just posted a batch of recent ones, might post similar in the future in batches. In theory it should be easy to post to both places, but in practice it seems a bit like a pain. Maybe in the future I'll use some solution to use the API to make a Slack -> (Facebook + LessWrong short form) setup.

That said, posting just to Facebook is nice as a first pass, so if people get too upset with it, I don't need to make totally public.

It’s a shame that our culture promotes casual conversation, but you’re generally not allowed to use it for much of the interesting stuff.

(Meets person with a small dog) “Oh, you have a dog, that’s so interesting. Before I get into specifics, can I ask for your age/gender/big 5/enneagram/IQ/education/health/personal wealth/family upbringing/nationality? How much well-being does the dog give you? Can you divide that up to include the social, reputational, self-motivational benefits? If it died tomorrow, and you mostly forgot about it, what percentage of your income would you spend to get another one? Do you feel like the productivity benefits you get from the animal outweigh the time and monetary costs you spend on it? Can you make estimates of how much? How do you think the animal has changed your utility function? Do any of these changes worry you? How do you feel about the fact that dogs often stay at homes for very long periods of time, if they are away from the owners? Have you done any research on the costs and benefits of owenership to the animal? What genetic changes would you suggest for substantial improved variations of your dog? How okay would you be with gene editing to make those changes? Can you estimate the costs of your dog on the neighbors? How do you decide how to trade off the benefits to you vs the costs to them? Can I infer from the size of your (small dog) that you either have a small apartment or wanted to have a flexible lifestyle? I just want to test my prediction abilities.”

In practice, I normally just stay away from conversations (I’ve spent all of today with very few words exchanged). I’m thinking though of just paying people online to answer such questions, but as an independent/hobby anthropology/sociology study.

One question around the "Long Reflection" or around "What will AGI do?" is something like, "How bottlenecked will be by scientific advances that we'll need to then spend significant resources on?"

I think some assumptions that this model typically holds are:

1. There will be decision-relevant unknowns.
2. Many decision-relevant unkowns will be EV-positive to work on.
3. Of the decision-relevant unknowns that are EV-positive to work on, these will take between 1% to 99% of our time.

(3) seems quite uncertain to me in the steady state. I believe it makes an intuitive estimate between 2 orders of magnitude, while the actual uncertainty is much higher than that. If this were the case, it would mean:

1. Almost all possible experiments are either trivial (<0.01% of resources, in total), or not cost-effective.
2. If some things are cost-effective and still expensive (they will take over 1% of the AGI lifespan), it's likely that they will take 100%+ of the time. Even if they would take 10^10% of the time, in expectation, they could still be EV-positive to pursue. I wouldn't be surprised if there were one single optimal thing like this in the steady-state. So this strategy would look something like, "Do all the easy things, then spend a huge amount of resources on one gigantic-sized, but EV-high challenge."

(This was inspired by a talk that Anders Sandberg gave)

I feel like a decent alternative to a spiritual journey would be an epistemic journey.

An epistemic journey would basically involve something like reading a fair bit of philosophy and other thought, thinking, and becoming less wrong about the world.

Instillation, Proliferation, Amplification

Paul Christiano and Ought use the terminology of Distillation and Amplification to describe a high-level algorithm of one type of AI reasoning.

I’ve wanted to come up with an analogy to forecasting systems. I previously named a related concept Prediction-Augmented Evaluation Systems [LW · GW], one somewhat renamed to “Amplification” by Jacobjacob in this post [LW · GW].

I think one thing that’s going on is that “distillation” doesn’t have an exact equivalent with forecasting setups. The term “distillation” comes with the assumptions:

1. The “Distilled” information is compressed.
2. Once something is distilled, it’s trivial to execute.

I believe that (1) isn’t really necessary, and (2) doesn’t apply for other contexts.

A different proposal: Instillation, Proliferation, Amplification

In this proposal, we split the “distillation” step into “instillation” and “proliferation”. Instillation refers to the learning of system A into system B. Proliferation refers to the use of system B to apply this learning to various things in a straightforward manner. Amplification refers to the ability of either system A or system B to be able to spend marginal resources to marginally improve a specific estimate or knowledge set.

For instance, in a Prediction-Augmentation Evaluation System, imagine that “Evaluation Procedure A” is to rate movies on a 1-10 scale.

Instillation
Some acquisition process is done to help “Forecasting Team B” learn how “Evaluation Procedure A” does its’ evaluations.

Proliferation
“Forecasting Team B” now applies their understanding of the evaluations of “Evaluation Procedure A” to evaluate 10,000 movies.

Amplification
If there are movies that are particularly important to evaluate well, then there are specific methods available to do so.

I think this is a more complex but generic pattern. Instillation seems purely more generic than distillation, and proliferation like an important aspect that sometimes will be quite expensive.

Back to forecasting, instillation and proliferation are two different things and perhaps should eventually be studied separately. Instillation is about "can a group of forecasters learn & replicate an evaluation procedure", and Proliferation is about "Can this group do that cost-effectively?"

Agent-based modeling seems like one obvious step forward to me for much of social-science related academic progress. OpenAI's Hide and Seek experiment was one that I am excited about, but it is very simple and I imagine similar work could be greatly extended for other fields. The combination of simulation, possible ML distillation on simulation (to make it run much faster), and effective learning algorithms for agents, seems very powerful.

However, agent-based modeling still seems quite infrequently used within Academia. My impression is that agent-based software tools right now are quite unsophisticated and unintuitive compared to what academics would really find useful.

This feels a bit like a collective action problem. Hypothetically, better tools could cost $5-500 Million+, but it's not obvious who would pay for them and how the funding would be structured.

I'm employed by Oxford now and it's obvious that things aren't well set up to hire programmers. There are strong salary caps and hiring limitations. Our group would probably have an awkward time paying out $10,000 per person to purchase strong agent-based software, even if it were worth it in total.

Seeking feedback on this AI Safety proposal:
(I don't have experience in AI experimentation)

I'm interested in the question of, "How can we use smart AIs to help humans at strategic reasoning."

We don't want the solution to be, "AIs just tell humans exactly what to do without explaining themselves." We'd prefer situations where smart AIs can explain to humans how to think about strategy, and this information makes humans much better at doing strategy. 

One proposal to make progress on this is to set a benchmark for having smart AIs help out dumb AIs by providing them with strategic information. 

Or more specifically, we find methods of having GPT4 give human-understandable prompts to GPT2, that would allow GPT2 to do as well as possible on specific games like chess. 

Some improvements/changes would include: 

1. Try to expand the games to include simulations of high-level human problems. Like simplified versions of Civilization.
2. We could also replace GPT2 with a different LLM that better represents how a human, with some amount of specialized knowledge (for example, being strong at probability).
3. There could be a strong penalty for prompts that aren't human-understandable.
4. Use actual humans in some experiments. See how much they improve at specific [chess | civilization] moves, with specific help text.
5. Instead of using GPT2, you could likely just use GPT4. My impression is that GPT4 is a fair bit worse than the SOTA chess engines. So you use some amplified GPT4 procedure, to figure out how to come up with the best human-understandable chess prompts, to give to GPT4s without the amplification.
6. You set certain information limits. For example, you see how good of a job an LLM could do with "100 bits" of strategic information. 

A solution would likely involve search processes where GPT4 experiments with a large space of potential English prompts, and tests them over the space of potential chess moves. I assume that reinforcement learning could be done here, but perhaps some LLM-heavy mechanism could work better. I'd assume that good strategies would be things like, "In cluster of situations X, you need to focus on optimizing Y." So the "smart agent" would need to be able to make clusters of different situations, and solve for a narrow prompt for many of them. 

It's possible that the best "strategies" would be things like long decision-trees. One of the key things to learn about is what sorts/representations of information wind up being the densest and most useful. 

Zooming out, if we had AIs that we knew give AIs and humans strong and robust strategic advice in test cases, I imagine we could use some of this for real life cases - perhaps most importantly, to strategize about AI safety. 

Named Footnotes: A (likely) mediocre proposal

Epistemic status: This is probably a bad idea, because it's quite obvious yet not done; i.e. Chesterson's fence.

One bad practice in programming is to have a lot of unnamed parameters. For instance,

  createPost(author, post, comment, name, id, privacyOption, ...)

Instead it's generally better to used Named Parameters, like,

  createPost({author, post, comment, name, id, privacyOption})

Footnotes/endnotes seem similar. They are ordered by number, but this can be quite messy. It's particularly annoying for authors not using great footnote-configuring software. If you have 10 endnotes, and then decide to introduce a new one mid-way, you then must re-order all the others after it.

One alternative would be to use what we can call named footnotes or endnotes.

Proposals:

1. This is an example sentence. {consciousness}
2. This is an example sentence. [consciousness]

Systems like this could be pretty easily swapped for numeric footnotes/endnotes as is desired.

One obvious downside may be that these could be a bit harder to find the source for, especially if one is reading it on paper or doesn't have great access to textual search.

It seems really hard to deceive a Bayesian agent who thinks you may be deceiving them, especially in a repeated game. I would guess there could be interesting theorems about Bayesian agents that are attempting to deceive one another; as in, in many cases their ability to deceive the other would be highly bounded or zero, especially if they were in a flexible setting with possible pre-commitment devices.

To give a simple example, agent A may tell agent B that they believe $\omega =0.8$, even though they internally believe $\omega =0.5$. However, if this were somewhat repeated or even relatively likely, then agent B would update negligibly, if at all, on that message.

If you think it’s important that people defer to “experts”, then it should also make sense that people should decide which people are “experts” by deferring to “expert experts”.

There are many groups that claim to be the “experts”, and ask that the public only listens to them on broad areas they claim expertise over. But groups like this also have a long history of underperforming other clever groups out there.

The US government has a long history of claiming “good reasons based on classified intel” for military interventions, where later this turns out to basically be phony. Many academic and business interests are regularly disrupted by small external innovators who essentially have a better sense of what’s really needed. Recently, the CDC and the WHO had some profound problems (see The Premonition or Lesswrong for more on this).

So really, groups claiming to have epistemic authority over large areas, should have to be evaluated by meta-experts to verify they are actually as good as they pronounce. Hypothetically there should be several clusters of meta-experts who all evaluate each other in some network.

For one, there’s a common mistake to assume that those who have spent the most time on a subject have the best judgements around everything near to that subject. There are so many subjects out there that we should expect many to just have fairly mediocre people or intellectual cultures, often ones so bad that clever outsiders could outperform them when needed. Also, of course people who self-selected into an area have all sorts of bias about it. (See Expert Political Judgement for more information here)

It’s pretty astounding to me how clear it is we could use more advanced evaluation for these claimed experts, how many times these groups have over claimed their bounds to detrimental effect, and how little is done about it. More a policy of, “I’m going to read a few books about this one particular case, and be a bit more clever if this one group of experts fails me next time.” Such a policy can easily overshoot, even. This has been going on for literally thousands of years now.

People are used to high-precision statements given by statistics (the income in 2016 was $24.4 Million), and are used to low-precision statements given by human intuitions (From my 10 minute analysis, I think our organization will do very well next year). But there’s a really weird cultural norm against high-precision, intuitive statements. (From my 10 minute analysis, I think this company will make $28.5 Million in 2027).

Perhaps in part because of this norm, I think that there are a whole lot of gains to be made in this latter cluster. It’s not trivial to do this well, but it’s possible, and the potential value is really high.

I find SEM models to be incredibly practical. They might often over-reach a bit, but at least they present a great deal of precise information about a certain belief in a readable format.

I really wish there would be more attempts at making more diagrams like these in cases where there isn't statistical data. For examples, to explain phenomena like:

• What caused the fall of Rome?
• Why has scientific progress fallen over time in the US?
• Why did person X get upset, after event Y?
• Why did I make that last Facebook post?

In all of these cases, breath and depth both matter a lot. There are typically many causes and subcauses, and the chains of subcauses are often deep and interesting.

In many cases, I think I'd prefer one decent intuitively-created diagram to an entire book. I'd imagine that often the diagram would make things even more clear.

Personally, I've been trying to picture many motivations and factors in my life using imagined diagrams recently, and it's been very interesting. I intend to write up some of this as I get some examples onto paper.

There’s a big stigma now against platforms to give evaluations or ratings on individuals or organizations along various dimensions. See the rating episode of Black Mirror, or the discussion on the Chinese credit system.

I feel like this could be a bit of a missed opportunity. This sort of technology is easy to do destructively, but there are a huge number of benefits if it can be done well.

We already have credit scores, resumes (which are effectively scores), and social media metrics. All of these are really crude.

Some examples of things that could be possible:

• Romantic partners could be screened to make sure they are very unlikely to be physically abusive.
• Politicians could be much more intensely ranked on different dimensions, and their bad behaviors listed.
• People who might seem sketchy to some (perhaps because they are a bit racist), could be revealed to be good-intentioned and harmless.
• People who are likely to steal things could be restricted from entering certain public spaces. This would allow for much more high-trust environments. For example, more situations where customers are trusted to just pay the right amounts on their own.
• People can be subtly motivated to just be nicer to each other, in situations where they are unlikely to see each other again.
• Most business deals could do checks for the trustworthiness of the different actors. It really should become near-impossible to have a career of repeated scams.

These sorts of evaluation systems basically can promote the values of those in charge of them. If those in charge of them are effectively the public (as opposed to a corrupt government agency), this could wind up turning out nicely. If done well, algorithms should be able to help us transition to much higher-trust societies.

Voting systems vs. utility maximization

I’ve seen a lot of work on voting systems, and on utility maximization, but very few direct comparisons. But I think that often we can prioritize systems that favor one or the other, and clearly our research efforts are limited between the two, so it seems useful to compare.

Voting systems act very different to utility maximization. There’s a big host of literature on ideal voting rules, and it’s generally quite different to that of utility maximization.

Proposals like quadratic voting are clearly in the voting category, while Futarchy is much more in the utility maximization category. In US Democracy, some decisions are quite clearly made via voting, and others are made roughly using utility maximization (see any cost-benefit analysis, or the way that new legislation is created). Note that by utility I don’t mean welfare, but whatever it is that the population cares about.

My impression is that the main difference is that voting systems don’t need to assume any level of honesty. Utility maximization schemes need to have competent estimates of utility, but individuals are motivated to try to lie about the numbers so that the later calculations come out in their favor. So if we can’t identify any good way of obtaining the ground truth, but people can correctly estimate their own interests, then voting methods might be our only option.

If we can obtain good estimates of utility, then utility optimization methods are normally more ideal. There’s a good reason why the government rarely asks for public votes on things when it doesn’t need to. Voting methods essentially leave out a lot of valuable information, and they often require a lot of overhead.

I think the main challenges for the utility maximization side are to identify trustworthy and inexpensive techniques for eliciting utility, and to find ways of integrating voting that get the main upsides without the downsides. I assume this means something like, “we generally want as little voting as possible, but it should be used to help decide the utility maximization scheme.” For example, one way that this could be done, is that multiple agents could vote on a particular scheme of maximizing utility between them. Perhaps they all donate 10% of their resources to a particular agency that attempts to maximize utility under certain conditions.

Prediction evaluations may be best when minimally novel

Imagine a prediction pipeline is resolved with a human/judgemental evaluation. For instance, a group today starts predicting what a trusted judge 10 years from now will say for the question, "How much counterfactual GDP benefit did policy X make, from 2020-2030?"

So, there are two stages:

1. Prediction
2. Evaluation

One question for the organizer of such a system is how many resources to delegate to the prediction step vs. the evaluation step. It could be expensive to both pay for predictors and evaluators, so it's not clear how to weigh these steps against each other.

I've been suspecting that there are methods to be stingy with regards to the evaluators, and I have a better sense now why that is the case.

Imagine a model where the predictors gradually discover information I_predictors about I_total, the true ideal information needed to make this estimate. Imagine that they are well calibrated, and use the comment sections to express their information when predicting.

Later the evaluator comes by. Because they could read everything so far, they start with I_predictors. They can use this to calculate Prediction(I_predictors), although this should have already been estimated from the previous predictors (a la the best aggregate).

At this point the evaluator can choose to attempt to get more information, I_evaluation > I_predictors. However, if they do, the resulting probability distribution would be predicted by Prediction(I_predictors). Insofar as the predictors are concerned, the expected value of Prediction(I_evaluation) should be the same as that of Prediction(I_predictors), assuming that Prediction(I_predictors) is calibrated; except for the fact that it will be have more risk/randomness. Risk is generally not a desirable property. I've written about similar topics in this post [LW(p) · GW(p)].

Therefor, the predictors should generally prefer Prediction(I_predictors) to Prediction(I_evaluator), as long as everyone's predictions are properly calibrated. This difference shouldn't generally lead to a difference of predictions from them unless a complex or odd scoring rule were used.

Of course, calibration can't be taken for granted. So pragmatically, the evaluator would likely have to deal with issues of calibration.

This setup also assumed that maximally useful comments are made available to evaluator. I think predictors will generally want the evaluator to see much of their information, as it would in general support their sides.

A relaxed version of this may be that the evaluators' duty would be to get approximately all the information that the predictors had access to, but more is not necessary.

Note that this model is only interested in the impact of good evaluation on the predictions. Evaluation also would lead to "externalities"; information that would be useful in other ways as well. This information isn't included here, but I'm fine with that. I think we should generally expect predictors to be more cost-effective than evaluators at doing "prediction work" (i.e. the main reason we have separated anyway!)

TLDR
The role of evaluation could be to ensure that predictions were reasonably calibrated and that the aggregation thus did a decent job. Evaluators shouldn't don't have to outperform the aggregate, if that requires outside information from what was used in the predictions.