LessWrong 2.0 Reader

I just realized that Paul Christiano and Dario Amodei both probably have signed non-disclosure + non-disparagement contracts since they both left OpenAI.

That impacts how I'd interpret Paul's (and Dario's) claims and opinions (or the lack thereof), that relates to OpenAI or alignment proposals entangled with what OpenAI is doing. If Paul has systematically silenced himself, and a large amount of OpenPhil and SFF money has been mis-allocated because of systematically skewed beliefs that these organizations have had due to Paul's opinions or lack thereof, well. I don't think this is the case though -- I expect Paul, Dario, and Holden all seem to have converged on similar beliefs (whether they track reality or not) and have taken actions consistent with those beliefs.

Very Spicy Take

Epistemic Note: 
Many highly respected community members with substantially greater decision making experience (and Lesswrong karma) presumably disagree strongly with my conclusion.

Premise 1: 
It is becoming increasingly clear that OpenAI is not appropriately prioritizing safety over advancing capabilities research.

Premise 2:
This was the default outcome. 

Instances in history in which private companies (or any individual humans) have intentionally turned down huge profits and power are the exception, not the rule. 

Premise 3:
Without repercussions for terrible decisions, decision makers have no skin in the game. 

Conclusion:
Anyone and everyone involved with Open Phil recommending a grant of $30 million dollars be given to OpenAI in 2017 shouldn't be allowed anywhere near AI Safety decision making in the future.

To go one step further, potentially any and every major decision they have played a part in needs to be reevaluated by objective third parties. 

This must include Holden Karnofsky and Paul Christiano, both of whom were closely involved. 

To quote OpenPhil:
"OpenAI researchers Dario Amodei and Paul Christiano are both technical advisors to Open Philanthropy and live in the same house as Holden. In addition, Holden is engaged to Dario’s sister Daniela."

I also expect AIs to be constrained by social norms, laws, and societal values. But I think there's a distinction between how AIs will be constrained and how AIs will try to help humans. Although it often censors certain topics, Google still usually delivers the results the user wants, rather than serving some broader social agenda upon each query. Likewise, ChatGPT is constrained by social mores, but it's still better described as a user assistant, not as an engine for social change or as a benevolent agent that acts on behalf of humanity.

When you rephrase this to be about search engines

I think the main reason why we won't censor search to some abstract conception of "community values" is because users won't want to rent or purchase search services that are censor to such a broad target

It doesn't describe reality. Most of us consume search and recommendations that has been censored (e.g. removing porn, piracy, toxicity, racism, taboo politics) in a way that pus cultural values over our preferences or interests.

So perhaps it won't be true for AI either. At least in the near term, the line between AI and search is a blurred line, and the same pressures exist on consumers and providers.

A before and after would be even better!

note that the Brier score at the bottom is a few percentage points lower than what's shown in the chart; the probability distributions GPT outputs differ a bit between runs despite a temperature of 0

It's now possible to get mostly deterministic outputs if you set the seed parameter to an integer of your choice, the other parameters are identical, and the model hasn't been updated.

A Theory of Usable Information Under Computational Constraints

We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer. The resulting \emph{predictive V-information} encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon's mutual information and in violation of the data processing inequality, V-information can be created through computation. This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning. Additionally, we show that by incorporating computational constraints, V-information can be reliably estimated from data even in high dimensions with PAC-style guarantees. Empirically, we demonstrate predictive V-information is more effective than mutual information for structure learning and fair representation learning.

h/t Simon Pepin Lehalleur

Organizational structure is an alignment mechanism. 

While I sympathize with the stated intentions, I just can't wrap my head around the naivety.  OpenAI corporate structure was a recipe for bad corporate governance. "We are the good guys here, the structure is needed to make others align with us."- an organization where ethical people can rule as benevolent dictators is the same mistake committed socialists made when they had power.  

If it was that easy, AI alignment would be solved by creating ethical AI committed to alignment and giving it as much power as possible. 

Altruists are normal humans. Nothing changes priorities faster than large sums of money.  Any mix of ideals and profit-making must be arranged in a way that concerns don't mix. People in charge of non-profits making life-changing money if the profit-making is a success can't work. 

Bad organizational structure  puts well meaning humans like Sutskever repeteatly into position where he must choose between wast sums of money  or his ethical commitments.  

What you are missing here is that S. mutants often lives in pockets between tooth an epithelium or between teeth with direct permanent contact to epithelium. Due to the geometry of these spaces access to saliva is very poor so metabolites can enrich to levels way beyond those you suggest here.

This mechanism is also a big problem with the pH study above.

hat's fantastic to hear! I am thrilled the information was helpful for me.