LessWrong 2.0 Reader

Idk, I guess the more fundamental issue is this treats the goal as simply being assigning probabilities to statements in predicate logic, whereas his point is more about whether one can do compositional reasoning about relationships while dealing with nebulosity, and it's this latter thing that's the issue.

The situation where AI is a good tool for manipulating public opinion, and the leading AI company has a bad reputation, seems unstable. Maybe AI just needs to get a little better, and then AI-written arguments in favor of AI will win public opinion decisively? This could "lock in" our trajectory even worse than now, and could happen long before AGI.

If I believe eating meat is bad because I engage in motivated reasoning, then this is, like all forms of motivated reasoning, just an irrational belief. But if I believe eating meat is bad because I believe it creates a disproportionate amount of suffering, there is nothing irrational about that belief. So motivated reasoning can only explain some irrational beliefs. However, when something being bad means that it decreases some sort of welfare in some general way, then we don't have this problem. Now, what exactly does "welfare" etc mean? That's a question that normative ethicists try to figure out. For example via various proposed theories of utilitarianism. If philosophers are analyzing a subject matter, it's safe to assume they are analyzing some of concept. Now, what's a concept? It's a meaning of a word. Like "good" or "bad".

Sam Altman demonstrating what kind of actions you can get away with in front of everyone's eyes seems problematic.

Very much agreeing with this.

Appreciating your inquisitive question!

One way to think about it:

For OpenAI to scale more toward “AGI”, the corporation needs more data, more automatable work, more profitable uses for working machines, and more hardware to run those machines. 

If you look at how OpenAI has been increasing those four variables, you can notice that there are harms associated with each. This tends to result in increasing harms.

One obvious example:  if they increase hardware, this also increases pollution (from mining, producing, installing, and running the hardware).

Note that the above is not a claim that the harms outweigh the benefits. But if OpenAI & co continue down their current trajectory, I expect that most communities would look back and say that the harms to what they care about in their lives were not worth it.

I wrote a guide to broader AI harms meant to emotionally resonate for laypeople here.

Related: StackExchange: Why are estimates treated like deadlines?

There is even a comment perfectly illustrating that some people will interpret all information as an assurance (and insist that it is your fault that they do so), unless you explicitly tell them not to:

I will consider your estimate to be an educated projection based your experience, skills and qualification. [...] if you made an estimate, you need to stand by it. [...] if you say a time then that time becomes yours to live and die by. You said it, you own it. I recalculated my projections, my finances and my resources based on it. You should have said a range of estimates but you didn't. Estimates becomes deadlines because of your own fault.

If a Doctor gave an estimate to a paitent of 6-12 months of life remaining and the actual amount was 1 month we would investigate why the Doctor was so incorrect. If a builder estimates X tons of concrete and the requirements are double we do not blame the owner of the house. [...] I would be perfectly comfortable raising a legal counter-charge against a contractor for poor estimates.

If your estimate is off you are either fired, charged or will fail to be re-hired. When a carpenter makes an estimate of between X and Y inches he is expected to be right. I apply the same scrutiny to developers regardless of voodoo regarding statistics. If you cannot make accurate estimates you are either not very good at your job (I doubt that) or are unwilling to accept responsibility.

[If you respond to this attitude by padding your estimates] you would be out of a contract for padding estimates by a ridiculous factor margin. Not to mention a counter-charge against you or your agency for misconduct. [...] don't think for one second a good development team can hide behind the word estimate or any agile principle as some sort of sandbag against criticism. [...] It sounds like you have no confidence in your own abilities.

(I guess this person wouldn't even have to fire me for making a wrong estimate; if I worked with them I would happily quit.)

I think the problem might be that you've given this definition of heuristic:

A heuristic is a local, interpretable, and simple function (e.g., boolean/arithmetic/lookup functions) learned from the training data. There are multiple heuristics in each layer and their outputs are used in later layers.

Taking this definition seriously, it's easy to decompose a forward pass into such functions.

But you have a much more detailed idea of a heuristic in mind. You've pointed toward some properties this might have in your point (2), but haven't put it into specific words.

Some options: A single heuristic is causally dependent on <5 heuristics below and influences <5 heuristics above. The inputs and outputs of heuristics are strong information bottlenecks with a limit of 30 bits. The function of a heuristic can be understood without reference to >4 other heuristics in the same layer. A single heuristic is used in <5 different ways across the data distribution. A model is made up of <50 layers of heuristics. Large arrays of parallel heuristics often output information of the same type.

Some combination of these (or similar properties) would turn the heuristics intuition into a real hypothesis capable of making predictions. 

If you don't go into this level of detail, it's easy to trick yourself into thinking that (2) basically kinda follows from your definition of heuristics, when it really really doesn't. And that will lead you to never discover the value of the heuristics intuition, if it is true, and never reject it if it is false.

I also have suffered from this (and still do, really). I will share some lessons that I have picked along the way, followed by a couple of book recommendations.

The lessons:

• Focus: Most of the time the issue is you are trying to communicate way too many things. Now I try to contain my message to one single point. This helped me immensely.
• Message House: A framework in branding and PR, I advise you to do a quick web search on this. With my previous bullet, I "construct" my message house with these components whenever possible: (1) Anecdote, preferably a personal one. Kicking it with a very short story that is central to your idea helps capture people's attention immediately; (2) why this matters, why I am telling you about this; (3) Sizzle, or a very quotable quote. If people wanted to tweet one sentence from your "speech" this is it; (4) Data point or one last anecdote to enforce the central theme.
• Fluency: You might know what you are talking about, but if you are not fluent about it, it takes a lot of time to put stuff together, and it feels like you don't know what you are talking about from the outside. To overcome this, you can drill things down. Anecdotes or data points you occasionally use; you can create word blocks of your world view that signals where you are looking at things from; you can have identity related short blocks and exercise on them constantly to get yourself fluent on very specific little blocks, which then you can use as springboards or solid middle- or end-points in your speech.

Book recommendations:

• Smart Brevity by Jim VandeHei, Mike Allen, and Roy Schwartz
• You've Got 8 Seconds by Paul Hellman

There is a sort of opposite to assurance, where someone communicates their intention to do something (not merely a prediction that they will do it), but without creating any responsibility to follow through. This is usually done via non-verbal communication, like tone of voice, facial expression and "body language". The fact that the intention was communicated non-verbally creates plausible deniability. This happens, for example, in relationships. E.g. when asking after a date if they want to go to his/her room.

I'd agree the OpenAI product line is net positive (though not super hung up on that). Sam Altman demonstrating what kind of actions you can get away with in front of everyone's eyes seems problematic.