Suddenly more poignant: 

Out of curiosity, were any patterns discovered during this process? For example, were the writing styles similar among the ones the AI could convert into successful music, or did ones by the same author churn out songs with specific similarities, or what have you?

This is great, bookmarked for future warm and fuzzies. I've just had my second, a son, on February 8th. My first, a daughter, is six next week.

Let it be known to all and sundry that kids are fantastic and fatherhood is wondrous. It is much work and a high cost in money and sleep, in exchange for which you are endowed with glorious purpose and wireheaded to the future.

Also there is the love. Strongly recommended.

Some relevant details for the American government case:

Popular election of the Senate began in 1913. Before that each state’s Senators were elected by the state legislature. This means factional dominance of the Senate was screened off, and actually determined the state level.

This is because the dominant group analysis at the time the constitution was written was people, state government, and federal government, and the conversation was about how to prevent a single group from gaining control over all of government.

The slave vs free grouping played out at the state level. Continuing the group analysis, state level politics is viewed as having been largely between urban and rural interests. In the South the rural interests - plantation owners - usually won, and in the North, urban industrialists usually did. The canonical example of the legacy of this divide is that state capitols are rarely the largest city in the state. The capitols - and therefore the state capital - are normally a much smaller city.

This brings us down to the local level, which in the US is where most of the competition between traditional divisions like race, religion and ethnicity played out.

I think at least in the American case, I model the key development as the creation of more and different groupings through federalism, rather than a veto mechanism for traditional groups.

On the other hand, separately I have heard the idea that traditional groups were weaker in the US than in Europe because of the disruption caused by the US’ colonial structure and immigration, so I could be mislead by these peculiar circumstances. I would need a much better understanding of the democratization of other European countries and preferably some outside of Europe. Unfortunately the data is pretty sparse there, as these democracies are usually very young and don’t have many cycles of competition to compare.

A few years after the fact: I suggested Airborne Contagion and Air Hygiene for Stripe’s (reprint program)[https://twitter.com/stripepress/status/1752364706436673620].

One measure of status is how far outside the field of accomplishment it extends. Using American public education as the standard, Leibniz is only known for calculus.

there is not any action that any living organism, much less humans, take without a specific goal

Ah, here is the crux for me. Consider these cases:

• Compulsive behavior: it is relatively common for people to take actions without understanding why, and for people with OCD this even extends to actions that contradict their specific goals.
• Rationalizing: virtually all people actively lie to themselves about what their goals are when they take an action, especially in response to prodding about the details of those goals after the fact.
• Internal Family Systems and related therapies: the claim on which these treatments rest is that every person intrinsically has multiple conflicting goals of which they are generally unaware, and the learning how to mediate them explicitly is supposed help.
• The hard problem of consciousness: similar to the above, one of the proposed explanations for consciousness is that it serves as a mechanism for mediating competing biological goals.

These are situations where either the goal is not known, or it is fictionalized, or it is contested (between goals that are also not known). Even in the case of everyday re-actions, how would the specific goal be defined?

I can clearly see an argument along the lines of evolutionary forces providing us with an array of specific goals for almost every situation, even when we are not aware of them or they are hidden from us through things like self-deception. That may be true, but even given that it is true I come to the question of usefulness. Consider things like food:

• I claim most of the time we eat, because we eat. As a goal it is circular.
• We might eat to relieve our stomach growling, or to be polite to our host, and these are specific goals, but these are the minority cases.

Or sex:

• Also circular, the goal is usually sex qua sex.
• Speaking for myself, even when I had a specific goal of having children (making explicit the evolutionary goal!), what was really happening under the hood is I was having sex qua sex and just very excited about the obvious consequences.

It doesn't feel to me like thinking of these actions in terms of manipulation adds anything to them as a matter of description or analysis. Therefore when talking about social things I prefer to use the word manipulation for things that are strategic (by which I mean we have an explicit goal and we understand the relationship between our actions and that goal) and unaligned (which I mean in the same sense you described in your earlier comment, the other person or group would not have wanted the outcome).

Turning back to the post, I have a different lens for how to view How To Win Friends and Influence People. I suggest that these are habits of thought and action that work in favor of coordination with other people; I say it works the same way rationality works in favor of being persuaded by reality. 

I trouble to note that this is not true in general of stuff about persuasion/influence/etc. A lot of materials on the subject do outright advocate manipulation even as I use the term. But I claim that Carnegie wrote a better sort of book, that implies pursuing a kind of pro-sociality in the same way we pursue rationality. I make an analogy: manipulators are to people who practice the skills in the book as Vulcan logicians are to us, here.

A sports analogy is Moneyball.

The counterfactual impact of a researcher is analogous to the insight that professional baseball players are largely interchangeable because they are all already selected from the extreme tail of baseball playing ability, which is to say the counterfactual impact of a given player added to the team is also low.

Of course in Moneyball they used this to get good-enough talent within budget, which is not the same as the researcher case.  All of fantasy sports is exactly a giant counterfactual exercise; I wonder how far we could get with 'fantasy labs' or something.

I agree that processor clock speeds are not what we should measure when comparing the speed of human and AI thoughts. That being said, I have a proposal for the significance the fact that the smallest operation for a CPU/GPU is much faster than the smallest operation for the brain.

The crux of my belief is that having faster fundamental operations means you can get to the same goal using a worse algorithm in the same amount of wall-clock time. That is to say, if the difference between the CPU and neuron is ~10x, then the CPU can achieve human performance using an algorithm with 10x as many steps as the algorithm that humans actually use in the same clock period.

If we view the algorithms with more steps than human ones as sub-human because they are less computationally efficient, and view a completion of the steps of an algorithm such that it generates an output as a thought, this implies that the AI can get achieve superhuman performance using sub-human thoughts.

A mechanical analogy: instead of the steps in an algorithm consider the number of parts in a machine for travel. By this metric a bicycle is better than a motorcycle; yet I expect the motorcycle is going to be much faster even when it is built with really shitty parts. Alas, only the bicycle is human-powered.

It isn't quoted in the above selection of text, but I think this quote from same chapter addresses your concern:

“I instantly saw something I admired no end. So while he was weighing my envelope, I remarked with enthusiasm: "I certainly wish I had your head of hair." He looked up, half-startled, his face beaming with smiles. "Well, it isn't as good as it used to be," he said modestly. I assured him that although it might have lost some of its pristine glory, nevertheless it was still magnificent. He was immensely pleased. We carried on a pleasant little conversation and the last thing he said to me was: "Many people have admired my hair." I'll bet that person went out to lunch that day walking on air. I'll bet he went home that night and told his wife about it. I'll bet he looked in the mirror and said: "It is a beautiful head of hair." I told this story once in public and a man asked me afterwards: "'What did you want to get out of him?" What was I trying to get out of him!!! What was I trying to get out of him!!! If we are so contemptibly selfish that we can't radiate a little happiness and pass on a bit of honest appreciation without trying to get something out of the other person in return - if our souls are no bigger than sour crab apples, we shall meet with the failure we so richly deserve.”

Out of curiosity, what makes this chapter seem Dark-Artsy to you?

So the smarter one made rapid progress in novel (to them) environments, then revealed they were unaligned, and then the first round of well established alignment strategies caused them to employ deceptive alignment strategies, you say.

Hmmmm.

I don't see this distinction as mattering much: how many ASI paths are there which somehow never go through human-level AGI? On the flip side, every human-level AGI is an ASI risk.

I would perhaps urge Tyler Cowen to consider raising certain other theories of sudden leaps in status, then? To actually reason out what would be the consequences of such technological advancements, to ask what happens?

At a guess, people resist doing this because predictions about technology are already very difficult, and doing lots of them at once would be very very difficult.

But would it be possible to treat increasing AI capabilities as an increase in model or Knightian uncertainty? It feels like questions of the form "what happens to investment if all industries become uncertain at once? If uncertainty increases randomly across industries? If uncertainty increases according to some distribution across industries?" should be definitely answerable. My gut says the obvious answer is that investment shifts from the most uncertain industries into AI, but how much, how fast, and at what thresholds are all things we want to predict.

I'm inclined to agree with your skepticism. Lately I attribute the low value of the information to the fact that the organization is the one that generates it in the first place. In practical terms the performance of the project, campaign, etc. will still be driven by the internal incentives for doing the work, and it is not remotely incompatible for bad incentives to go unchanged leading to consistently failing projects that are correctly predicted to consistently fail. In process terms, it's a bit like what's happening with AI art when it consumes too much AI art in training.

The way info from the non-numerate gets incorporated into financial markets today is that more sophisticated people & firms scrape social media or look at statistics (like generated by consumer activity). markets do not need to be fully accessible for markets to be accurate.

I agree with this in general, but it doesn't seem true for the specific use-case motivating the post. The problem I am thinking about here is how to use a prediction market inside an organization. In this case we cannot rely on anyone who could get the information to put it into the market because the public does not participate - we either get the specific person who actually knows to participate, or the market lacks the information.

I expect this to run into all the usual problems of getting people at work to adopt a toolchain unrelated to their work. These projects normally fail; it looks like it needs to be basically zero effort to bet your information for it to work, which is heroically difficult.

I really want to read the takedown of Helion.

I like the reasoning on the front, but I disagree. The reason I don't think it holds is because the Western Front as we understand it is what happened after the British Expeditionary Force managed to disrupt the German offensive into France, and the defenses that were deployed were based on the field conditions as they existed.

What I am proposing is that initial invasion go directly into the teeth of the untested defenses which were built for the imagined future war (which was over a period of 40 years or so before actual war broke out). I reason these defenses contained all of the mistaken assumptions which the field armies made and learned from in the opening months of the war in our history, but built-in and having no time or flexibility to correct in the face of a general invasion. Even if Britain eventually enters the war, I strongly expect there would be no surprise attack by the expeditionary force during Germany's initial invasion, and so predict the Germans take Paris.

That being said, my reasoning does work in reverse and so supports your proposed plan: if we are able to persuade Germany of the historically proven defenses and update them about the true logistical burden, they absolutely could greet the French with a Western Front-grade of defenses on their side of the border. This provides more than enough time to subjugate Russia before mobilization, or perhaps drive them to surrender outright with confirmation that their chief ally is useless. The less aggressive option with France makes the British and US entries into the war even less likely, I'd wager.

Frankly, conquering France isn't even a real win condition, it was just what I expected because that's where the invasion went historically. This makes the whole affair look simpler, where Germany and Austria-Hungary are able to prosecute a war on just the Russian and Balkan fronts, it stops being a world war and reduces to a large European war, and they get to exploit the territorial gains going forward.

My idea is a smaller intervention, but I think I like yours better!

Indeed you might - in fact I suggested attacking through the French border directly in the other question where we aid Germany/Austria rather than try to prevent the war.

The idea of defending against France is an interesting one - the invasion plans called for knocking out France first and Russia second based on the speed with which they expected each country to mobilize, and Russia is much slower to conquer just based on how far everyone has to walk. Do you estimate choosing to face an invasion from France would be worth whatever they gain from Russia, in the thinking of German command?

I genuinely don't know anything about Germany's plans for Russia post invasion in the WW1 case, so I cannot tell.

Under these conditions yes, through the mechanism of persuading German High Command to invade through the French border directly rather than going through Belgium. Without the Belgian invasion, Britain does not enter the war (or at least not so soon); without Britain in the war Germany likely does not choose unrestricted submarine warfare in the Atlantic; without unrestricted submarine warfare the US cannot be induced to enter the war on the side of the French.

As to why the direct invasion would work, we have the evidence from clashes in the field that the German armies were in general superior to the French ones, including those with defensive positions, and field experience also showed that the innovations which went into the new defenses (and the war generally) were poorly understood and inefficiently used (I have in mind here particularly the habit of radically overshooting targets and extreme underestimates of the supply requirements to sustain fire).

My extremely rough guess is that the fortifications along the border add a few days to a week of delay, with the rest of the German strategy and timetable going according to plan.

My best path for a yes is through the mechanism of Great Britain being very explicit with Germany about their intent to abide by the 1839 Treaty of London.

For context, this is the one where the signatories promise to declare war on whoever invades Belgium, and was Britain's entry point into the war. There were at least some high ranking military officers who believed that had Britain said specifically that they would go to war if Belgium were invaded, Germany would have chosen not to invade.

Power seeking mostly succeeds by the other agents not realizing what is going on, so it either takes them by surprise or they don’t even notice it happened until the power is exerted.

Yet power seeking is a symmetric behavior, and power is scarce. The defense is to compete for power against the other agent, and try to eliminate them if possible.

I agree with this, and I am insatiably curious about what was behind their decisions about how to handle it.

But my initial reaction based on what we have seen is that it wouldn’t have worked, because Sam Altman comes to the meeting with a pre-rallied employee base and the backing of Microsoft. Since Ilya reversed on the employee revolt, I doubt he would have gone along with the plan when presented a split of OpenAI up front.

I agree in the main, and I think it is worth emphasizing that power-seeking is a skillset, which is orthogonal to values; we should put it in the Dark Arts pile, and anyone involved in running an org should learn it at least enough to defend against it.

I think the central confusion here is: why, in the face of someone explicitly trying to take over the board, would the rest of the board just keep that person around?

None of the things you suggested have any bearing whatsoever on whether Sam Altman would continue to try and take over the board. If he has no board position but is still the CEO, he can still do whatever he wants with the company, and also try to take over the board. If he is removed as CEO but remains on the board, he will still try to take over the board. Packing the board has no bearing on the things Sam can do to expand his influence there, it just means it takes longer.

The problem the board had to solve was not the immediacy of Sam taking over, but the inevitability of it. The gambit with removing Helen Toner failed, but other gambits would follow. Also notice that the 4-2 split is not a fixed faction: Ilya switched sides as soon as the employees revolted putting us at 3-3, and it appears Adam D’Angelo was instrumental in the negotiation to bring Sam back. What looks at first like a 4-2 split and therefore safe was more like a 2-1-1-2 that briefly coalesced into a 4-2 split in response to Sam trying to make it a 1-1-1-3 split instead. Under those conditions, Sam would be able to do anything that wasn't so egregious it caused the other three to unify AND one of his team's seats to defect.

Well gang, it looks like we have come to the part where we are struggling directly over the destiny of humanity. In addition to persuasion and incentives, we'll have to account for the explicit fights over control of the orgs.

Silver lining: it means we have critical mass for enough of humanity and enough wealth in play to die with our boots on, at least!

I agree with all of this in principal, but I am hung up on the fact that it is so opaque. Up until now the board have determinedly remained opaque.

If corporate seppuku is on the table, why not be transparent? How does being opaque serve the mission?

While I feel very unsure of a lot of my picks, I chose to interpret this as declaring a prior before the new evidence we expect comes out. I also took the unsure button route on several where my uncertainty is almost total.

I'm not sure this makes sense, except as a psychological trick.

This is indeed about half the pitch in my view. The strategy comes in two parts as I understand it: one, the psychological trick of triggering a fear of successful lawsuits; two, slightly increasing the likelihood that if the risk becomes reality they will have to pay significant penalties.

Ah, oops! My expectations are reversed for Shear; him I strongly expect to be as exact as humanly possible.

With that update, I'm inclined to agree with your hypothesis.

I would normally agree with this, except it does not seem to me like the board is particularly deliberate about their communication so far. If they are conscientious enough about their communication to craft it down to the word, why did they handle the whole affair in the way they seem to have so far?

I feel like a group of people who did not see fit to provide context or justifications to either their employees or largest shareholder when changing company leadership and board composition probably also wouldn't weigh each word carefully when explaining the situation to a total outsider.

We still benefit from a very close reading, mind you; I just believe there's a lot more wiggle room here than we would normally expect from corporate boards operating with legal advice based on the other information we have.

I don't understand the mechanism of the double-cross here. How would they get the pro-EA and safety side to trigger the crisis? And why would the safety/EA people, who are trying to make everything more predictable and controllable, be the ones who are purged from influence?

I am also confused. It would make me happy if we got some relevant information about this in the coming days.

If there is an email chain where all the engineers are speculating wildly about what could go wrong, then that posses a legal risk to the company, if and only if, they are later being sued because one of those wild speculations was actually correct.

Close - the risk being managed is one of total costs to go through the process, rather than their outcomes per se. So the risk to the company is increased if any of the wild speculations happens to be consistent with any future lawsuit, whether correct or spurious. How I think legal departments model this is that the length of the lawsuit determines the costs, and lawsuit length increases with the amount of purported evidence (since anything the other side says still needs to be argued against or countered, even if it is not ultimately a factor in any judgment).

I agree on the maximize effectiveness criterion, and furthermore I suspect that this could be a source of strategic advantage for companies to implement. I suspect this because I have seen a lot of commentary about how companies are much more risk-averse about lawsuits than is warranted as measured in settlements and judgments. I haven't validated these comments, but I do observe being utterly cavalier about legal risk doesn't seem to hurt tech companies near as much as you would expect given how other industries treat it.

The owner of a firm investing in R&D doesn’t account for all the benefits their technology might bring to non-paying consumers and firms, but they do care about the benefits that R&D will bring to the firm long into the future, even after their death. One part of this is that owners don’t face term limits that incentivize pump-and-dump attempts to garner voter support.

This does not match my expectations, even if it agrees with how I would feel were I the owner.

For example, the top ten Nasdaq companies spent ~$222B between them on R&D, which is almost half of the $463B spend from the private sector.

But I notice these are publicly traded companies, where the owner is in fact the shareholders. The average holding period for stocks is less than a year; if memory serves in 2020 it was less than 6 months. The average shareholder definitely does not care about the value of R&D to the firm long after their deaths, or I suspect any time at all after they sell the stock.

I notice that Amazon and Tesla are both on the list, and maybe by owner we really mean founder; I could easily see Bezos and Musk feeling that way about R&D at their companies. But most of R&D spend from the private sector is not from founder-run companies; it is from professionally managed companies with hired CEOs and executives.

On that note, I feel like I have seen plenty of cases where executives and CEOs do precisely the pump-and-dump phenomenon on the project side. Maybe executives have longer tenures than legislators, so it makes more sense for them to take a longer view? Yet when I check, CEOs have on average shorter tenure than Congress does: the CEO has 7.2 years; Representatives and Senators have 8.9 and 11 years respectively. I also note that the CEO has much more power over the budget than legislators do; the legislator at least requires the agreement or indifference of a majority of the legislature to get R&D stuff added, whereas the CEO can very often decide budget priorities unilaterally.

That being said, this doesn't seem to change the overall thrust of the post. My suspicion lands on the 2% captured value number as being misleading; for the individual company, 2% of a huge number can easily be more than 98% of a much smaller number. I'm also curious about how that number 2% is built, so my next step here is to check out the Nordhaus paper more deeply.

You stochastic parrot!

This, but unironically. What's the magic thing which keeps a human from qualifying as a stochastic parrot? And won't that magic thing be resolved when all the LLMs deploy with vector databases to resolve that pesky emphasis on symbols?

Because they have never shown an example in any of their takedowns of GTP4 that I have not also heard a human say.

I register a guess this is to keep the content of lesswrong from being scraped for LLMs and similar purposes.

That is indeed the tool I use for the purpose! Strongly recommended. They also did a public announcement here.

Because it makes it look like they're trying to conceal evidence, which is much worse for them than simply maybe being negligent.

The part that confuses me about this is twofold: one, none of the communication policies I have worked under went as far as to say something like "don't talk about risks vial email because we don't want it to be discoverable in court" which is approximately what would be needed to establish something like adverse inference; two, any documented communication policy is discoverable by itself, so I expect that it will wind up in evidence regardless.

Returning to the conversation with the CEO example, if the communication policies were like those I have worked under, I expect it would go more like this:

• The witness says they told the CEO about the risks in a conversation on such and such a date.
• Defense counsel asks where they documented this concern.
• Witness says they're not supposed to do that.
• Defense counsel introduces communication policy as exhibit A, which contains what is surely a mildly kafkaesque procedure for raising risks of that type.
• Defense counsel asks why, if the risk was so great and the witness so certain, they were unwilling to follow the clearly established procedure under section 34.B.II.c of the communication policy.

After that the defense and prosecution/plaintiff can wrangle with more witnesses showing a pattern of discouraging communication about risk (undocumented of course) vs previous risks communicated according to policies and how they were properly dealt with etc, but this is hardly a slam-dunk in either direction. As you mentioned in your personal experience, proving that you are right in court costs money, and I strongly expect that this mechanism favors the defense as the actual risks increase because they have so much more to lose in the event of an actual judgement and the defense can get by with a strategy of maintaining uncertainty, unlike the plaintiff or prosecution who has to actually prove stuff.

Linking this back to the OP, the strategy of sending an email to make the risk discoverable as suggested is reliant on converting the actual risks to a legal risk because the legal risk is a bigger factor in company decision making, and the goal is to make the company choose not to take the risk at all. I feel like this is a bigger version of your experience of a real but small risk: the company thought the juice just isn't worth the squeeze.

Out of curiosity, did you get any static afterward, or was it just an "oops" and done?

If I testify that I told the CEO that our widgets could explode and kill you, the opposition isn't going to be so stupid as to ask why there isn't any record of me bringing this to the CEO's attention. The first lawyer will be hardly able to contain his delight as he asks the court to mark "WidgetCo Safe Communication Guidelines" for evidence. The opposition would much rather just admit that it happened.

This feels like an extreme claim. What makes you conclude that personal testimony weighs more under the law than written documents? Why would the defense prefer to basically concede the case than have their communication policy entered into evidence?

My model of this, based on personal experience, is that the communication policies don't go much farther than trivial inconveniences. Because trivial inconveniences are so effective, this is a powerful risk reducer for them. My personal experience also suggests that the primary enforcement of the communication policy is to filter employee speculation. Mainly because I was part of a thread doing precisely that - wild speculation over email - and someone from legal popped in to admonish everyone about discoverability and to follow the communication policies if anyone had direct knowledge of a genuine problem to bring up. It seems to me even companies that always do the right thing don't want to have to litigate about email chains where a bunch of people are talking out of their backside.

This is a really good post. It is simple, coherent, and does a good job accounting for problems.

A concrete example of this kind of thinking focused on getting stuff is jacobjacob's How I buy things when Lightcone wants them fast post.

introduces concepts . . . into the framework of the legislation.

I strongly agree that the significance of the EO is that it establishes concepts for the government to work with.

From the perspective of the e/acc people, it means the beast has caught their scent.

Woo! That's two in the span of one week!

Hey, that's the first time I've seen safety emphasized as a budgetary commitment. Huzzah for another inch in the overton window!

While unusual for interpersonal stuff, practicing your speeches in every detail down to facial expressions and gestures is simply correct if you want to be good at making them.

So what's data science work like in the Israeli military?

Question: why is a set of ideas about alignment being adjacent to capabilities only a one-way relationship? More directly, why can't this mindset be used to pull alignment gains out of capability research?

Yes, and the virtue that is most important is the one that allowed Petrov to not doom the world. By contrast, the two most popular choices were about refusing to doom the world, and resisting social pressure, neither of which were features of the event.

If there was a poll in connection to Arkhipov, my answer might change.

I voted for correctly reporting your epistemic state. I claim that this is the actual virtue Petrov displayed, and that his primary virtue being "don't take actions which destroy the world" because he decided to buck the chain of command is a mistaken belief. From the Wikipedia article:

Petrov later indicated that the influences on his decision included that he had been told a US strike would be all-out, so five missiles seemed an illogical start;^[3] that the launch detection system was new and, in his view, not yet wholly trustworthy; that the message passed through 30 layers of verification too quickly;^[15] and that ground radar failed to pick up corroborating evidence, even after minutes of delay.^[16] However, in a 2013 interview, Petrov said at the time he was never sure that the alarm was erroneous. He felt that his civilian training helped him make the right decision. He said that his colleagues were all professional soldiers with purely military training and, following instructions, would have reported a missile launch if they had been on his shift.^[4]

More specifically I claim two things:

1. Stanislav Petrov actually believed that it was a false alarm.
2. Had he not believed it was a false alarm, he would have reported an attack.

There seems to often be a missing ‘wait this is super scary, right?’ mood attached to making AI everyone’s therapist. There is obvious great potential for positive outcomes. There are also some rather obvious catastrophic failure modes.

Reach out and touch faith, amirite?