LessWrong 2.0 Reader

Should AI safety people/funds focus more on boring old human problems like (especially cyber-and bio-)security instead of flashy ideas like alignment and decision theory? The possible impact of vulnerabilities will only increase in the future with all kinds of technological progress, with or without sudden AI takeoff, but they are much of what makes AGI dangerous in the first place. Security has clear benefits regardless and people already have a good idea how to do it, unlike with AGI or alignment.

If any actor with or without AGI can quickly gain lots of money and resources without alarming anyone, can take over infrastructure and weaponry, or can occupy land and create independent industrial systems and other countries cannot stop it, our destiny is already not in our hands, and it would be suicidal to think we don't need to fix these first because we expect to create an aligned AGI to save us.

If we grow complacent about the fragility of our biology and ecosystem, and continue to allow the possibility of any actor releasing pandemics and arbitrary malwares and deadly radiation etc (for example by allowing global transport without reliable pathogen removal, or using operating systems and open-source libraries that have not been formally proven to be safe), and keep thinking the universe should keep our environment safe and convenient by default, it would be naive to complain when these things happen and hope AGI would somehow preserve human lives and values without having to change our lifestyle or biology to adapt to new risks.

Yes, fixing vulnerabilities of our biology and society is hard and inconvenient and not as glamorous as creating a friendly god to do whatever you want, but we shouldn't let motivated reasoning and groupthink lead us into thinking the latter is feasible when we don't have a good idea about how to do it, just because the former requires sacrifices and investments and we'd prefer if it's not needed. After all, it's a fact that there exist small configurations of matter and information that can completely devastate our world, and just wishing it wasn't true is not going to make it go away.

But the probability? :O

"Bird Flu H5N1: Not Chaos, but Conspiracy?" By Alexander Pruss 
Two months ago, I was puzzled how bird flu, potentially capable of killing tens of millions, went rampant on American livestock farms and began infecting workers, yet no urgent measures were being taken. Even standard epidemiological threat monitoring was happening unsystematically, with months-long delays, and results weren't being made public for months afterward. What happened to the bitter lessons from the coronavirus pandemic? Why such chaos? Since then, the sense of criminal inaction has only intensified. Missouri discovered the first outbreak of human cases unrelated to farm workers, but molecular testing was neglected and infection paths remained undiscovered. 

In California, a more pathogenic variant of bird flu spread to hundreds of dairy farms, reportedly killing up to 15% of cows, with almost daily new cases of virus transmission to humans. The virus apparently came to California through cattle transportation from Idaho, despite belatedly introduced rules formally prohibiting the transport of infected cows across state lines. The problem was that infection in transported cows was checked through selective testing, and as reported, the sampling wasn't random: before government testing, farmers secretly tested cows for bird flu in private laboratories and selected only healthy ones for official testing. Here's the continuation of the translation:

A new Vanity Fair investigation shows this isn't random chaos. The USDA (U.S. Department of Agriculture) has been blocking research and data about the new infection in America's dairy herds from the start to protect the multi-billion-dollar American dairy export industry and the interests of giant national dairy processing companies. The idea was simple: most cows recover after a few weeks, and while the bird flu virus does get into milk in huge quantities, it should die during pasteurization. Therefore, the economic losses from the pathogen aren't that severe. However, if consumers in America and especially abroad raise the alarm, it could result in much greater dollar losses. USDA Secretary Thomas Vilsack knows this firsthand: before his government appointment, he worked as president of the U.S. Dairy Export Council. 

And immediately after it was finally discovered in March 2024 that dairy farms in Texas and Kansas were hit by bird flu, veterinarians and state officials began receiving calls from personal mobile phones of USDA veterinary institute workers: "we're officially forbidden to discuss this problem without permission from the very top, and unofficially we're asking you to keep quiet about it too." But what about the danger that the virus, having settled in mammals and especially humans, could recombine with our seasonal flu viruses and produce hybrid viruses that combine the infectious and pathogenic potential of human viruses with immunity to our regular antibodies inherited from their avian ancestor? 

This, generally speaking, isn't USDA's concern. This alarm was raised by the White House Office of Pandemic Preparedness and Response (OPPR), created in 2023, under the leadership of military doctor and biosecurity expert Paul Friedrichs. In early April, dairy industry representatives raised concerns that some upstart from the White House was muddying the waters. USDA's response was their new policy of official secrecy. Secretary Vilsack responded only a month later to state veterinarians' inquiries about the sudden communication breakdown, and his response was essentially a brush-off. And his ally in Texas, state agriculture commissioner Sid Miller, even hinted that if Friedrichs' people stick their noses into Texas farms, they might be met with bullets. 

A number of veterinarians who disagreed with USDA's actions soon lost their jobs, and the country fell into an atmosphere of "work-to-rule," where veterinary authorities appear to be doing their job, but as slowly as possible and with all the red tape that can be justified by regulations. Meanwhile, flu season is approaching, and encounters between bird and human flu in people infected with both viruses are inevitable in the near future.

I forgot to add that by May, a vaccine for bird flu became available for cows, but the USDA chose not to use it.

100% agreed. And you're definitely not alone. Someone once told me, when they asked me to describe the company I worked for, that "Your 5 minute pitch is much better than your 30 second pitch." The reasons are similar. And when I do try to condense and refine my language, I tend to run into the problem that most readers/listeners almost completely ignore qualifiers and caveats, however explicitly stated. I like to say that most of the meaning of a sentence is buried in the small words that are easy to ignore. It's very hard to express a web or network of meanings. That's also why people talk about "unpacking" and "close reading." 

From CS Lewis, in the Space Trilogy: 

“Of course I realise it’s all rather too vague for you to put into words,” when he took me up rather sharply, for such a patient man, by saying, “On the contrary, it is words that are vague. The reason why the thing can’t be expressed is that it’s too definite for language.”

And of course, from HPMOR ch 70:

Godric Gryffindor's autobiography had been a lot more compressed than the books Hermione was used to reading, he used one sentence to say things that should've taken thirty inches just by themselves, and then there was another sentence after that...

Why aren’t there 2^{1000} less programs with such dead code and a total length below10^90 for p_2, compared to p_1?

Yes, you are missing something.

Any DEADCODE that can be added to a 1kb program can also be added to a 2kb program. The net effect is a wash, and you will end up with a $2^{1000}$ ratio over priors

I have a similar feeling, but there are some forces in the opposite direction:

• Nvidia seems to limit how many GPUs a single competitor can acquire.
• training frontier models becomes cheaper over time. Thus, those that build competitive models some time later than the absolute frontier have to invest much less resources.

AGI companies merging within next 2-3 years inevitable?

There are currently about a dozen major AI companies racing towards AGI with many more minor AI companies. The way the technology shakes out this seems like unstable equilibrium. 

It seems by now inevitable that we will see further mergers, joint ventures - within two years there might only be two or three major players left. Scale is all-dominant. There is no magic sauce, no moat. OpenAI doesn't have algorithms that her competitors can't copy within  6-12 months. It's all leveraging compute. Whatever innovations smaller companies make can be easily stolen by tech giants. 

e.g. we might have xAI- Meta, Anthropic- DeepMind-SSI-Google, OpenAI-Microsoft-Apple. 

Actuallly, although this would be deeply unpopular in EA circles it wouldn't be all that surprising if Anthropic and OpenAI would team up. 

And - of course - a few years later we might only have two competitors: USA, China. 

I agree that there are significant uncertainties on the specific consequences of AI accelerating bio/medicine R&D, but I think even without buying into Amodei's specific speculations on life extension, you would still get wildly transformative breakthroughs and unforeseen consequences. I do agree it seems to make sense to be wary of just extrapolating past increases in life expectancy. 

Time will tell!

I think an important thing here is:

A random person gets selected for office. Maybe they need to move to the capital city, but their friends are still "back home." Once they serve their term, they will want to come back to their community most likely. So lobbying needs to be able to pay to get you out of your community, break all your bonds and all that during your short stint in power. Currently, politicians slowly come to power and their social clique is used to being lobbies and getting rich and selling out ideals.

This would cut down on corruption a lot (see also John Huang's comment https://www.lesswrong.com/posts/veebprDdTbq2Xmnyj/could-randomly-choosing-people-to-serve-as-representatives?commentId=NEtq8QtayXZY5a38J [LW(p) · GW(p)]) and would undo a lot of the damage done from politicians not having to live normal lives under the current system.