I would expect by their natures of how AI gets deployed that a lot of cooperation happens pretty soon. 

Let's say I want my agent to book me a doctor's appointment because I have an issue. I would expect that it's fairly soon that my AI agent is able to autonomously send out emails to book a doctor's appointment. On the other side, it makes a lot of sense for the doctor's office to have an AI that runs doctors' appointments on their side. 

In Germany, where I live, how soon the appointment is can depend on factors like how urgent the appointment is and the type of insurance the patient is using.

This is a simple case of two AIs cooperating with each other so that the doctors' appointment gets scheduled. 

Interestingly, the AI of the patient has the choice whether or not to defect with the AI of the doctor's office. The AI of the patient can lie about how critical the condition of the patient happens to be to get an appointment that's sooner.

There's no need for AI that is near human capabilities to be around for the ability of AI to negotiate with each other to be relevant. All the major companies will need training data to optimize how their AI negotiates fairly soon.

If you go to Yudkowski and say "Hey, I want to build an AGI, can you please tell me how to do so safely?", he will answer "I can't tell you how to build an AGI safely. If you are going to successfully build an AGI you are likely to kill us all. Don't build an AGI".

Yudkoswky is self-aware that he doesn't have the expertise that would be necessary to build a safe AGI. If you then search for an expert who gives you some advice about how to build your AGI and you succeed in building your AGI and kill everyone, that was a trap of your own making.

If you want a past example, David Chapman´s article about Domestic Science is great. More recently, you had a flatlining of childhood obesity rates in the second Bush administration. Afterwards, Michelle Obama started a huge program to improve childhood nutrition through various measures recommended by nutrition scientists. After those programs childhood obesity rates were rising again. 

Scientists studying nutrition do believe themselves to be experts, but seem to give advice that produces more problems than it solves. 

Reality doesn´t just reward you for having put in a lot of effort. 

Before asking for help, they made a wrong decision, which maybe solved their short term problems but shot them in the foot in the long term. This bad decision made other bad decisions more likely until they were helplessly stuck in a trap of their own making.

It's worth noting that there are domains where there are no experts. In those domains people who explore them still tie themselves up.

A class of people who thinks of themselves as experts but doesn't really have a clue is the most dangerous when it comes to trapping themselves in traps of their own making.

AGI development is the obvious example for LessWrong and other examples are better left as exercise to the reader.

The US does not have laws that forbid people who don't have a security clearance from publishing classified material. The UK is a country that has such laws but the first amendment prevents that.

I don't think that chosing jurisdiction in the hope that they will protect you is a a good strategy. If you want to host leaks from the US in China, it's possible that China's offers to surpress that information as part of a deal.

4chan has a single point of failure. If the NSA would be motivated enough to burn some of their 0-days, taking it offline wouldn't be hard. 

Taking a decentralized system with an incentive structure like ArDrive down is significantly harder.

Attacking ArDrive is likely also politically more costly as it breaks other usages of it. The people with NFT that store data on ArDrive can pay lobbyists to defend it.

Just convincing the developers is not enough. You also need the patch they created to be accepted by the network, and it's possible for the system to be forked if different network participants want different things.

Torrents are also bad for privacy everybody can see the IP addresses of all the other people who subscribe to a torrent.

For privacy onion routing is great. Tor uses that. Tor however doesn't have a data storage layer.

Veiled and the network on which Session runs use onion routing as well and have a data storage layer.

In the case of Veiled you get the nice property that the more people want to download a certain piece of content the more notes in the network store the information.

As far as creating public knowledge goes, I do think that Discord, servers and Telegram chats serve currently as social media.

When it comes to meta data and plain text extraction it's worth noting that meta data can both be used to verify documents and to expose whistleblowers. If a journalist can verify authenticity of emails because they have access to the meta data that's useful.

• guidelines for latest hard-to-censor social media
  • to publish torrent link, maybe raw docs, and social media discussions
  • guidelines must be country-wise and include legal considerations. always use a social media of a country different from the country where leak happened.

The Session messenger is probably better than country-specific social media.

country-wise torrents (not sure if this is needed)

• torrents proposed above are illegal in all countries. instead we can legally circulate country A's secrets via torrent within country B and legally circulate country B's secrets via torrent within country A. only getting the info past the border is illegal, for that again need securedrop or hard disk dead drop if any country or org seals their geographic borders.

The US does have the first amendment. That currently means that all the relevant information about AI labs is legal to share. It's possible to have a legal regime where sharing model weights of AI gets legally restricted but for the sake of AI safety I don't think we want Open AI researchers to leak model weights of powerful models.

The main information that's currently forbidden from being shared legally in the US is child pornography, but  whistleblowing is not about intentionally sharing child pornography. When it comes to child pornography, the right thought isn't "How can we host it through a jurisdiction where it's legal", but to try to avoid sharing it.

While sharing all the bitcoin blocks involves sharing child pornography, nobody went after bitcoin minors for child pornography. People who develop cryptography who don't intend to share child pornography generally has not been prosecuted. 

Torrents are not a good technology for censorship-resistant hosting. Technology like veilid, where a data set that gets queried by a lot of people automatically gets distributed over more of the network is better because it prevents the people who hosts the torrents from being DDoSed. 

If you just want to host plaintext, blockchain technology like ArDrive also exists. You need to pay ~12$ per GB but if you do so, you get permanent storage that's nearly impossible to censor. 

I don't think it's a spectrum. A spectrum is something one-dimensional. The problem with your distinction is that someone might think that there are safe from problems arising from power seeking (in a more broad definition) if they prevent the AI from doing things that they don't desire. 

There are probably three variables that matter:

1. How much agency has the human in the interaction.
2. How much agency has the AI agent in the interaction.
3. Does the AI cooperate or defect in the game theoretic sense.

If you have a low agency CEO and a high agency very smart middle manager that always cooperates, that middle manager can still acquire more and more power over the organization.

Plenty of things I desire happen without me intending for them to happen.

In general, the reference class is misalignment of agents and AIs aren't the only agents. We can look at how the terms works in a corporation.

There are certain powers that a CEO of a big corporation intentionally delegates to a mid-level manager. I think there's plenty that a CEO appreciates his mid-level manager to do that the CEO does not explicitly task the mid-level manager to do. The CEO likely appreciates if the mid-level manager autonomously takes charge of solving problems without bothering the CEO about it. 

On the other hand, there are also ways where the mid-level manager does company politics and engineers a situation so that the CEO giving the mid-level manager certain powers that the CEO doesn't desire to give the mid-level manager. The CEO feels forced to do so because of how the company politics play out, so the CEO does intentionally give the powers out to the mid-level manager. 

What's the difference between being given affordances and getting power? If you are given more affordances you have more power. However seeking is about doing things that increase the affordances you have. 

I searched a bit more and it seems they don't have personal relationships with other members of the same species the way mammals and birds can.

Personal relationships seem to something that needs intelligence and that birds and mammals evolved separately.

The 2019 update add many codes that orthodox Western medicine disagrees with. 

If someone wants Chinese medicine codes they got it in the update. Ayurveda got codes. Activists for Chronic Lyme got their codes as well.  

The philosophy of that update seemed to be "If there anything that doctors want to diagnose, it should get a code so that it can go into standardized electronic health records."

The Humboldt squid is an octupus that can coordinate to hunt together. 

It's important to be clear what we mean with intelligence in this context. The last common ancestors of birds and mammals lived 325 million years ago or roughly 200 million years after the Cambrian revolution. It was likely a lizard-like creature of maybe 20-30cm length.

It was likely intelligent enough to coordinate the movement of its four legs and make some decisions about what to eat. It had eyes. It probably had a brain with neurons. Maybe capable of a freeze response to avoid getting eaten by a predator but no fight/flight. It was likely much dumber than today's mammals and birds. 

Given the same building blocks of neurons and axions that you connect together to form a brain birds and mammals seem to have found different strategies of how to create higher intelligence.

Octopi who diverged before the Cambrian evolution would likely make a much better argument of independent development of intelligence.

The key is still to distinguish good from bad ideas. 

In the linked post, you essentially make the argument that "Whole brain emulation artificial intelligence is safer than LLM-based artificial superintelligence". That's a claim that might be true or not true. On aspect of spending more time with that idea would be to think more critically about whether that's true.

However, even if it would be true, it wouldn't help in a scenario where we already have LLM-based artificial superintelligence.

In 2019, the WHO's added "nociplastic pain" (another word for neuroplastic pain) as an official new category of pain, alongside the long established nociceptic and neuropathic pain categories

It's worth noting that in 2019 the WHO also added various diagnosis from Chinese traditional medicine. The process that the WHO uses is not about finding truth but to provide codes that allow healthcare providers to talk with each other and store diagnoses. 

I did agree with the framing of the problem at the beginning but it's unclear to me why the conclusion is something besides "focus on how we can produce more reliable sources of information".

If you take an issue like 'people don't believe in CDC guidance' it's possible to reform the CDC in a way that forces it to give explicit reasoning that's backed for all their recommendations.

Instead of fighting misinformation it would be possible to focus on improving the processes inside institutions to make those more reliable.

You made statements that you know why the field is stuck. That both contains an assertion of the field being stuck and you knowing the explanation is due to diminishing returns. I made no claim that I know the answer to either of those questions.

#1 There is not a significant group of young capable researchers saying we are taking things in the wrong direction, but a smaller number of older ones. 

If you think that theoretical physics is going in the wrong direction and are unlikely going to get to research in a way you think will make progress, there are good reasons for not being not making your PHD in theoretical physics. The strongest disbelievers filter themselves out.

But even among the people who are actually in the field, I don't see a good reason why you would publically see signs.

You might ask PHD students: "If you wouldn't need to seek grants and would get a lab with 5 million dollar per year, would you pursue the same research agendas as you are currently do or would you pursue research for which you wouldn't get grants in the current academic environment?"

It might be interesting to ask that question for researchers in every field and see the responses, but unfortunately I don't know of any source that asks such a question in a good way. 

I followed Hinton et all before they became famous. I saw the paradigm thing play out in front of my eyes there.

It's certainly possible for their to be fields where it can be obvious to outsiders that a particular paradigm changing approach will work, but that's not necessary for there to be superior paradigms that could be persued.

While the Trump administration campaigned on deporting very many people till now they haven't. Instead they seem to pursue a policy of taking actions that produce fear I potential immigrants and managed to drastically reduce the inflow illegal immigrants that way. 

Both mainstream media and the Trump administration like it if the policies look scary.

If you want to decide whether it makes sense to be personally scared it's important to think through your personal threat models.

There are people like Sabine Hossenfelder who think that there are experiments we could run that we currently don't run, because they don't fit into the string physics paradigm. 

If you look back through history, in most times you would not see data that shows that a specific non-hegemonic scientific paradigm is better than the hegemonic paradigm. We usually only have clear data that a different paradigm is better when the old paradigm stops being hegemonic. 

The idea that progress is stalled because everyone is hypnotized by string theory, I think is simply false, and I say that despite having studied alternative theories of physics, much much more than the typical person who knows some string theory.

Are you saying that progress in physics hasn't stalled or that string theory isn't to blame?

The paper you linked about the last big breakthrough seems to be from 1997, so roughly 28 years ago. What do you consider to be the biggest breakthrough since then?

In many cases, new paradigms care about different metrics than older paradigms. In the beginning, successful new paradigms usually don't fulfill the qualities that heterodox researchers in the field are looking to. You might want to read Thomas Khun's "The Structure of Scientific Revolutions".

There are ways you can make it easier to overturn paradigms. You can change the way research is funded. You could change the grant-making processes in a way that makes it easier for very smart young people to pursue research agendas that heterodox old people don't find interesting. 

There's the Max Planck quote of "Science advances one funeral at a time". In the last decades, old researchers got more power over which research gets conducted than back in 1950 when Planck wrote his autobiography. 

I think the problem is that you ignore the idea that science works via paradigms. Even if there's a possible paradigm besides string theory that would produce more progress, there are a lot of different things that people who aren't working on string theory could work on. Most of them won't lead anywhere.

If a new paradigm could be found that has more potential, that paradigm would have new low hanging fruit. 

However, researchers that would write papers about that low hanging fruit, might have trouble getting published in journals of the old paradigm because they are solving problems of interest to the new paradigm and not problems of interest of the old paradigm. Getting funding to work on problems of a new paradigm is also harder. 

It's worth noting that we observe other forms of simplication of language as well. English reduced the amount of inflections of verbs. The distinction between singular and plural pronouns disappeared. 

In many cases, there are diminishing returns to a given scientific paradigm. The fact that you observe a field getting diminishing returns doesn't mean that there isn't a paradigm that the field could adopt that would allow for returns to flow again. Paradigm change is about pursuing ideas that people in the old paradigm don't find promising.

Just adding more smart people who follow a hegemonic paradigm doesn't automatically get you paradigm shifts that unlock new returns. If string theory stiffles progress, it would look from the inside like there are diminishing returns to theoretical physics.

There seems to be papers that show that if you naively train on chain of thought, you train models not to verbalize potentially problematic reasoning in their chain of thought. I however don't see discussion about how to train chain of thought models to better verbalize their reasoning. 

If you can easily train a model to hide it's reasoning you should also be able to train models the other way around to be more explicit about their reasoning. 

One approach I imagine is to take a query like diagnosing medical issues and replace key words that change the output and then see how well the chain of thought reflects that change. If the chain of thought tells you something about the change in outcome, you reinforce the chain of thought. If the chain of thought doesn't reflect the outcome well, you punish the chain of thought.

All it takes is trusting that people believe what they say over and over for decades across all of society, and getting all your evidence about reality filtered through those same people.

I seems to me like you also need to have no desire to figure things out on your own. A lot of rationalists have experiences of seeking truth and finding out that certain beliefs people around them hold aren't true. Rationalists who grow up in communities where many people believe in God frequently deconvert because they see enough signs that the beliefs of those people around them aren't really fitting together. 

Given that most people living in religious communities grow up believing in God just as the people around them do, it's might be very normal to think that way, but it still feels really strange to me and probably does feel strange to many other rationalists as well. 

What do you mean with 'must'? The word has to different meanings in this context and it seems bad epistemology not to distinguish them.

Have you thought about making an altered version that strips out enough of the My Little Pony-IP to be able to sell the book on Amazon KDP? (or let someone else do that for you if you don't want to do the work?) 

The existing ontology that we have around consciousness is pretty unclear. A better understanding the nature of consciousness and thus what's valuable will likely come with new ontology. 

When it comes to reasoning around statistics, robustness of judgements, causality, what it means not to Goodhart it's likely that getting better at reasoning also means to come up with new ontology.

Regardless of the details, we ought to prioritize taking all of our power plants, water purification stations, and nuclear facilities out of the world-wide-web. 

I think it's very questionable, to make major safety policy "regardless of the details". If you want to increase the safety of power plants, listening to the people who are responsible for the safety of power plants and their analysis of the details, is likely a better step instead of making these kind of decisions without understanding the details.

Orcas already seem to have language to communicate with other orcas. Before trying to teach them a new language, it would make more sense to better understand the capabilities of their existing language and maybe think about how it could be extended to communicate with them about what humans want to talk about with them.

The author seems to just assume that his proposal will lead to a world where humans have a place instead of critically trying to argue that point. 

It depends on how much Pokémon-like tasks are available. Given that a lot of capital goes into creating each Pokémon game, there aren't that many Pokémon games. I would expect the number of games that are very Pokémon-like to also be limited. 

It's quite easy to use Pokemon playing as feedback signal for becoming better at playing Pokemon. If you naively do that, the AI would learn how to solve the game but doesn't necessarily train executive function. 

A task like doing computer programming where you have to find a lot of different solutions is likely providing better feedback for RL.

Good good strategy might be to cross post post and see what reception they get on Less wrong as far as up votes go. If a post would stay in the single digits, don't cross post other posts like that. If it gets 50+ karma, people on Less wrong wants to see more like it. 

What is the chance that these octopuses (at the point of research scientist level) are actively scheming against us and would seize power if they could?

And the related question would be: Even if they are not "actively scheming" what are the chances that most of the power to make decisions about the real world gets delegated to them, organizations that don't delegate power to octopuses get outcompeted, and they start to value octopuses more than humans over time?

Left-vs-right is not the only bias that matters. Before the pandemic, I would have thought that virologists care about how viruses are transmitted. It seems, that they don't consider that to be their field. 

Given that virologists are higher status in academia than people in environmental health who actually care about how viruses are transmitted outside the lab, the COVID19 seems to have been bad. Pseudoscience around 6-feet distancing was propagated by government regulations. Even Fauci admits that there was no sound reasoning that supported the 6-feet rule.

Fauci also decided against using use money from the National Institute of Allergy and Infectious Diseases to fund studies about community masking as a public health intervention. You don't need virologists to run studies about masking, so probably that's why he didn't want to give money to it.

While Fauci was likely more to the left, that did not create the most harmful biases in the policy response that didn't want to use science to it's fullest potential to reduce transmission of COVID19 but rather wanted to give billions to the Global Virome Project. 

In another case, grid-independent rooftop solar installations are a lot more expensive than they would need to be. Building codes are made by a firefighter interest group in the US, and for firefighters it's practical if the rooftop solar cells shut of when disconnected from the grid and as a result the pushed based on flimsy evidence for regulation that means that most rooftop solar in the US doesn't work if the grid is cut off. 

The question of whether you want grid-independent rooftop solar, is not one of left-vs-right but the biases are different. 

Especially, today where many experts are very narrow in their expertise and have quite specific interests because of their expertise, thinking in terms of left-wing and right-wing is not enough.

IIUC human intelligence is not in evolutionary equilibrium; it's been increasing pretty rapidly (by the standards of biological evolution) over the course of humanity's development, right up to "recent" evolutionary history.

Why do you believe that? Do we have data that mutations that are associated with higher IQ are more prevalent today than 5,000 years ago?

If you have a mutation that gives you +10 IQ that doesn't make it hard for you to relate with your fellow tribe of hunter-gatherers. 

There´s a lot more inbreeding in hunter-gatherer tribes that results in mutations being distributed in the tribe than there is in modern Western society. 

The key question is whether you get more IQ if you add IQ-increasing mutations from different tribes together, I don't think that it being disadvantageous to have +30 IQ more than fellow tribe members would be a reason why IQ-increasing mutations that are additive should not exist.

Consumer Reports is a nonprofit. They run experiments and whatnot to determine, for example, the optimal toothpaste for children. 

The link says nothing about them having run any experiments in their quest to make toothpaste recommendations and they recommend toothpaste based on arguments that aren't about their own experimental results. Claiming that a process that doesn't test how effective toothpaste is at creating beneficial clinical outcomes like having lower caries as determining "optimal toothpaste", sounds strange to me. 

Their process might be better than just using marketing processes, but it's very far from actually running a clinical trial that looks at which toothpaste is optimal for dental outcomes. They don't even seem to understand enough of the domain to understand that Xylitol is an active ingredient in toothpaste.

They do not get paid by the companies they test the products of.

That claim also seems wrong given that they say "When you shop through retailer links on our site, we may earn affiliate commissions. 

If you want to approach toothpaste rationally, the way you do it build a mental model of the evidence landscape.

A German legal advice Youtube channel talks about scams via fake voice getting more common and being used against normal people. One of the examples seems to be needing money to make bail. 

If you haven't talked about with your parents or grandparents about these kinds of scams, now is the time to find protocols to deal with them. 

Do you have hope that someone else does the required research, so that it's ready by the time the first superbabies are created?

If not, do you think it's okay to create superintelligent babies without it?

A lot of curves are sigmoid. Let's say there's a neurotransmitter where having to double the amount of it increases IQ but there are no gains from having four times as much of the neurotransmitter.

There are two genes that both double the production of the neurotransmitter. If both genes individually are +5 IQ both genes together don't give you +10 IQ.

It would even be possible that overproduction of that neurotransmitter produces problems at 4x the normal rate but not a 2x the normal rate.

When it comes to chicken and their size I would expect the relationship of there being two genes that both increase muscle production to be happen more frequently than for intelligence.

If you have genetic mutations that increase intelligence without cost evolution works to spread them through the whole population. If you have wild chicken for whom a given size is optimal there's no strong selection pressure to get rid of all the +x or -y size genes from the gene pool.

One way to look into this would be to see how many of the genes that increase physical size more when there are two copies of the gene compared to how many genes increase intelligence more when there are two copies of it.

And how many genes increase size/intelligence with one copy but decrease it with two copies.

You are failing to distinguish the claim "It's possible to read faster" with "There's is single easy trick of removing subvocalization that will make you read faster without."

A big aspect of why the article from Scott is noteworthy is because Scott used to make money with promoting speed reading (it was one of his top blog posts) and later changed his mind. He's not someone who started out skeptic.

Today, we do have the ability to speed up podcast we hear by 4X and it's people can still process the audio. While following a podcast along at 4x isn't easy, it's possible. 

Googling finds me: "The provided book at the 2021 championship consisted of total 15,823 words which Emma Alam read in 20 minutes and 4 seconds at 789 words per minute with the extraordinary comprehension of 97%"

Given the way the human eye works 20000wpm seems implausible. That number suggests that people can read without being able to use the eye to focus to see individual letters. 

When it comes to recording race, it's important to understand design criteria.

Allowing more possible choices is not always better in clinical trials. The more data you have, the more degrees of freedom you have in the data and the more spurious correlations you are going to pick up.

If you add a new category that only appears in one or two people in your trial, you pay the cost but you are not going to learn anything from it.

This is one of the few things we were taught at university in our statistics for bioinformatics course (which was run by someone who looks over the statistics of clinical trials) that aren't often made in discussion of statistics I see online. 

Minorities like Black people and Native American have lower trust in the medical system because the system historically treated them poorly.

Creating rules for representation if Black and Native Americans in clinical trials has the purpose of winning the trust of those communities.

As far as I understand, FDA regulators do read free text fields. While free text feels don't allow for quantitative analysis the allow for qualitative analysis and new hypothesis generation.

Currently, we have smart people who are using their intelligence mainly to push capabilities. If we want to grow superbabies into humans that aren't just using their intelligence to push capabilities, it would be worth looking at which kind of personality traits might select for actually working on alignment in a productive fashion.

This might be about selecting genes that don't correlate with psychopathy but there's a potential that we can do much better than just not raising psychopaths. If you want to this project for the sake of AI safety, it would be crucial to look into what kind of personality that needs and what kind of genes are associated with that personality.

The key problem here are your epistemics. My reading speed doesn't really matter for this discussion. You are dealing with a topic that has an existing discourse and instead of familarizing yourself with that discourse, you are reasoning with anecdotal data. 

Scott H Young for example writes:

Here the evidence is clear: subvocalization is necessary to read well. Even expert speed readers do it, they just do it a bit faster than untrained people do. We can check this because that inner voice sends faint communication signals to the vocal cords, as a residue of your internal monolog, and those signals can be measured objectively.

It might be that Scott is wrong here, I don't think the kind of observation that you use to support your belief that subvocalization is bad are strong enough to doubt Scott here.  

The claim that pronouncing things is a bad reading habit that's frequently made but I have never seen good evidence for it. Why do you believe it?

Family-run businesses, often cited as collateral damage in such discussions, could be granted transitional arrangements to ensure they remain viable while still upholding the principle that wealth should not be inherited unearned.

If you want to argue that, actually say how the arrangement should look like. 

A tax structure would need to close these gaps, treating all lifetime wealth transfers as taxable events

If you say 100% death tax and want to treat all lifetime wealth transfers as taxable events, do you mean nobody is allowed to give any wealth away? 

No birthday gifts at all? No donations to charity where someone transfers wealth to charity?

I think 20st century big bureaucracy is quite different from the way 18st century governance. The Foreign Office of the United Kingdom managed work with 175 employees at the height of the British Empire in 1914.