How about geology, ecology and history? It seems like you are focused on mechanisms rather than contents.

That said, I'm using "quantum mechanics" to mean "some generalization of the standard model" in many places.

I think this still has the ambiguity that I am complaining about.

As an analogy, consider the distinction between:

• Some population of rabbits that is growing over time due to reproduction
• The Fibonacci sequence as a model of the growth dynamics of this population
• A computer program computing or mathematician deriving the numbers in or properties of this sequence

The first item in this list is meant to be analogous to the quantum mechanics qua the universe, as in it is some real-world entity that one might hypothesize acts according to certain rules, but exists regardless. The second is a Platonic mathematical object that one might hypothesize matches the rules of the real-world entity. And the third are actual instantiations of this Platonic mathematical object in reality. I would maybe call these "the territory", "the hypothetical map" and "the actual map", respectively.

In practice, the actual experimental predictions of the standard model are something like probability distributions over the starting and ending momentum states of particles before and after they interact at the same place at the same time, so I don't think you can actually run a raw standard model simulation of the solar system which makes sense at all. To make my argument more explicit, I think you could run a lattice simulation of the solar system far above the Planck scale and full of classical particles (with proper masses and proper charges under the standard model) which all interact via general relativity, so at each time slice you move each particle to a new lattice site based on its classical momentum and the gravitational field in the previous time slice. Then you run the standard model at each lattice site which has more than one particle on it to destroy all of the input particles and generate a new set of particles according to the probabilistic predictions of the standard model, and the identities and momenta of the output particles according to a sample of that probability distribution will be applied in the next time slice. I might be making an obvious particle physics mistake, but modulo my own carelessness, almost all lattice sites would have nothing on them, many would have photons, some would have three quarks, fewer would have an electron on them, and some tiny, tiny fraction would have anything else. If you interpreted sets of sites containing the right number of up and down quarks as nucleons, interpreted those nucleons as atoms, used nearby electrons to recognize molecules, interpreted those molecules as objects or substances doing whatever they do in higher levels of abstraction, and sort of ignored anything else until it reached a stable state, then I think you would get a familiar world out of it if you had the utterly unobtainable computing power to do so.

Wouldn't this fail for metals, quantum computing, the double slit experiment, etc.? By switching back and forth between quantum and classical, it seems like you forbid any superpositions/entanglement/etc. on a scale larger than your classical lattice size. The standard LessWrongian approach is to just bite the bullet on the many worlds interpretation (which I have some philosophical quibbles with, but those quibbles aren't so relevant to this discussion, I think, so I'm willing to grant the many worlds interpretation if you want).

Anyway, more to the point, this clearly cannot be done with the actual map, and the hypothetical map does not actually exist, so my position is that while this may help one understand the notion that there is an rule that perfectly constrains  the world, the thought experiment does not actually work out.

Somewhat adjacently, your approach to this is reductionistic, viewing large entities as being composed of unfathomably many small entities. As part of LDSL I'm trying to wean myself off of reductionism, and instead take large entities to be more fundamental, and treat small entities as something that the large entities can be broken up into.

This is tangential to what I'm saying, but it points at something that inspired me to write this post. Eliezer Yudkowsky says things like the universe is just quarks, and people say "ah, but this one detail of the quark model is wrong/incomplete" as if it changes his argument when it doesn't. His point, so far as I understand it, is that the universe runs on a single layer somewhere, and higher-level abstractions are useful to the extent that they reflect reality. Maybe you change your theories later so that you need to replace all of his "quark" and "quantum mechanics" words with something else, but the point still stands about the relationship between higher-level abstractions and reality.

My in-depth response to the rationalist-reductionist-empiricist worldview is Linear Diffusion of Sparse Lognormals. Though there's still some parts of it I need to write. The main objection I have here is that "single layer" is not so much the true rules of reality so much as it is the subset of rules that are unobjectionable due to applying everywhere and every time. It's like the minimal conceivable set of rules.

The point of my quantum mechanics model is not to model the world, it is to model the rules of reality which the world runs on.

I'd argue the practical rules of the world are determined not just by the idealized rules, but also by the big entities within the world. The simplest example is outer space; it acts as a negentropy source and is the reason we can assume that e.g. electrons go into the lowest orbitals (whereas if e.g. outer space was full of hydrogen, it would undergo fusion, bombard us with light, and turn the earth into a plasma instead). More elaborate examples would be e.g. atmospheric oxygen, whose strong reactivity leads to a lot of chemical reactions, or even e.g. thinking of people as economic agents means that economic trade opportunities get exploited.

It's sort of conceivable that quantum mechanics describes the dynamics as a function of the big entities, but we only really have strong reasons to believe so with respect to the big entities we know about, rather than all big entities in general. (Maybe there are some entities that are sufficiently constant that they are ~impossible to observe.)

Quantum mechanics isn't computationally intractable, but making quantum mechanical systems at large scales is.

But in the context of your original post, everything you care about is large scale, and in particular the territory itself is large scale.

That is a statement about the amount of compute we have, not about quantum mechanics.

It's not a statement about quantum mechanics if you view quantum mechanics as a Platonic mathematical ideal, or if you use "quantum mechanics" to refer to the universe as it really is, but it is a statement about quantum mechanics if you view it as a collection of models that are actually used. Maybe we should have three different terms to distinguish the three?

Couldn't one say that a model is not truly a model unless it's instantiated in some cognitive/computational representation, and therefore since quantum mechanics is computationally intractable, it is actually quite far from being a complete model of the world? This would change it from being a map vs territory thing to more being a big vs precise Pareto frontier.

(Not sure if this is too tangential to what you're saying.)

This also kind of reveals why bad faith is so invalidating. If the regulatory commission can trust others to outsource its investigations, then it might be able to save resources. However, that mainly works if those others act in sufficiently good faith that they aren't a greater resource sink than investigating it directly and/or just steamrolling the others with a somewhat-flawed regulatory authority.

Neither full-contact psychoanalysis nor focusing on the object-level debate seems like a good way to proceed in the face of a regulatory commission. Instead, the regulatory commission should just spend its own resources checking what's true, and maybe ask the parties in the debate to account for their deviances from the regulatory commission's findings. Or if the regulatory commission is a sort of zombie commission that doesn't have the capacity to understand reality, each member in the conflict could do whatever rituals best manipulate the commission to their own benefit.

One thing to consider is that until you've got an end-to-end automation of basic human needs like farming, the existence of other humans remains a net benefit for you, both to maintain these needs and to incentivize others to share what they've done.

Automating this end-to-end is a major undertaking, and it's unclear whether LLMs are up to the task. If they aren't, it's possible we will return to a form of AI where classical alignment problems apply.

There might be humans who set it up in exchange for power/similar, and then it continues after they are gone (perhaps simply because it is "spaghetti code").

The presence of the regulations might also be forced by other factors, e.g. to suppress AI-powered frauds, gangsters, disinformation spreaders, etc..

Not if the regulation is sufficiently self-sustainably AI-run.

These aren't the only heavy tails, just the ones with highest potential to happen quickly. You could also have e.g. people regulating themselves to extinction.

I think this is a temporary situation because no sufficiently powerful entity has invested sufficiently much in AI-based defence. If this situation persists without any major shift in power for long enough, then it will be because the US and/or China have made an AI system to automatically suppress AI-powered gangs, and maybe also to automatically defend against AI-powered militaries. But the traditional alignment problem would to a great degree apply to such defensive systems.

She also frequently compared herself to Glaistig Uaine and Kyubey.

Reminder not to sell your soul(s) to the devil.

What I don't get is, why do you have this impulse to sanewash the sides in this discussion?

Is this someone who has a parasocial relationship with Vassar, or a more direct relationship? I was under the impression that the idea that Michael Vassar supports this sort of thing was a malicious lie spread by rationalist leaders in order to purge the Vassarites from the community. That seems more like something someone in a parasocial relationship would mimic than like something a core Vassarite would do.

I have been very critical of cover ups in lesswrong. I'm not going to name names and maybe you don't trust me. But I have observed this all directly. If you are let people toy with your brain while you are under the influence of psychedelics you should expect high odds of severe consequences. And your friends mental health might suffer as well.

I would highlight that the Vassarite's official stance is that privacy is a collusion mechanism created to protect misdoers, and so they can't consistently oppose you sharing what they know.

all FSAs are equivalent

??????

I think one underused trick for training LLMs is to explicitly "edit" them. That is, suppose they generate some text X in response to prompt Y, and it has some error or is missing something. In that case you can create a text X' that fixes this problem, and do a gradient update to increase log P(X'|Y)/P(X|Y).

For example, if we generate virtual comments in the style of certain LW users, one could either let those users browse the virtual comments that have been created in their style and correct them, or one could let the people who receive the virtual comments edit them to remove misunderstanding or similar.

If we think of the quantified abilities as the logarithms of the true abilities, then taking the log has likely massively increased the correlations by bringing the outliers into the bulk of the distribution.

Bayesianism was a mistake.

Your post is an excellent example of how the supposedly-reasonable middle ground tends to be so clueless as to be plausibly worse than the extremes.

Like, e.g. Blanchard doesn’t think trans men have AGP

You mean AAP here, right?

He accepts autohomoeroticism, which is close enough to AAP that the difference doesn't matter. The real problem here is Michael Bailey who has a sort of dogmatic denial of AAP.

doesn’t think trans women who are attracted to men have AGP

That's pretty common in people's second-hand version; the real issue here is that this is sometimes wrong and some androphiles are AGP.

Oversimplification 2: Bisexuals exist. Many trans women report their sexual orientation changing when they start taking hormones. The correlation between having AGP and being attracted to women can’t be as 100% as Blanchard appears to believe it is.

Blanchard explicitly measured that some trans women identified as bisexual, and argued that they were autogynephilic and not truly bisexual. There's some problems with that assertion, but uncovering those problems really ought to engage with more of the nuances than what you imply here.

Oversimplification 4: Do heterosexual cisgender women have AGP? (Cf. Comments by Aella, eigenrobot etc.) if straight cisgender women also like being attractive in the same way as (some) trans women do, it becomes somewhat doubtful that it’s a pathology.

According to qualitative studies I've done, around 15% of women are at least somewhat AGP (though I think it correlates with being bi/lesbian), but the assertion that this implies it's not a pathology for males seems like magical thinking. E.g. ~100% of women have breasts, but this does not mean that developing breasts would not be considered a pathology for males.

If you consider the "true ability" to be the exponential of the subtest scores, then the extent to which the problem I mention applies depends on the base of the exponential. In the limiting case where the base goes to infinity, only the highest ability matter, whereas in the limiting case where the base goes to 1, you end up with something basically linear.

As for whether it's a crux, approximately nobody has thought about this deeply enough that they would recognize it, but I think it's pretty foundational for a lot of disagreements about IQ.

The analogy that I'm objecting to is, if you looked at e.g. the total for a ledger or a budget, it is an index that sums together expenses in a much more straightforward way. For instance if there is a large expense, the total is large.

Meanwhile, IQ scores are more like the geometric mean of the entries on such an entity. The geometric mean tells you whether the individual items tend to be large or small, which gives you broad-hitting information that distinguishes e.g. people who live in high-income countries from people who live in low-income countries, or large organizations from individual people; but it won't inform you if someone got hit by a giant medical bill or if they managed to hack themselves to an ultra-cheap living space. These pretty much necessarily have to be low-rank mediators (like in the g model) rather than diverse aggregates (like in the sum model).

(Well, a complication in this analogy is that a ledger can vary not just in the magnitude of the transfers but also qualitatively in the kinds of transfers that are made, whereas IQ tests fix the variables, making it more analogous to a standardized budget form (e.g. for tax or loan purposes) broken down by stuff like "living space rent", "food", "healthcare", etc..)

That's part of the problem, often the bad middle ground looks superficially plausible, so it's very sticky and hard to get rid of, because it's not exactly that people get told the wrong things but rather that they spontaneously develop the wrong ideas.

The three basic issues with this viewpoint are:

1. IQ test batteries do not measure even close to all cognitive abilities and realistically could never do that.
2. Many of the abilities that IQ scores weight highly are practically unimportant.
3. Differential-psychology tests are in practice more like log scales than like linear scales, so "sums" are more like products than like actual suns; even if you are absurdly good at one thing, you're going to have a hard time competing with someone in IQ if they are moderately better at many things.

Ever since the situation with Blanchardianism, I've become extremely bearish on the possibility on this, considering how everyone on all sides including rationalists on all sides of the debate just massively failed on it.

With IQ realism, you also get insane stuff where the supposedly reasonable middle ground tends to have skepticism about the g factor and thinks of IQ as an index that sums together cognitive abilities.

I haven't thought of this in relation to wild animal welfare or birthrates but I don't immediately see the argument that we can outperform the abysmal track record seen in these two other cases.

A possible model is that while good startups have an elevation in the "cult-factor", they have an even greater elevation in the unique factor related to the product they are building. Like SpaceX has cult-like elements but SpaceX also has Mars and Mars is much bigger than the cult-like elements, so if we define a cult to require that the biggest thing going on for them is cultishness then SpaceX is not a cult.

This is justified by LDSL (I really should write up the post explaining it...).

I feel like one should use a different term than vitalism to describe the unpredictability, since Henri Bergson cane up with vitalism based on the idea that physics can make short-term predictions about the positions of things but that by understanding higher powers one can also learn to predict what kinds of life will emerge etc..

Like let's say you have a big pile of grain. A simple physical calculation can tell you that this pile will stay attached to the ground (gravity) and a more complex one can tell you that it will remain ~static for a while. But you can't use Newtonian mechanics, relativity, or quantum mechanics to predict the fact that it will likely grow moldy or get eaten by mice, even though that will also happen.

Singapore and the US both have a military, a police, and taxation. This seems much more clear-cut to me than "cults" do.

I think maybe one could treat "cult" more like a pronoun than like a theoretical object. Like when one is in the vicinity of one of the groups Ben Pace mentioned, it makes sense to have a short term to talk about the group, and "the cult" is useful for disambiguating the cult from other groups that might be present.

I like vaccines and suspect they (or antibiotics) account for the majority of the value provided by the medical system. I don't usually see discussion of what can be done to promote or improve vaccines, so I don't know much about it, but the important part is they remain available and get improved and promoted in whatever ways are reasonable.

Beyond that, a major health problem is obesity and here semaglutide seems like it would help a lot.

I think there's something to this. Also since making the OP, I've been thinking that human control of fire seems important. If trees have the majority of the biomass, but humans can burn the trees for energy or just to make space, then that also makes humans special (and overlaps a lot with what you say about energy controlled).

This also neatly connects human society to the evolutionary ecology since human dominance hierarchies determine who is able to control what energy (or set fire to what trees).

The OP is more of a statement that you get different results depending on whether you focus on organism count or biomass or energy flow. I motivate this line of inquiry by a question about what evolution selects for, but that's secondary to the main point.

In the case of gradient flow, we expect almost-all starting conditions to end up in a similar functional relationship when restricting attention to their on-distribution behavior. This allows us to pick a canonical winner.

Evolution is somewhat different from this in that we're not working with a random distribution but instead a historical distribution, but that should just increase the convergence even more.

The noteworthy part is that despite this convergence, there's still multiple winners because it depends on your weighting (and I guess because the species aren't independent, too).

But the problem I mention seems to still apply even if you hold the environment fixed.

E.g. suppose there's some game where you can reproduce by getting resources, and you get resources by playing certain strategies, and it turns out there's an equilibrium where there's 90% strategy A in the ecosystem (by some arbitrary accounting) and 10% strategy B. It's kind of silly to ask whether it's A or B that's winning based on this.

But this is an abstraction that would never occur in reality. The real systems that inspire this sort of thing have lots of pelagibacter communis and the strategies A and B are constantly diverging off into various experimental organisms that fit neither strategy and then die out.

When you choose to model this as a mixture of A and B, you're already implicitly picking out both A and B as especially worth paying attention to - that is, as "winners" in some sense.

Actually I guess I endorse this response in the real world too, where if a species is materially changing to exploit a new niche, it seems wrong to say "oh, that old species that's totally dead now sure were winners." If the old species had particular genes with a satisfying story for making it more adaptable than its competitors, perhaps better to take a gene's-eye view and say those genes won. If not, just call it all a wash.

But in this case you could just say A' is winning over A. Like if you were training a neural network, you wouldn't say that your random initialization won the loss function, you'd say the optimized network scores better loss than the initial random initialization.

I like to treat the environment/ecology as the cause. So that e.g. trees are caused by the sun.

I kind of feel like pelagibacter communis could maybe be seen as "evolutionary heat" or "ecological heat", like in the sense that the ecology has "space for" some microscopic activity so whatever major ecological causes pop up, some minimal species will evolve to fill up the minimal-species-niche.

I wouldn't go this far yet. E.g. I've been playing with the idea that the weighting where humans "win" evolution is something like adversarial robustness. This just wasn't really a convincing enough weighting to be included in the OP. But if something like that turns out correct then one could imagine that e.g. humans get outcompeted by something that's even more adversarially robust. Which is basically the standard alignment problem.

Like I did not in fact interject in response to Nate or Eliezer. Someone asked me what triggered my line of thought, and I explained that it came from their argument, but I also said that my point was currently too incomplete.

Right, I think there are variants of it that might work out, but there's also the aspect where some people argue that AGI will turn out to essentially be a bag-of-heuristics or similar, where inner alignment becomes less necessary because the heuristics achieve the outer goal even if they don't do it as flexibly as they could.

Richard Kennaway asked why I would think in those lines but the point of the OP isn't to make an argument about AI alignment, it's merely to think in those lines. Conclusions can come later once I'm finished exploring it.

Some people say that e.g. inner alignment failed for evolution in creating humans. In order for that claim of historical alignment difficulty to cash out, it feels like humans need to be "winners" of evolution in some sense, as otherwise species that don't achieve as full agency as humans do seem like a plausibly more relevant comparison to look at. This is kind of a partial post, playing with the idea but not really deciding anything definitive.

I had been playing on and off with the idea that an ecological argument would show dolphins to be ultimately fish-like, but with my switch in general approach to things, I think ultimately the "dolphins are not fish" side wins out. Some of the most noteworthy characteristics of dolphins is that they are large, intelligent, social animals which provide parental care for extensive periods of time. There are literally 0 fish species with this combination of traits, whereas meanwhile the combo obviously screams "mammal!".

I was wondering if it was specific to mammals or if it applied to land vertebrates in general. Other animals that have evolved to live in the ocean include sea turtles and sea snakes, but they are quite land-animal-like to me. Frogs are an interesting edge-case in that at least they have marine-like offspring counts, but they literally have legs so obviously they don't count.

But red exists only because we have the experience of seeing red. That is, red exists because we have found it useful to tell red apart from other colors. We can "objectively" define red to be light with a wavelength between 620 and 750 nanometers, but we define it thus because those wavelengths correspond to what many people subjectively identify as red. Thus, whether or not an apple is red is neither a properly objective nor subjective fact, but intersubjective knowledge that depends on both the world and how we experience it. So it goes for all truth that can be known.

At first glance, red seems like such a special color to me. It's the color of blood and many fruit, the advanced color that primates and flying animals see and other animals can't distinguish, it's the most primitive step towards heat vision, and it's the color at the lowest end of the range of common electron bandgap. Obviously the word itself is kind of arbitrary but the color seems as non-arbitrary as could be.

And I guess I should say, I have a more sun-oriented and less competition-oriented view. A surplus (e.g. in energy from the sun or negentropy from the night) has a natural "shape" (e.g. trees or solar panels) that the surplus dissipates into. There is some flexibility in this shape that leaves room for choice, but a lot less than rationalists usually assume.

Kind of. First, the big exception: If you manage to enforce global authoritarianism, you can stockpile surplus indefinitely, basically tiling the world with charged-up batteries. But what's the point of that?

Secondly, "waste/signaling cascade" is kind of in the eye of the beholder. If a forest is standing in some region, is it wasting sunlight that could've been used on farming? Even in a very literal sense, you could say the answer is yes since the trees are competing in a zero-sum game for height. But without that competition, you wouldn't have "trees" at all, so calling it a waste is a value judgement that trees are worthless. (Which of course you are entitled to make, but this is clearly a disagreement with the people who like solarpunk.)

But yeah, ultimately I'm kind of thinking of life as entropy maximization. The surplus has to be used for something, the question is what. If you've got nothing to use it for, then it makes sense for you to withdraw, but then it's not clear why to worry that other people are fighting over it.

Electoral candidates can only be very bad because the country is very big and strong, which can only be the case because there's a lot of people, land, capital and institutions.

Noticing that two candidates for leading these resources are both bad is kind of useless without some other opinion on what form the resources should enter. A simple option would be that the form of the resources should lessen, e.g. that people should work less. The first step to this is to go away from Keynesianism. But if you take that to its logical conclusion, it implies e/acc replacement of humanity, VHEM, mass suicide, or whatever. It's not surprising that this is unpopular.

So that raises the question: What's some direction that the form of societal resources could be shifted towards that would be less confusing than a scissor statement candidate?

Because without an answer to this question, I'm not sure we even need elaborate theories on scissor statements.

Offense/defense balance can be handled just by ensuring security via offense rather than via defense.

I guess as a side-note, I think it's better to study oxidation, the habitable zone, famines, dodo extinction, etc. if one needs something beyond the basic "dangerous domains" that are mentioned in the OP.

I think one thing that's missing here is that you're making a first-order linear approximation of "research" as just continually improving in some direction. I would instead propose a quadratic model where there is some maximal mode of activity in the world, but this mode can face certain obstacles that people can remove. Research progress is what happens when there's an interface for removing obstacles that people are gradually developing familiarity with (for instance because it's a newly developed interface).

Different people have different speeds by which they reach the equillibrium, but generally those who have an advantage would also exhibit an explosion of skills and production as they use their superior understanding of the interface.

My original response contained numerous strategies that people were using:

• Keeping one's cryptocurrency in cold storage rather than easily usable
• Using different software than that with known vulnerabilities
• Just letting relatively-trusted/incentive-aligned people use the insecure systems
• Using mutual surveillance to deescalate destructive weaponry
• Using aggression to prevent the weak from building destructive weaponry

You dismissed these as "just-so stories" but I think they are genuinely the explanations for why stuff works in these cases, and if you want to find general rules, you are better off collecting stories like this from many different domains than to try to find The One Unified Principle. Plausible something between 5 and 100 stories will taxonomize all the usable methods and you will develop a theory through this sort of investigation.

No, it was a lot of words that describe why your strategy of modelling stuff as more/less "dangerous" and then trying to calibrate to how much to be scared of "dangerous" stuff doesn't work.

The better strategy, if you want to pursue this general line of argument, is to make the strongest argument you can for what makes e.g. Bitcoin so dangerous and how horrible the consequences will be. Then since your sense of danger overestimates how dangerous Bitcoin will be, you can go in and empirically investigate where your intuition was wrong by seeing what predictions of your intuitive argument failed and what obstacles caused them to fail.

You shouldn't use "dangerous" or "bad" as a latent variable because it promotes splitting. MAD and Bitcoin have fundamentally different operating principles (e.g. nuclear fission vs cryptographic pyramid schemes), and these principles lead to a mosaic of different attributes. If you ignore the operating principles and project down to a bad/good axis, then you can form some heuristics about what to seek out or avoid, but you face severe model misspecification, violating principles like realizability which are required for Bayesian inference to get reasonable results (e.g. converge rather than oscillate, and be well-calibrated rather than massively overconfident).

Once you understand the essence of what makes a domain seem dangerous to you, you can debug by looking at what obstacles this essence faced that stopped it from flowing into whatever horrors you were worried about, and then try to think through why you didn't realize those obstacles ahead of time. As you learn more about the factors relevant in those cases, maybe you will learn something that generalizes across cases, but most realistically what you learn will be about the problems with the common sense.

Your error is in having inferred that there is a general rule that this necessarily happens. MAD is obviously governed by completely different principles than crypto is. Or maybe your error is in trusting common sense too much and therefore being too surprised when stuff contradicts it, idk.

For almost everything, yeah, you just avoid the bad parts.

In order to predict the few exceptions, one needs a model of what functions will be available in society. For instance, police implies the need to violently suppress adversaries, and defense implies the need to do so with adversaries that have independent industrial capacity. This is an exception to the general principle of "just avoid the bad stuff" because while your computer can decline to process a TCP packet, your body can't decline to process a bullet.

If someone is operating e.g. an online shop, then they also face difficulty because they have to physically react to untrusted information and can't avoid that without winding down the shop. Lots of stuff like that.

Imagine your typical computer user (I remember being mortified when running anti-spyware tool on my middle-aged parents' computer for them).  They aren't keeping things patched and up-to-date. What I find curious is how can it be the case that their computer is both: filthy with malware and they routinely do things like input sensitive credit-card/tax/etc information into said computer.

I don't know what exactly your parents are using their computer for.

If we say credit-card information, I know at least in my country there's a standard government-mandated 2-factor authentication which helps with security. Also, banks have systems to automatically detect and block fraudulent transactions, as well as to reverse and punish fraudulent transactions, which makes it harder for people to exploit.

In order to learn how exactly the threats are stopped, you'd need to get more precise knowledge of what the threats are. I.e., given a computer with a certain kind of spyware, what nefarious activities could you worry that spyware enables? Then you can investigate what obstacles there are on the way to it.

I fully expect to live in a world where its BOTH true that: Pilny the Liberator can PWN any LLM agent in minutes AND people are using LLM agents to order 500 chocolate cupcakes on a daily basis.

Using an LLM agent to order something is a lot less dangerous than using an LLM agent to sell something, because ordering is kind of "push"-oriented; you're not leaving yourself vulnerable to exploitation from anyone, only from the person you are ordering from. And even that person is pretty limited in how they can exploit you, since you plan to pay afterwards, and the legal system isn't going to hold up a deal that was obviously based on tricking the agent.

In crypto, a lot of people just HODL instead of using it for stuff in practice. I'd guess the more people use it, the more likely they are to run into one of the 99.9% of projects that are scams. (Though... if we count the people who've been hit by ransomware, it is non-obvious to me that the majority of users are HODLers rather than ransomeware victims.) To prevent losing one's crypto, there have also been developed techniques like "cold storage", which are extremely secure.

The HTTP server logs you posted aren't based on insecurity of most webservers, they are based on the insecurity of particular programs (or versions of programs or setups of programs). Important systems (e.g. online banking) almost always use different systems than the ones that are currently getting attacked. Attacks roll the dice in the hope that maybe they'll find someone with a known vulnerability to exploit, but presumably such exploits are extremely temporary.

Copilot is general instructed via the user of the program, and the user and is relatively trusted. I mean, people are still trying to "align" to be robust against the user, but 99.9% of the time that doesn't matter, and the remaining time is often stuff like internet harassment which is definitely not existentially risky, even if it is bad.

Some people are trying to introduce LLM agents into more general places, e.g. shops automatically handling emails from businesses. I'm pretty skeptical about this being secure, but if it turns out to be hopelessly insecure, I'd expect the shops to just decline using them.

Nuclear weapons were used twice when only the US had them. They only became existentially dangerous as multiple parties built up enormous stockpiles of them, but at the same time people understood that they were existentially dangerous and therefore avoided using them in war. More recently they've agreed that keeping such things around is bad and have been disassembling them under mutual surveillance. And they have systems set up to prevent other, less-stable countries from developing them.