Thought I'd comment in brief. I very much enjoyed your post and I think it is mostly right on point. I agree that EA does not have a great epistemic hygiene, given what their aspirations are, and the veganism discussion is a case in point. (Other issues related to EA and CEA have been brought up lately in various posts, and are not worth rehashing here.)

As far as the quoted exchange with me, I agree that I have not stated a proper disclaimer, which was quite warranted, given the thrust of the post. My only intended point was that, while a lot of people do veganism wrong and some are not suited to it at all, an average person can be vegan without adverse health effects, as long as they eat varied and enriched plant-based diet and periodically check their vitamins/nutrients/minerals levels and make dietary adjustments as necessary. Some might find out that they are in the small minority for whom vegan diet is not feasible, and they would do well to focus on what works for them and contribute to EA in other ways. Again, I'm sorry this seems to have come across wrong. 

Oh, and cat veganism is basically animal torture, those who want to wean cats off farmed animal food should focus on vat-grown meat for pet food etc. 

Sure, it's not necessary that a sufficiently advance AI has to work like the brain, but there has to be an intuition about why is not need it to at least create an utility maximizer.

Octopus' brain(s) is nothing like that of mammals, and yet it is equally intelligent.

"Sanity" may not be a useful concept in edge cases, but yes, being able to trust your mind to autopilot is definitely within the central definition of sanity, it's a good observation.

You may also be interested in Scott's post series on the topic, the latest being https://www.astralcodexten.com/p/contra-kirkegaard-on-evolutionary

FTFY: "Smile at strangers iff it has non-negative EV, because smiling is cheap and sometimes it does". 

"I am going to read you mind and if you believe in a decision theory that one-boxes in Newcomb's Paradox I will leave you alone, but if you believe in any other decision theory I will kick you in the dick"

Sure, that's possible. Assuming there are no Newcomb's predictors in that universe, but only DK, rational agents believe in two-boxing. I am lost as to how it is related to your original point.

Let me clarify what I said. Any decision theory or no decision theory at all that results in someone one-boxing is rewarded. Examples: Someone hates touching transparent boxes. Someone likes a mystery of an opaque box. Someone thinking that they don't deserve a guaranteed payout and hoping for an empty box. Someone who is a gambler. Etc. What matters is the outcome, not the thought process.

How well can you predict what will be good in 100 years? For perspective, given the contemporary zeitgeist of 1923, rather than knowing all we know now, what would have been considered a great policy to enact then that would take effect in 2023?

What if you believe in DKRUDT, the "Dick Kick'em rewards you" decision theory?

Seriously though, Newcomb's setup is not adversarial in the same way, the predictor rewards or punishes you for actions, not beliefs. Your internal reasoning does not matter, as long as you end up one-boxing you walk away with more money.

As someone who gets rate-limited due to downvoted comments occasionally, I can see a rather strong correlation between the style/tone and the response. It is very much possible to express controversial, contrarian or even outright silly views without being downvoted. The rule of thumb is to be respectful and charitable to your opponents and, well, read the room. The more your view diverges from the local mainstream, the better, tighter and clearer your argument must be. 

It is standard practice when writing scientific papers to first demonstrate intimate familiarity with the views you argue against later. This is what you would want from someone else writing a refutation of your argument, right? So give them the same courtesy. You can cut corners if your post or comment is close to the mainstream, but it is not a great habit. If you want to learn how to do it right, read ACX, Scott is a master of this art.

it is a common pitfall to blame the society/subculture for the negative reaction you get. It is almost never a constructive way to proceed. 

If you are in doubt as to how your post would be received, and you want to get through to more people, consider reaching out to someone familiar with this site to review your draft. Or to anyone, really. The bar for writing a post or a comment with non-negative expected karma is pretty low.

it's not a probability question. Blood tests exist.

Your potential contribution to timeline shortening as an undergrad is probably negligible on the margins, unless you are in the top 1% of all applicants, or even then. The field is crowded by very intelligent, well paid and very determined people who do this for a living. So whatever considerations you have, they need not be influenced by the AGI killeveryoneism. 

That's a fair point, different directions have different landmines. Culture war issues are tempting and most people seem to have strong yet poorly informed opinions that are not obviously poorly informed. I think that the original collection by Rob Bensinger The Library of Scott Alexandria is really good to start with, it is very light on political and culture war topics and grasps the essence of rational thinking, without going into the esoteric and irrelevant topics like quantum mechanics.

I find it hard to not assign whatever issues or problems we have to how the Universe works.

Indeed, the Universe just is, everything else is an emergent concept for the tiny embedded agents in it (bacteria, ants and humans). That includes sugar gradient, scent of food and laws of physics, respectively.

One of these emergent concepts is personal responsibility. Societies that do not have it do not last as long.

Step away from this place for a time and read Scott Alexander first. https://www.lesswrong.com/posts/vwqLfDfsHmiavFAGP/the-library-of-scott-alexandria is a good start, then continue on his blogs directly. He links back to this site where appropriate. You might get inured to some of the common pitfalls newbies here are prone to.

I agree with your analysis of the current situation. However, the technological issues arise when trying to correct it without severe unintended consequences, and that is not related to profit. You can't transplant a house easily. You cannot easily feed only those who go hungry without affecting the economy (food banks help to some degree). There are people in need of companionship that cannot find it, even though there is a companion that would match somewhere out there. There are potential technological solutions to all those that are way outside our abilities (teleportation! replication! telepathy!) that would solve these issues. You can also probably find a few examples where what looks like profit-based incentive is in fact a technological deficiency.

I guess my point is the standard one: in many ways even poor people live a lot better now than royalty 300 years ago.

I'd assume that people themselves would define what they need, within the limits of what is possible given the technology of the time.

You might be thinking about it in a wrong way. Societal structures follow capabilities, not wants. If you try to push for "each person works and is paid according to their abilities and needs" too early, you end up with communist dystopias. If we are lucky, the AGI age will improve our capabilities enough where "to everyone according to their needs" may become feasible, aligning the incentives with well-being rather than with profit. So, to answer your questions:

• It is currently impossible to "align the incentives" without causing widespread suffering.
• It is undesirable if you do not want to cause suffering.
• It is ineffective to try to align the incentives away from profit if your goal is making them aligned with "human well being".

That said, there are incremental steps that are possible to take without making things worse, and they are discussed quite often by Scott Alexander and Zvi, as well as by others in the rationalist diaspora. So read them.

There is no a priori reason to believe that world has to be learnable. But if it were not, then we wouldn’t exist, nor would (most?) animals. The existing world, thus, is learnable. The human sensorium and motor system are necessarily adapted to that learnable structure, whatever it is.

I have a post or a post draft somewhere discussing this issue. The world indeed just is. It does not have to be internally predictable to an arbitrary degree of accuracy, but it needs to be somewhat internally predictable in order for the long-lived patterns we identify as "agents" or "life" to exist. Internal predictability (i.e. that an incredibly tiny part of the universe that is a human (or a bacterium) can infer enough about the world to not immediately poof away) is not something that should be a given in general. Further, even if some coarse-grained internal predictability can be found in such a world, there is no guaranteed that it can be extended to arbitrarily fine accuracy. it might well be the case in our world that at some point we hit the limit of internal predictability and from then on things will just look random for us. Who knows, maybe we hit it already, and the outstanding issues in the Standard Model of Cosmology and/or the Standard Model of Particle Physic, and/or maybe some of the Millennium prize problems, and/or the nature of consciousness are simply unknowable. I hope this is not the case, but I do not see any good argument that says "yep, we can push much further", other than "it worked so far, if in fits and starts".

I guess that is one way to say it. But the statement is stronger than that, I think. They do not care about the box or about anything else. They react to stimuli, then go silent again.

Yeah, I get that. But to look into the future one must take stock of the past and present and reevaluate models that gave wrong predictions. I am yet to see this happening.

I do not know if we will or will not build something recognizable agentic any time soon. I am simply pointing out that currently there is a sizable gap that people did not predict back then. Given that we still have no good model what constitutes values or drives (definitely not a utility function, since LLMs have plenty of that), I am very much uncertain about the future, and I would hesitate to unequivocally state that "AGI isn't just a technology". So far it most definitely is "just a technology", despite the original expectations to the contrary by the alignment people.

Maybe we understand agency differently. You give LLMs tools to use, they will use it. But there is no discernable drive or "want" to change the world to their liking. Not saying that it won't show up some day, it's just conspicuously lagging the capabilities. 

I've been following the "tools want to become agents" argument since Holden Karnofsky raised the topic a long time ago, and I was almost convinced by the logic, but the LLMs show a very surprising lack of agency, and, as far as I can tell, this gap between apparent intelligence and apparent agency was never predicted or expected by the alignment theorists. I would trust their cautions more if they had a model that makes good predictions.

This seems to assume that AI has its own drives and values. So far this has not been the case.

Interesting! It seems like something like that should be a canonical reference for "let's enter a problem" e.g. smoking lesion, then "select decision theory", and out pops the answer. Of course, formalizing the problem seems like the hard part.

Is there a formalization of FDT that can be fed into a computer rather than argued about by fallible humans?

I guess it's not universally considered a fallacy. But think of it as waiting on a bus stop without knowing the schedule. By a similar argument (your random arrival time has the mean and median in the middle between two buses), the time you should expect to wait for the bus to arrive is the same as the time you have already been waiting. This is not much of a prediction, since your expectation value changes constantly, and the eventual bus arrival, if any, will be a complete surprise. The most you can act on is giving up and leaving after waiting for a long time, not really a possible action in either the doomsday scenario or when talking about AGI consciousness.

The argument is simple. If the future is filled with artificial intelligence of human origin, and if that AI is conscious, then any given observer should expect to be one of those AIs.

That's a doomsday argument fallacy.

Notably, about three quarters of decision theorists two box.

I know... and I cannot wrap my head around it. They talk about causality and dominant strategies, and end up assigning non-zero weight to a zero-probability possible world. It's maddening. 

Eliezer's specific argument against physicalism shows that he doesn't know what he's talking about

I see. Not very surprising given the pattern. I guess my personal view is that non-physicalism is uninteresting given what we currently know about the world, but I am not a philosopher.

I didn't comment on Everetianism because I don't know enough (just that I think it's suspicious that Eliezer is so confident) nor on probability theory.  I didn't claim there was a contradiction between Bayesian and frequentist methods. 

Right, my point is that is where a critique of Eliezer's writings (and exposing his overconfidence and limited knowledge) would be a lot stronger.

Not directly related, but:

Pain is a subset of suffering--it's the physical version of suffering, but the same argument can be made for suffering.

Pain can be fun! If your mindset is right. it can also alleviate suffering, physical and especially emotional. But yes, there is a big overlap, and the central example of pain is something that causes suffering. 

First, the sentiment

Eliezer sounds good whenever he’s talking about a topic that I don’t know anything about.

is widely shared. This local version of the Gell-Mann amnesia effect is very pervasive here.

It is also widely acknowledged even by people who respect Eliezer that he is wildly overconfident (and not very charitable to different views).

However, the examples you pick are kind of... weak? If you argue for non-physicalism being a viable model of reality, you are fighting not just Eliezer, but a lot of others who do not suffer from Eliezer's shortcomings. If you think that two-boxing in the classic Newcomb's has merit, you are definitely not in a good company, at least not financially. Even your arguments related to animal consciousness are meh. Pain and suffering are two different things. There are many different definitions of consciousness, and Eliezer's seems to be the one close to self-awareness (an internal narrator). 

There are better arguments. Sean Carroll, a very outspoken Everettian, has described why he picks this model, but also explicitly describes what experiments would change his mind. If you talk to actual professional probabilists, they will explain why there is no contradiction between Bayesian and frequentist methods. They will also carefully and explicitly acknowledge where Eliezer is right and where his dilettantism makes him go off the rails. 

Ironically, you fall into the same trap you accuse Eliezer of: being wrong yet dismissive and overconfident. And also kind of rude. I understand the anguish you felt once you escaped the near-hypnotic state that many people experience when reading the sequences. There are better ways to get your point across.

That said, I think your conclusion actually make a lot of sense. The sequences are really good at systematizing, clarifying and simplifying. There are a number of original ideas, too, but that is not the main appeal. I definitely approve your caution that 

My goal in this is not to cause people to stop reading Eliezer. It’s instead to encourage people to refrain from forming views on things he says just from reading him

This is probably as far toward pure Bayesianism as is reasonable to go:

https://arxiv.org/abs/1801.05016

Tragically, you can't connect to the real world using experiment alone.

Yes, you absolutely need both. I don't think anyone argues that point?

These are all good ideas, but they are not a substitution for testing. 

If your intent is to argue that it is futile to reason in absence of the ability to test hypotheses

The goal of reasoning is to eventually connect with experiment, i.e. to make accurate predictions. I have read the sequences (and helped name the book), but I strongly disagree with a lot of the posts. Specifically, When Science Can't Help is very misleading. Sorry you got misled. The simulation hypothesis, is not a hypothesis, it's a speculation with no way to connect to the real world, whatever it might be. I am not saying it's wrong, it's not even wrong. Focus on something that can be helpful to you and ignore this rubbish.

as a bayesian, you can reason about the probability of hypotheses you can't test. for an example in this context, you could check Bostrom's paper on this.

How do you know when your reasoning is faulty (as all human reasoning is) without experimental feedback? In absence of one, it is indeed about how it makes you feel.

In other words, the plan has two steps:

1. Make a machine that can do literally anything
2. Configure it to do good things

I don't think this is the plan? The hope is that, as capabilities grow, so does alignment, whatever this "alignment" thing is. The reality is different, of course.

Not sure about quantifying, but it is very sad to see that TikTok videos, movies and shows is what people think DID is like. It is sort of like believing that John Cena (or Chuck Norris, for those from an older generation) is like what we see on TV and in memes. In reality there is basically nothing in common. Most systems are covert and one would no more recognize one in daily life, as one would recognize a gay person in a country where "sexual deviance" is heavily persecuted. Actually, for context, most times DID is not self-diagnosed, but noticed by the mental health professionals. Most systems have a strong denial alter or multiple alters, a lot of self-hate and a lot of other trauma-related issues. Hopefully the hype dies down, and people who suffer terribly with this condition get proper professional help (there is a dearth of therapists understanding the condition, misconceptions abound), but the future is not looking bright so far.

No-one dies because I redpilled.

That's where bluepillers disagree. They feel they cannot hide behind the numbers. Someone WILL bluepill, humans are diverse enough for that to guarantee to happen. There is an easy way to contribute to their survival, unless you don't have empathy for people who are not like you. As you say "Nothing of value is actually being saved." --- bluepillers are not people, as far as you are concerned.

As mentioned in another comment, efficiency is not a bottleneck. You don't have to rely on solar radiation toward Earth to increase entropy for fun and profit. There are other energy sources around. We are not using fission much, we are not spending nearly enough resources to figure out fusion, geothermal is all but untapped, there are countless other sources, as well, and that's just on Earth. 

This problem is not a utilitarian expected utility calculation. If you redpill and someone dies partly because you redpilled, well, some people don't care and others can't take it.

Fixed the link, it has been discussed here several times already, the blue pill vs the red pill poll, just not as nearly-scissor statement.

I agree with your point that bandwagoning is a known exploit, and LLMs are a force multiplier there, compared to, say tweet bots. Assuming I got your point right. Get LLMs to generate and iterate the message in various media forms until something sticks is definitely a hazard. I guess my point is that this level of sophistication may not be necessary, as you said

The human brain is a kludge of spaghetti code, and it therefore follows that there will be exploitable "zero days" within most or all humans.

Your cognitive empathy skills are sorely lacking.

A couple of points.

Note that human valorize successful brain hacking. We call them "insights", "epiphanies" and "revelations", but all they are is successful brain hacks. If you take a popular novel or a movie, for example, there will invariably be a moment when the protagonist hears some persuasive argument and changes their mind on the spot. Behold, a brain hack. We love it and eat it up.

Also note that it is easy to create a non-lethal scissor statement with just a few words, like this bluepilling vs redpilling debate from a couple of days ago:  https://twitter.com/lisatomic5/status/1690904441967575040 . This seems to have a similar effect.

I guess the latter confirms your point about the power of social media, though without any bells and whistles of video, LLMs or bandwagoning. 

I feel like you are being unnecessarily obtuse

I guess there is no useful discussion possible after a statement like that.

A bit less than game-theoretical argument here.

Note that a significant consideration is that one does not want to be "morally responsible" for "those clueless bluepillers" dying. Absent uniformity or perfect coordination, PEOPLE DIE!!!111. 

Another note: people are often willing to risk their own lives in order to avoid this guilt. Survivor's guilt is a thing you can't easily logic away. These people will vote blue, because the alternative is unthinkable. 

Finally, if you care about these people, even though you realize that redpilling is the safe personal choice, bluepilling is a decision worth considering seriously.

Well, yes, we do understand the low-temp superconductors and its limitations. We do not really understand High-Tc superconductors, and it seems to impair the search for better versions. It's quite possible that if LK-99 turns out to be more than a false positive, its workings might still remain an open problem for a long time. Or maybe, as you suggest, we figure out how it works, but still will not be able to use it to guide the design of something significantly more convenient. So, yeah, it is possible that the search for more convenient versions will be left to material scientists rather than physicists proper. Still, it seems like there is a lot of improvement needed before we can produce these potential ambient superconductors in useful quantities and sizes.

One of the first steps would be figure out the theory of how it works, and based on it, synthesize ambient SC materials that are easy to produce repeatably and reliably and are easy to work with. The next steps depend on the outcome of that undertaking.

GPT-4 is an AGI, but not a stereotypical example of AGI as generally discussed on this forum for about a decade (no drive to do anything, no ability to self-improve), but basically an Oracle/tool AI, as per Karnofsky's original proposal in the discussion with Eliezer here. The contrast between the apparent knowledge and the lack of drive to improve is confusing. if you recall, the main argument against a Tool AI was "to give accurate responses it will have to influence the world, not just remain passive". GPT shows nothing of the sort. It will complete the next token the best it can, hallucinate pretty often, and does not care if it hallucinates even when it knows it does. I don't think even untuned models like Sydney showed any interest in changing anything about their inputs. GPT is not an agentic AGI that we all had in mind for years, and not clearly close to being one.

In contrast, Waymo robotaxis are precisely the stereotypical example of a robotaxi: a vehicle that gets you from point A to point B without a human driver. It will be a profitable business, and how soon depends mostly on regulations, not on capabilities. There are still ways to go to improve reliability in edge cases, but it is already better than a human driver most of the time.

One argument is that state-of-the art generative models might become a lot more agentic if given real world sensors and actuators... say, put inside a robotaxi. Who knows, but it does not look obviously so at this point. There is some sort of AI controlling most autonomous driving vehicles already, I have no idea how advanced compared to GPT-4.

Sorry if this is not the feedback you want.

Epistemology is the branch of philosophy concerned with how we figure out what's true.

That's... not quite what epistemology is? It is a study of how humans construct, test and improve their models? Focused on accuracy and domain of applicability, not specifically on "truth".

Why this particular number?