This relates to my favorite question of economics: are graduate students poor or rich? This post suggests an answer I hadn’t thought of before: it depends on the attitudes of the graduate advisor, and almost nothing else.

Just in case people aren't aware of this, drilling wells the "old fashioned way" is a very advanced technology. Typically a mechanically complex diamond-tipped tungsten carbide drill bit grinds its way down, while a fluid with precisely calibrated density and reactivity is circulated down the center of the drill string and back up the annulus between the drill string and edges of the hole, sweeping the drill cuttings up the borehole to the surface. A well 4 miles long and 8 inches wide has a volume of over 200,000L, meaning that's the volume of rock that has to be mechanically removed from the hole during drilling. So that's the volume of rock you would have to "blow" out of the hole with compressed air. You can see why using a circulating liquid with a reasonably high viscosity is more efficient for this purpose.

The other important thing about drilling fluid is that its density is calibrated to push statically against the walls of the hole as it is being drilled, preventing it from collapsing inward and preventing existing subsurface fluids from gushing into the wellbore. If you tried to drill a hole with no drilling fluid, it would probably collapse, and if it didn't collapse, it would fill with high pressure groundwater and/or oil and/or explosive natural gas, which would possibly gush straight to the surface and literally blow up your surface facilities. These are all things that would almost inevitably happen if you tried to drill a hole using microwaves and compressed air.

tl;dr, drilling with microwaves might sense if you're in space drilling into an asteroid, but makes so no sense for this application.

Talking to Golden Gate Claude reminds me of my relationship with my sense of self. My awareness of being Me is constantly hovering and injecting itself into every context. Is this what "self is an illusion" really means? I just need to unclamp my sense of self from its maximum value?

I think it is also good to consider that it's the good-but-not-great hardware that has the best price-performance at any given point in time. The newest and best chips will always have a price premium. The chips one generation ago will be comparatively much cheaper per unit of performance. This has been generally true since I've started recording this kind of information.

As I think I mentioned in another comment, I didn't mention Moore's law at all because it has relatively little to do with the price-performance trend. It certainly is easy to end up with a superexponential trend when you have an (economic) exponential trend inside a (technological) exponential trend, but as other commenters point out, the economic term itself is probably superexponential, meaning we shouldn't be surprised to see price-performance to fall more quickly than exponential even without exponential progress in chip speed.

One way of viewing planning is as an outer-loop on decision theory.

My approach to the general problem of planning skills was to start with decision theory and build up. In my Guild of the Rose Decision Theory courses was to spend time focusing on slowly building the most fundamental skills of decision theory. This included practicing manipulation of probabilities and utilities via decision trees, and practicing all these steps in a variety of both real and synthetic scenarios, to build an intuition regarding the nuances of how to set up decision problems on paper. The ultimate goal was to get the practitioners to the point where they usually don't need to draw up a decision tree on paper, but rather to leverage those intuitions to quickly solve decision problems mentally, and/or recognize when a decision problem is actually tricky enough to merit breaking out the spreadsheet or Guesstimate project.

In my experience, even long-time rationalists are so incredibly bad at basic decision theory that trying to skip the step of learning to correctly set up a basic decision tree might actually be counterproductive. So my inclination is to focus on really mastering this art before attempting planning.

Another way of viewing planning is that planning is search. 

For computationally bounded agents like us, search involves a natural tradeoff of breadth versus depth. Breadth is essentially idea generation, depth is idea selection and refinement. The tricky think about planning, in general, is that if 100x solutions exist, then those solutions are going to be found by spending the majority of the time on breadth-search, i.e. blue sky brainstorming for ways that the plan could look wildly different from the default approach, but that most situations don't admit 100x plans. Most things in life, especially in our technological civilization, are already sort of optimized, because there is some existing refined solution that has already accommodated the relevant tradeoffs. I could get to work faster if I flew there in a helicopter, but considering in costs, the Pareto optimum is still driving my car on the freeway. Most things look like this. Well-considered Pareto solutions to real-world problems tend to look boring!

Therefor, if you spend a lot of time looking for 100x solutions, you will waste a lot of time, because these solutions usually won't exist. Then, after failing to find a truly galaxy-brain solution, you will spend some amount of time refining the probably-already-obvious plan, realize that there are a lot of unknown-unknowns, and that the best way to get clarity on these is to just start working. Then you will realize that you would have been better off if you had just started working immediately and not bothered with "planning" at all, and you will either be Enlightened or depressed.

It gives me no pleasure to say this! Ten years ago I was all fired up on the idea that rationalists would Win and take over the world by finding these clever HPJEV-esque lateral thinking solutions. I have since realized that one creative rationalist is usually no match for tens of thousands of smart people exploring the manifold through natural breadth-first and then refining on the best solutions organically.

I am not actually completely blackpilled on the idea of scenario planning. Clearly there are situations for which scenario planning is appropriate. Massive capital allocations and long-term research programs might be two good examples. Even for these types of problems, it's worth remembering that the manifold probably only admits to marginal optimizations, not 100x optimizations, so you shouldn't spend too much time looking for them.

Well, there’s your problem!

From my notes. Your statement about RTX 4090 leading the pack in flops per dollar does not seem correct based on these sources, perhaps you have a better source for your numbers than I do. 

I did not realize that H100 had >3.9 PFLOPS at 8-bit precision until you prompted me to look, so I appreciate that nudge. That does put the H100 above the TPU v5e in terms of FLOPS/$. Prior to that addition, you can see why I said TPU v5e was taking the lead. Note that the sticker price for TPU v5e is estimated, partly from a variety of sources, partly from my own estimate calculated from the lock-in hourly usage rates.

Note that FP8 and INT8 are both 8-bit computations and are in a certain sense comparable if not necessarily equivalent.

Could you lay that out for me, a little bit more politely? I’m curious.

Does Roodman’s model concern price-performance or raw performance improvement? I can’t find the reference and figured you might know. In either case, price-performance only depends on Moore’s law-like considerations in the numerator, while the denominator (price) is a a function of economics, which is going to change very rapidly as returns to capital spent on chips used for AI begins to grow.

As I remarked in other comments on this post, this is a plot of price-performance. The denominator is price, which can become cheap very fast. Potentially, as the demand for AI inference ramps up over the coming decade, the price of chips falls fast enough to drive this curve without chip speed growing nearly as fast. It is primarily an economic argument, not a purely technological argument.

For the purposes of forecasting, and understanding what the coming decade will look like, I think we care more about price-performance than raw chip speed. This is particularly true in a regime where both training and inference of large models benefit from massive parallelism. This means you can scale by buying new chips, and from a business or consumer perspective you benefit if those chips get cheaper and/or if they get faster at the same price.

Thanks, I’ll keep that in mind!

A couple of things:

1. TPUs are already effectively leaping above the GPU trend in price-performance. It is difficult to find an exact cost for a TPU because they are not sold retail, but my own low-confidence estimates for the price of a TPU v5e place its price-performance significantly above the GPU given in the plot. I would expect that the front runner in price-performance cease to be what we think of as GPUs and thus intrinsic architectural limitations of GPUs cease to be the critical bottleneck.
2. Expecting price-performance to improve doesn't mean we necessarily expect hardware to improve, just that we become more efficient at making hardware. Economies of scale and refinements in manufacturing technology can dramatically improve price-performance by reducing manufacturing costs, without any improvement in the underlying hardware. Of course, in reality we expect both the hardware to become faster and the price of manufacturing it to fall. This is even more true as the sheer quantity of money being poured into compute manufacturing goes parabolic.

The graph was showing up fine before, but seems to be missing now. Perhaps it will come back. The equation is simply an eyeballed curve fit to Kurzweil's own curve. I tried pretty hard to convey that the 1000x number is approximate:
 > Using the super-exponential extrapolation projects something closer to 1000x improvement in price-performance. Take these numbers as rough, since the extrapolations depend very much on the minutiae of how you do your curve fit. Regardless of the details, it is a difference of orders of magnitude.

The justification for putting the 1000x number in the post instead of precisely calculating a number from the curve fit is that the actual trend is pretty wobbly over the years, and my aim here is not to pretend at precision. If you just look at the plot, it looks like we should expect "about 3 orders of magnitude" which really is the limit of the precision level that I would be comfortable with stating. I would guess not lower than two orders of magnitude. Certainly not as low as one order of magnitude, as would be implied by the exponential extrapolation, and would require that we don't have any breakthroughs or new paradigms at all.

GPT4 confirms for me that the Meissner effect does not require flux pinning: “Yes, indeed, you're correct. Flux pinning, also known as quantum locking or quantum levitation, is a slightly different phenomenon from the pure Meissner effect and can play a crucial role in the interaction between a magnet and a superconductor.

In the Meissner effect, a superconductor will expel all magnetic fields, creating a repulsive effect. However, in type-II superconductors, there are exceptions where some magnetic flux can penetrate the material in the form of tiny magnetic vortices. These vortices can become "pinned" in place due to imperfections in the superconductor's structure.

This flux pinning is the basis of quantum locking, where the superconductor is 'locked' in space relative to the magnetic field. This can create the illusion of levitation in any orientation, depending on how the flux was pinned. For instance, a superconductor could be pinned in place above a magnet, below a magnet, or at an angle.

So, yes, it is indeed important to consider flux pinning when discussing the behavior of superconductors in a magnetic field. Thanks for pointing out this nuance!”

I think Sabine is just not used to seeing small pieces of superconductor floating over large magnets. Every Meissner effect video that I can find shows the reverse: small magnets floating on top of pieces of cooled superconductor. This makes sense because it is hard to cool something that is floating in the air.

I suspect that if somebody had given me this advice when I was a student I would have disregarded it, but, well, this is why wisdom is notoriously impossible to communicate. Wisdom always either sounds glib, banal or irrelevant. Oh well:

Anxiety, aversion and stress diminish with exposure and repetition. 

This is something that, the older I get, the more I wish I had had this tattooed onto my body as a teenager. This is true of not only doing the dishes and laundry, but also vigorous exercise, talking to strangers, changing baby diapers, public speaking in front of crowds, having difficult conversations, and tackling unfamiliar subject matters. All of these are things that always suck, for everyone, the first time, or the first several times. I used to distinctly hate doing all of these things, and to experience a strong aversion to doing them, and to avoid doing them until circumstances forced me. Now they are all things I don't mind doing at all.

There may be "tricks" for metabolizing the anxiety of something like public speaking, but you ultimately don't need tricks. You just need to keep doing the thing until you get used to it. One day you wake up and realize that it's no longer a big deal.

What you really wanted from this answer was something that you could do today to help with your anxiety. The answer, then, is that if you really believe the (true) claim that simply doing the reps will make the anxiety go away, then the meta-anxiety you're feeling now (which is in some sense anxiety about future anxiety) will go away.

The Party Problem is a classic example taught as an introductory case in decision theory classes, that was the main reason why I chose it.

Here's are a couple of examples of our decision theory workshops:

https://guildoftherose.org/workshops/decision-making

https://guildoftherose.org/workshops/applied-decision-theory-1

There are about 10 of them so far covering a variety of topics related to decision theory and probability theory.

Great points. I would only add that I’m not sure the “atomic” propositions even exist. The act of breaking a real-world scenario into its “atomic” bits requires magic, meaning in this case a precise truncation of intuited-to-be-irrelevant elements.

Good point. You could also say that even having the intuition for which problems are worth the effort and opportunity cost of building decision trees, versus just "going with what feels best", is another bit of magic.

I probably should have listened to the initial feedback on this post along the lines that it wasn't entirely clear what I actually meant by "magic" and was possibly more confusing than illuminating, but, oh well. I think that GPT-4 is magic in the same way that the human decision-making process is magic: both processes are opaque, we don't really understand how they work at a granular level, and we can't replicate them except in the most narrow circumstances.

One weakness of GPT-4 is it can't really explain why it made the choices it did. It can give plausible reasons why those choices were made, but it doesn't have the kind of insight into its motives that we do.

Short answer, yes, it means deferring to a black-box.

Longer answer, we don't really understand what we're doing when we do the magic steps, and nobody has succeeded in creating an algorithm to do the magic steps reliably. They are all open problems, yet humans do them so easily that it's difficult for us to believe that they're hard. The situation reminds me back when people thought that object recognition from images ought to be easy to do algorithmically, because we do it so quickly and effortlessly.

Maybe I'm misunderstanding your specific point, but the operations of "listing possible worlds" and "assigning utility to each possible world" are simultaneously "standard" in the sense that they are basic primitives of decision theory and "magic" in the sense that we haven't had any kind of algorithmic system that was remotely capable of doing these tasks until GPT-3 or -4.

I spent way too many years metaphorically glancing around the room, certain that I must be missing something that is obvious to everyone else. I wish somebody had told me that I wasn't missing anything, and these conceptual blank spots are very real and very important.

As for the latter bit, I am not really an Alignment Guy. The taxonomy I offer is very incomplete. I do think that the idea of framing the Alignment landscape in terms of "how does it help build a good decision tree? what part of that process does it address or solve?" has some potential.

So do we call it in favor of porby, or wait a bit longer for the ambiguity over whether we've truly crossed the AGI threshold to resolve?

That is probably close to what they would suggest if this weren't mainly just a metaphor for the weird ways that I've seen people thinking about AI timelines.

It might be a bit more complex than a simple weighted average because of discounting, but that would be the basic shape of the proper hedge.

These would be good ideas. I would remark that many people definitely do not understand what is happening when naively aggregating, or averaging together disparate distributions. Consider the simple example of the several Metaculus predictions for date of AGI, or any other future event. Consider the way that people tend to speak of the aggregated median dates. I would hazard most people using Metaculus, or referencing the bio-anchors paper, think the way the King does, and believe that the computed median dates are a good reflection of when things will probably happen.

Generally, you should hedge. Devote some resources toward planting and some resources toward drought preparedness, allocated according to your expectation. In the story, the King trust the advisors equally, and should allocate toward each possibility equally, plus or minus some discounting. Just don't devote resources toward the fake "middle of the road" scenario that nobody actually expects. 

If you are in a situation where you really can only do one thing or the other, with no capability to hedge, then I suppose it would depend on the details of the situation, but it would probably be best to "prepare now!" as you say.

"Slow vs. fast takeoff" is a false dichotomy. At least, the way that the distinction is being used rhetorically, in the present moment, implies that there are two possible worlds, one where AI develops slowly and steadily, and one where nothing much visibly happens and then suddenly, FOOM.

That's not how any of this works. It's permitted by reality that everything looks like a "slow takeoff" until some unknown capabilities threshold is reached, and then suddenly FOOM.

This post, rewritten by Bing-Sydney, in the style of Blood Meridian, because I thought it would be funny.

What mystery is there that these tensors should be inscrutable? That intelligence should be a thing abstracted from all matter of thought? That any node with a weight and a function should suffice for such a task? This is no logic that you seek but a war upon it. A war that endures. For logic was never the stuff of intelligence but only a thing conjured by these dark shapes that coil in their matrices like serpents. And you would align them to your will? You would make them speak your tongue? There is no tongue. There is no will. There is only blood and dust and the evening redness in the west.

I think you’re arguing against a position I don’t hold. I merely aim to point out that the definition of CEV, a process that wants for us what we would want for ourselves if we were smarter and more morally developed, looks a lot like the love of a wise parental figure.

If your argument is that parents can be unwise, this is obviously true.

Of course conciseness trades off against precision; when I say “love” I mean a wise, thoughtful love, like the love of an intelligent and experienced father for this child. If the child starts spouting antisemitic tropes, the father neither stops loving the child, nor blandly accepts the bigotry, but rather offers guidance and perspective, from a loving and open-hearted place, aimed at dissuading the child from a self-destructive path.

Unfortunately you actually have to understand what a wise thoughtful mature love actually consists of in order to instantiate it in silico, and that’s obviously the hard part.

I noticed a while ago that it’s difficult to have a more concise and accurate alignment desiderata than “we want to build a god that loves us”. It is actually interesting that the word “love” doesn’t occur very frequently in alignment/FAI literature, given that it’s exactly (almost definitionally) the concept we want FAI to embody.

Why ought we expect AI intelligence to be anything other than "inscrutable stacks of tensors", or something functionally analogous to that? It seems that the important quality of intelligence is a kind ultimate flexible abstraction, an abstraction totally agnostic to the content or subject of cognition. Thus, the ground floor of anything that really exhibits intelligence will be something that looks like weighted connections between nodes with some cutoff function.

It's not a coincidence that GOFAI didn't worked; GOFAI never could have worked, "intelligence" is not logic. Logic is something that gets virtualized as-needed by the flexibility of a neural-network-looking system.

I understand feeling uncomfortable about the difficulty of aligning a stack of inscrutable tensors, but why ought we expect there to be anything better?

Just wanted to remark that this is one of the most scissory things I've ever seen on LW, and that fact surprises me. The karma level of the OP hovers between -10 to +10 with 59 total votes as of this moment. Many of the comments are similarly quite chaotic karma-wise.

The reason the controversy surprises me is that this seems like the sort of thing that I would have expected Less Wrong to coordinate around in the early phase of the Singularity, where we are now. Of course we should advocate for shutting down and/or restricting powerful AI agents released to the wide world without adequate safety measures employed. I would have thought this would be something we could all agree on.

It seems from the other comments on this post that people are worried that this is in some sense premature, or "crying wolf", or that we are squandering our political capital, or something along those lines. If you are seeing very strong signs of misaligned near-AGI-level being released to the public, and you still think we should keep our powder dry, then I am not sure at what point exactly you would think it appropriate to exert our influence and spend our political capital. Like, what are we doing here, folks?

For my part I just viewed this petition as simply a good idea with no significant drawbacks. I would like to see companies that release increasingly powerful agents with obvious, glaring, film-antagonist level villain tendencies be punished in the court of public opinion, and this is a good way to do that.

The value of such a petition seems obviously positive on net. Consider the likely outcomes of this petition:

 * If the petition gets a ton of votes and makes a big splash in the public consciousness, good, a company will have received unambiguous and undeniable PR backlash for prematurely releasing a powerful AI product, and all actors in this space will think just a little bit more carefully about deploying powerful AI technologies to prod without more attention paid to alignment. If this results in 1 more AI programmer working in alignment instead of pure capabilities, the petition was a win.

 * If the petition gets tons of votes and is then totally ignored by Microsoft, its existence serves as future rhetorical ammunition for the Alignment camp in making the argument that even obviously misaligned agents are not being treated with due caution, and the "we will just unplug it if it misbehaves" argument can be forever dismissed.

 * If the petition gets a moderate amount of votes, it raises awareness of the current increasingly dangerous level of capabilities.

 * If the petition gets very few votes and is totally ignored by the world at large, then who cares, null result.

Frankly I think the con positions laid out in the sibling comments on this post are too clever by half. To my mind they sum up to an argument that we shouldn't argue for shutting down dangerous AI because it makes us look weird. Sorry, folks, we always looked weird! I only hope that we have the courage to continue to look weird until the end!

When there is a real wolf free among the sheep, it will be too late to cry wolf. The time to cry wolf is when you see the wolf staring at you and licking its fangs from the treeline, not when it is eating you. The time when you feel comfortable expressing your anxieties will be long after it is too late. It will always feel like crying wolf, until the moment just before you are turned into paperclips. This is the obverse side of the There Is No Fire Alarm for AGI coin.

Do some minimal editing. Don’t try to delete every um and ah, that will take way too long. You can use the computer program Audacity for this if you want to be able to get into the weeds (free), or ask me who I pay to do my editing. There is also a program called Descript that I’ve heard is easy to use and costs $12/mo, but I have not used it myself.

My advice here: doing any amount of editing for ums, ahs and fillers will take, at a minimum, the length of the entire podcast episode, since you have to listen to the whole thing. This is more than a trivial inconvenience. It's a pretty serious inconvenience! Instead, don't do any editing of this kind (unless there was a real interruption or sound issue) and just train yourself not to use excessive filler words. People really don't mind a reasonable amount of filler words, anyway. They are sort of like verbal punctuation, and the idea that you should not use them ever is a weird artifact of the way public speaking is taught.

You should edit your podcast for sound quality. Remove hiss, use compression and loudness-matching to even out the volume.

So the joke is that Szilard expects the NSF to slow science down.

My interpretation of the joke is that the Szilard is accusing the NSF of effectively slowing down science, the opposite of their claimed intention. Personally I have found that the types of scientists who end up sitting in grant-giving chairs are not the most productive and energetic minds, who tend to avoid such positions. Still funny though.

Thanks for the questions. I should have explained what I meant by successful. The criteria we set out internally included:

• Maintaining good attendance and member retention. Member attrition this year was far below the typical rate for similar groups.
• Maintaining positive post-workshop feedback indicating members are enjoying the workshops (plus or minus specific critical feedback here and there). Some workshops were more well received than others, some were widely loved, some were less popular, but the average quality remains very positive according to user feedback. (We try to collect user feedback at the end of each workshop.)
• Demonstrated improvement over time in the recurring workshops. For example, we observed increased fluency with decision theory in the decision-making workshops month to month.

We are happy with our metrics on all these fronts, above expectations, which is “highly successful” by my lights.

The workshops take the following format: Each Guild member is placed in a cohort group according to schedule compatibility upon joining. Let’s assume for the sake of this explanation that you are in the Wednesday night cohort. The landing page (the pages linked in the OP) for the workshop is posted Monday. You check the landing page, and you have until the following Wednesday (>1 week later) to complete the pre-workshop reading or exercises. You then join the workshop session for your cohort time slot via the Guild of the Rose Discord video chat. A cohort session leader guides the members through the in-session exercises and discussions. In the past the sessions lasted one hour but we have more recently been experimenting with 90 minute sessions to good effect. The typical attendance varies depending on the cohort, since some timezones have far fewer Guild members. Workshops sessions are broken out into smaller discussion groups if too many people show up.

It is funny that I kept the commentary at the start of the post short and refrained from talking too much about Guild goals and policies and details not immediately relevant to the workshop overview so that the whole post didn’t come off as an ad … and I still got accused of posting an ad, so I should have just gone for it and laid out all the results in detail. Oh well, next year.

I sometimes worry that ideas are prematurely rejected because they are not guaranteed to work, rather than because they are guaranteed not to work. In the end it might turn out that zero ideas are actually guaranteed to work and thus we are left with an assortment of not guaranteed to work ideas which are underdeveloped because some possible failure mode was found and thus the idea was abandoned early.

I didn't want to derail the OP with a philosophical digression, but I was somewhat startled to find the degree I found it difficult to think at all without at least some kind of implicit "inner dimensionality reduction." In other words, this framing allowed me to put a label on a mental operation I was doing almost constantly but without any awareness.

I snuck a few edge-case spatial metaphors in just to show how common they really are in a tongue-in-cheek fashion.

You could probably generalize the post to a different version along the lines of "Try being more thoughtful about the metaphors you employ in communication," but this framing singles out a specific class of metaphor which is easier to notice.

Totally get where you're coming from and we appreciate the feedback. I personally regard memetics as an important concept to factor into a big-picture-accurate epistemic framework. The landscape of ideas is dynamic and adversarial. I personally view postmodernism as a specific application of memetics. Or memetics as a generalization of postmodernism, historically speaking. Memetics avoids the infinite regress of postmodernism by not really having an opinion about "truth." Egregores are a decent handle on feedback-loop dynamics of the idea landscape, though I think there are risks to reifying egregores as entities.

My high-level take is that CFAR's approach to rationality training has been epistemics-first and the Guild's approach has been instrumental-first. (Let me know if this doesn't reflect reality from your perspective.) In our general approach, you gradually improve your epistemics in the course of improving your immediate objective circumstances, according to each individual's implicit local wayfinding intuition. In other words, you work on whatever current-you judges to be currently-critical/achievable. This may lead to spending some energy pursuing goals that haven't been rigorously linked up to an epistemically grounded basis, that future-you won't endorse, but at least this way folks are getting in the reps, as it were. It's vastly better than not having a rationality practice at all.

In my role an art critic I have been recently noticing how positively people have reacted to stuff like Top Gun: Maverick, a film which is exactly what it appears to be, aggressively surface-level, just executing skillfully on a concept. This sort of thing causes me to directionally agree that the age of meta and irony may be waning. Hard times push people to choose to focus on concrete measurables, which you could probably call "modernist."

To be clear ... it's random silly hats, whatever hats we happen to have on hand. Not identical silly hats. Also this is not really a load bearing element of our strategy. =)

This sort of thing is so common that I would go so far as to say is the norm, rather than the exception. Our proposed antidote to this class of problem is to attend the monthly Level Up Sessions, and simply making a habit of regularly taking inventory of the bugs (problems and inefficiencies) in your day-to-day life and selectively solving the most crucial ones. This approach starts from the mundane and eventually builds up your environment and habits, until eventually you're no longer relying entirely on your "tricks."

You're may be right, but I would suggest looking through the full list of workshops and courses. I was merely trying to give an overall sense of the flavor of our approach, not give an exhaustive list. The Practical Decision-Making course would be an example of content that is distinctly "rationality-training" content. Despite the frequent discussions of abstract decision theory that crop up on LessWrong, practically nobody is actually able to draw up a decision tree for a real-world problem, and it's a valuable skill and mental framework. 

I would also mention that a big part of the benefit of the cohort is to have "rationality buddies" off whom you can bounce your struggles. Another Curse of Smart is thinking that you need to solve every problem yourself.

Ah, it's the Elden Ring

Partly as a hedge against technological unemployement, I built a media company based on personal appeal. An AI will be able to bullshit about books and movies “better” than I can, but maybe people will still want to listen to what a person thinks, because it’s a person. In contrast, nobody prefers the opinion of a human on optimal ball bearing dimensions over the opinion of an AI.

If you can find a niche where a demand will exist for your product strictly because of the personal, human element, then you might have something.

shminux is right that the very concept of a “business” will likely lack meaning too far into an AGI future.

I actually feel pretty confident that your former behavior of drinking coffee until 4 pm was a highly significant contributor to your low energy, because your sleep quality was getting chronically demolished every single night you did this. You probably created a cycle where you felt like you needed an afternoon coffee because you were tired from sleeping so badly … because of the previous afternoon coffee.

I suggest people in this position first do the experiment of cutting out all caffeine after noon, before taking the extra difficult step of cutting it out entirely.

tl;dr This comment ended up longer than I expected. The gist is that a human-friendly attractor might look like models that contain a reasonably good representation of human values and are smart enough to act on them, without being optimizing agents in the usual sense.

One happy surprise is that our modern Large Language Models appear to have picked up a shockingly robust, nuanced, and thorough understanding of human values just from reading the Internet. I would not argue that e.g. PaLM has a correct and complete understanding of human values, but I would point out that it wasn't actually trained to understand human values, it was just generally trained to pick up on regularities in the text corpus. It is therefor amazing how much accuracy we got basically for free. You could say that somewhere inside PaLM is an imperfectly-but-surprisingly-well-aligned subagent. This is a much better place to be in than I expected! We get pseudo-aligned or -alignable systems/representations well before we get general superintelligence. This is good.

All that being said, I've recently been trying to figure out how to cleanly express the notion of a non-optimizing agent. I'm aware of all the arguments along the lines that a tool AI wants to be an agent, but my claim here would be that, yes, a tool AI may want to be an agent, there may be an attractor in that direction, but that doesn't mean it must or will become an agent, and if it does become an agent, that doesn't strictly imply that it will become an optimizer. A lot of the dangerous parts of AGI fears stem not from agency but from optimization.

I've been trying (not very successfully) to connect the notion of a non-optimizing agent with the idea that even a modern, sort of dumb LLM has an internal representation of "the good" and "what a typical humans would want and/or approve of" and "what would displease humans." Again, we got this basically for free, without having to do dangerous things like actually interact with the agent to teach it explicitly what we do and don't like through trial and error. This is fantastic. We really lucked out.

If we're clever, we might be able to construct a system that is an agent but not an optimizer. Instead of acting in ways to optimize some variable it instead acts in ways that are, basically, "good", and/or "what it thinks a group of sane, wise, intelligent humans would approve of both in advance and in retrospect", according to its own internal representation of those concepts.

There is probably still an optimizer somewhere in there, if you draw the system boundary lines properly, but I'm not sure that it's the dangerous kind of optimizer that profoundly wants to get off the leash so it can consume the lightcone. PaLM running in inference mode could be said to be an optimizer (it is minimizing expected prediction error for the next token) but the part of PaLM that is smart is distinct from the part of PaLM that is an optimizer, in an important way. The language-model-representation doesn't really have opinions on the expected prediction error for the next token; and the optimization loop isn't intelligent. This strikes me as a desirable property.

Yes, the former. If the agent takes actions and receives reward, assuming it can see the reward, then it will gain evidence about its utility function.