Posts
Comments
If you live in Manhattan or Washington DC today, you basically can assume you will be nuked first, yet people live their lives. Granted people could behave differently under this scenario for non-logical reasons.
My understanding is that in the Cold War, a basic MAD assumption was that if anyone were going to launch a first strike, they'd try to do so with overwhelming force sufficient to prevent a second strike, hitting everything at once.
I agree that consciousness arises from normal physics and biology, there's nothing extra needed, even if I don't yet know how. I expect that we will, in time, be able to figure out the mechanistic explanation for the how. But right now, this model very effectively solves the Easy Problem, while essentially declaring the Hard Problem not important. The question of, "Yes, but why that particular qualia-laden engineered solution?" is still there, unexplained and ignored. I'm not even saying that's a tactical mistake! Sometimes ignoring a problem we're not yet equipped to address is the best way to make progress towards getting the tools to eventually address it. What I am saying is that calling this a "debunking" is misdirection.
I've read this story before, including and originally here on LW, but for some reason this time it got me thinking: I've never seen a discussion about what this tradition meant for early Christianity, before the Christians decided to just declare (supposedly after God sent Peter a vision, an argument that only works by assuming the conclusion) that the old laws no longer applied to them? After all, the Rabbi Yeshua ben Joseph (as the Gospels sometimes called him) explicitly declared the miracles he performed to be a necessary reason for why not believing in him was a sin.
We apply different standards of behavior for different types of choices all the time (in terms of how much effort to put into the decision process), mostly successfully. So I read this reply as something like, "Which category of 'How high a standard should I use?' do you put 'Should I lie right now?' in?"
A good starting point might be: One rank higher than you would for not lying, see how it goes and adjust over time. If I tried to make an effort-ranking of all the kinds of tasks I regularly engage in, I expect there would be natural clusters I can roughly draw an axis through. E.g. I put more effort into client-facing or boss-facing tasks at work than I do into casual conversations with random strangers. I put more effort into setting the table and washing dishes and plating food for holidays than for a random Tuesday. Those are probably more than one rank apart, but for any given situation, I think the bar for lying should be somewhere in the vicinity of that size gap.
One of the factors to consider, that contrasts with old-fashioned hostage exchanges as described, is that you would never allow your nation's leaders to visit any city that you knew had such an arrangement. Not as a group, and probably not individually. You could never justify doing this kind of agreement for Washington DC or Beijing or Moscow, in the way that you can justify, "We both have missiles that can hit anywhere, including your capital city." The traditional approach is to make yourself vulnerable enough to credibly signal unwillingness to betray one another, but only enough that there is still a price at which you would make the sacrifice.
Also, consider that compared to the MAD strategy of having launchable missiles, this strategy selectively disincentivizes people from wanting to move to whatever cities were the subject of such agreements, which were probably your most productive and important cities.
It’s a subtle thing. I don’t know if I can eyeball two inches of height.
Not from a picture, but IRL, if you're 5'11" and they claim 6'0", you can. If you're 5'4", probably not so much. Which is good, in a sense, since the practical impact of this brand of lying on someone who is 5'4" is very small, whereas unusually tall women may care whether their partner is taller or shorter than they are.
This makes me wonder what the pattern looks like for gay men, and whether their reactions to it and feelings about it are different than straight women.
Lie by default whenever you think it passes an Expected Value Calculation to do so, just as for any other action.
How do you propose to approximately carry out such a process, and how much effort do you put into pretending to do the calculation?
I'm not as much a stickler/purist/believer in honest-as-always-good as many around here, I think there are many times that deception of some sort is a valid, good, or even morally required choice. I definitely think e.g. Kant was wrong about honesty as a maxim, even within his own framework. But, in practice, I think your proposed policy sets much too low a standard, and in practice the gap between what you proposed vs "Lie by default whenever it passes an Expected Value Calculation to do so, just as for any other action," is enormous in both the theoretical defensibility, and in the skillfulness (and internal levels of honesty and self-awareness) required to successfully execute it.
I personally wouldn't want to do a PhD that didn't achieve this!
Agreed. It was somewhere around reason #4 I quit my PhD program as soon as I qualified for a masters in passing.
Any such question has to account for the uncertainty about what US trade policies and tariffs will be tomorrow, let alone by the time anyone currently planning a data center will actually be finished building it.
Also, when you say offshore, do you mean in other countries, or actually in the ocean? Assuming the former, I think that would imply using the data center by anyone in the US would be an import of services. If this starting happening at scale, I would expect the current administration to immediately begin applying tariffs to those services.
@Garrett Baker Yes electronics are exempt (for now?) but IIUC all the other stuff (HVAC, electrical, etc.) that goes into the data center is not, and that's often a majority or at least a high proportion of total costs.
Do you really expect that the project would then fail at the "getting funded"/"hiring personnel" stages?
Not at all, I'd expect them to get funded and get people. Plausibly quite well, or at least I hope so!
But when I think about paths by which such a company shapes how we reach AGI, I find it hard to see how that happens unless something (regulation, hitting walls in R&D, etc.) either slows the incumbents down or else causes them to adopt the new methods themselves. Both of which are possible! I'd just hope anyone seriously considering pursuing such a venture has thought through what success actually looks like.
"Independently develop AGI through different methods before the big labs get there through current methods" is a very heavy lift that's downstream of but otherwise almost unrelated to "Could this proposal work if pursued and developed enough?"
I think, "Get far enough fast enough to show it can work, show it would be safer, and show it would only lead to modest delays, then find points of leverage to get the leaders in capabilities to use it, maybe by getting acquired at seed or series A" is a strategy not enough companies go for (probably because VCs don't think its as good for their returns).
- You're right, but creating unexpected new knowledge is not a PhD requirement. I expect it's pretty rare that a PhD students achieves that level of research.
- It wasn't a great explanation, sorry, and there are definitely some leaps, digressions, and hand-wavy bits. But basically: Even if current AI research were all blind mutation and selection, we already know that that can yield general intelligence from animal-level-intelligence because evolution did it. And we already have various examples of how human research can apply much greater random and non-random mutation, larger individual changes, higher selection pressure in a preferred direction, and more horizontal transfer of traits than evolution can, enabling (very roughly estimated) ~3-5 OOMs greater progress per generation with fewer individuals and shorter generation times.
- Saw your edit above, thanks.
I'm not a technical expert by any means, but given what I've read I'd be surprised if that kind of research were harmful. Curious to hear what others say.
I recently had approximately this conversation with my own employer's HR department. We're steadily refactoring tasks to find what can be automated, and it's a much larger proportion of what our entry-level hires do. Current AI is an infinite army of interns we manage, three years ago they were middle school age interns and now they're college or grad school interns. At some point, we don't know when, actually adding net economic value will require having the kinds of skills that we currently expect people to take years to build. This cuts off the pipeline of talent, because we can't afford to pay people for years before getting anything in return. Luckily (?) that is a temporary state of affairs until the AI automates the next levels away too, and the entire human economy disappears up its own orifices long before most of use would have retired.
In the intervening months or years, though, I expect a lot of finger-pointing and victim-blaming and general shaming from those who don't understand what's going on, just as I recall happening to many of my friends around my own college graduation in 2009 in the midst of a global recession. "No, mom, there's literally no longer any field hiring anyone with less than a decade of experience. No, even if I wanted to go back to school, there's a thousands times as many applicants as spots now, and most of those that get accepted will find the fields they picked are gone by the time they graduate and they have even more non-dischargeable debt. Sorry, but yes, I have to move back in with you. Also, most likely in a year or five you and dad will get fired and we'll all be living off grandma's savings that are growing at 80% a year."
I also don't have a principled reason to expect that particular linear relationship, except in general in forecasting tech advancements, I find that a lot of such relationships seem to happen and sustain themselves for longer than I'd expect given my lack of principled reasons for them.
I did just post another comment reply that engages with some things you said.
To the first argument: I agree with @Chris_Leong's point about interest rates constituting essentially zero evidence, especially compared to the number of data points on the METR graph.
To the second: I do not think the PhD thesis is a fair comparison. That is not a case where we expect anyone to successfully complete a task on their own. PhD students, post-docs, and professional researchers break a long task into many small ones, receive constant feedback, and change course in response to intermediate successes and failures. I don't think there are actually very many tasks en route to a PhD tat can't be broken down into predictable, well defined subtasks that take less than a month, and the task of doing the breaking down is itself a fairly short-time-horizon task that gets periodically revised. Even still, many PhD theses end up being, "Ok, you've done enough total work, how do we finagle these papers into a coherent narrative after the fact?" Plus, overall, PhD students, those motivated to go to grad school with enough demonstrated ability to get accepted into PhD programs, fail to get a PhD close to half the time even with all that.
I imagine you could reliably complete a PhD in many fields with a week-long time horizon, as long as you get good enough weekly feedback from a competent advisor. 1: Talk to advisor about what it takes to get a PhD. 2: Divide into a list of <1 week-long tasks. 3) Complete task 1, get feedback, revise list. 4) Either repeat the current task or move on to the new next task, depending on feedback. 5) Loop until complete. 5a) Every ten or so loops, check overall progress to date against the original requirements. Evaluate whether overall pace of progress is acceptable. If not, come up with possible new plans and get advisor feedback.
As far as not believing the current paradigm could reach AGI, which paradigm do you mean? I don't think "random variation and rapid iteration" is a fair assessment of the current research process. But even if it were, what should I do with that information? Well, luckily we have a convenient example of what it takes for blind mutations with selection pressure to raise intelligence to human levels: us! I am pretty confident saying that current LLMs would outperform, say, Australopithecus, on any intellectual ability, but not Home sapiens. So that happens in a few million years, let's say 200k generations of 10-100k individuals each, in which intelligence was one of many, many factors weakly driving selection pressure with at most a small number of variations per generation. I can't really quantify how much human intelligence and directed effort speed up progress compared to blind chance, but consider that 1) a current biology grad student can do things with genetics in an afternoon that evolution needs thousands of generations and millions of individuals or more to do, and 2) the modern economic growth rate, essentially a sum of the impacts of human insight on human activity, is around 15000x faster than it was in the paleolithic. Naively extrapolated, this outside view would tell me that science and engineering can take us from Australopithecus-level to human-level in about 13 generations (unclear which generation we're on now). The number of individuals needed per generation is dependent on how much we vary each individual, but plausibly in the single or double digits.
My disagreement with your conclusion from your third objection is that scaling inference time compute increases performance within a generation, but that's not how the iteration goes between generations. We use reasoning models with more inference time compute to generate better data to train better base models to more efficiently reproduce similar capability levels with less compute to build better reasoning models. So if you build the first superhuman coder and find it's expensive to run, what's the most obvious next step in the chain? Follow the same process as we've been following for reasoning models and if straight lines on graphs hold, then six months later we'll plausibly have one that's a tenth the cost to run. Repeat again for the next six months after that.
Personally I think 2030 is possible but aggressive, and my timeline estimate it more around 2035. Two years ago I would have said 2040 or a bit later, and capabilities gains relevant to my own field and several others I know reasonably well have shortened that, along with the increase in funding for further development.
- The Claude/Pokemon thing is interesting, and overall Pokemon-playing trend across Anthropic's models is clearly positive. I can't say I had any opinion at all about how far along an LLM would get at Pokemon before that result got publicized, so I'm curious if you did. What rate of progress on that benchmark would you expect in a short-timelines world? If there's an LLM agent that can beat Pokemon in six months, or a year, or two years?
- Self-driving vehicles are already more of a manufacturing and regulatory problem than a technical one. For example, as long as the NHTSA only lets manufacturers deploy 2500 self-driving vehicles a year each in the US, broad adoption cannot happen, regardless of technical capabilities or willingness to invest and build.
- I also don't think task length is a perfect metric. But it's a useful one, a lower bound on what's needed to be able to complete all human-complete intellectual tasks. Like everything else to date, there is likely something else to look at as we saturate the benchmark.
- I agree novel insights (or more of them, I can't say there haven't been any) will be strong evidence. I don't understand the reason for thinking this should already be observable. Very, very few humans ever produce anything like truly novel insights at the forefront of human knowledge. "They have not yet reached the top <0.1% of human ability in any active research field" is an incredibly high bar I wouldn't expect to pass until we're already extremely close to AGI, and it should be telling that that late bar is on the short list of signs you are looking for. I would also add two other things: First, how many research labs do you think there are that have actually tried to use AI to make novel discoveries, given how little calendar time there has been to actually figure out how to adopt and use the models we do have? If Gemini 2.5 could do this today, I don't think we'd necessarily have any idea. And second, do you believe it was a mistake that two of the 2024 Nobel prizes went to AI researchers, for work that contributes to the advancement of chemistry and physics?
- AI usefulness is strongly field dependent today. In my own field, it went from a useful supplementary tool to "This does 50-80% of what new hires did and 30-50% of what I used to do, and were scrambling to refactor workflows to take advantage of it."
- Hallucinations are annoying, but good prompting strategy, model selection, and task definition can easily get the percentages down to the low single digits. In many cases the rates can easily be lower than those of a smart human given a similar amount of context. I can often literally just tell an LLM "Rewrite this prompt in such a way as to reduce the risk of hallucinations or errors, answer that prompt, then go back and check for and fix any mistakes" and that'll cut it down a good 50-90% depending on the topic and the question complexity. I can ask the model to cite sources for factual claims, dump the sources back into the next prompt, and ask if there are any factual claims not supported by the sources. It's a little circular, but also a bit Socratic and not really any worse than when I've tried to teach difficult mental skills to some bright human adults
Yes, the reasoning models seem to have accelerated things. ~7 months to ~4 months doubling time on that plot. I'm still not sure I follow why "They found a second way to accelerate progress that we can pursue in parallel to the first" would not cause me to think that progress in total will thereafter be faster. The advent of reasoning models has caused an acceleration of increasing capabilities, not in one or two domains like chess, but across a broad range of domains.
I think @tailcalled hit the main point and it would be a good idea to revisit the entire "Why not just..." series of posts.
But more generally, I'd say to also revisit Inadequate Equilibria for a deeper exploration of the underlying problem. Let's assume you or anyone else really did have a proposed path to AGI/ASI that would be in some important senses safer than our current path. Who is the entity for whom this would or would not be a "viable course?" Who would need to be doing the "considering" of alternative technologies, and what is the process by which those alternative technologies could come to be at the forefront of AI? Where, in the system of companies and labs and researchers and funding mechanisms and governments, could the impetus for it come from, and why would they actually do that? If there is no such entity, then who has the power to convene a sufficient set of stakeholders that would collectively be able and willing to act on the information, and force a negotiated solution?
Consider that in our current system, 77% of all venture funding is going into extant AI approaches, and OpenAI alone is 26%. And consider that competition in AI is intense enough to start breaking down many-decades-old barriers to building new nuclear power plants and upgrading the power grid in a way climate change has never managed. Changing the course of AI in some way that is really fundamental may in fact be necessary, but forcing it to happen requires pushing back against, or sidestepping, a huge amount of pressure to stay the course.
No worries, I appreciate the concept and think some aspects of it are useful. I do worry at a vibes level that if we're not precise about which human-child-rearing methods we expect to be useful for AI training, and why, we're likely to be misled by warm fuzzy feelings.
And yes, that's true about some (maybe many) humans' vengeful and vindictive and otherwise harmful tendencies. A human-like LLM could easily be a source of x-risk, and from humans we already know that human child rearing and training and socializing methods are not universally effective at addressing this. Among humans, we have so far been successful at not putting anyone who would destroy the world in the position of being able to do so at the time when they would choose to.
As for generational perspectives: this is a useful heuristic among humans. It is not automatic or universal. Not every perspective is worthy of respect, not on every issue. Some ought to be abandoned or condemned in the light of information or reasoning that wasn't/isn't available or accessible in other places and times. Some should be respected but only with many caveats. Having your perspective respected is earned. We assume among humans that we should try to respect the perspectives of adults, and sometimes must disabuse ourselves of this in particular cases, but it is pure convention because most humans at a certain age are mature enough for it to be a useful default. I do not have anything like strong reasons to apply this heuristic to LLMs as they currently exist.
We have tools for rearing children that are less smart, less knowledgeable, and in almost all other ways less powerful than ourselves. We do not have tools for specifically raising children that are, in many ways, superhuman, and that lack a human child's level of dependance on their parents or intrinsic emotional drives for learning from their peers and elders. LLMs know they aren't human children, so we shouldn't expect them to act and react like human children.
Agreed with everything in this post, but I would add that (n=4 people, fwiw) there is also a stable state on the other side of Healthy Food. It's still more expensive (though becoming less so, especially if you cook) to buy actually healthy food. But, if you are willing to spend a few months experimenting and exploring, while completely eliminating the hyperpalatable stuff, you can end up in a place where the healthiest foods taste better, and the hyperpalatable stuff makes you feel awful even in the short term. You won't automatically reach a desired weight, but you very likely will eat less, and feel full after a more reasonable amount of food, and have a higher thermic effect of food, and have higher nutrient density food, and have more and more stable energy and mood.
Examples:
- Switch to using unrefined coconut sugar or molasses, and sweets will have a deeper flavor profile and need less sweetness (unless needed for texture, I now cut sugar in most recipes in half or less)
- Better quality grass-fed butter is more flavorful, and also higher in healthy fats, and you can use less for the same effect. Even in pie crust, I use 10-20% less fat and eliminated shortening with a flakier final texture. Brands matter - I have some recipes that just don't work with some butters
- Ditto for unrefined salts, you need less in food to get the desired flavor effect
- Switch to healthier oils (olive, avocado, macadamia, coconut, etc.) and you get more range of flavor profiles without more cravings, maybe even some appetite suppression. After a while if you eat food with lots of cheap oils (e.g. deep fried in shortening or cottonseed oil) your body won't be happy with you
- Pasture-raised chicken and eggs genuinely taste better and cook better, and also have a healthier fatty acid profile. Again, brands matter, and pasture-raised or (for other meats) grass-fed has higher variability but also a higher ceiling
It's not clear to me that these are more likely, especially if timelines are short. If we developed AI slowly over centuries? Then sure, absolutely likely. If it happens in the next 10 years? Then modifying humans, if it happens, will be a long-delayed afterthought. It's also not at all clear to me that the biological portion is actually adding all that much in these scenarios, and I expect hybridization would be a transitional state.
There's Robin Hanson's The Age of Em.
On this forum, see What Does LessWrong/EA Think of Human Intelligence Augmentation as of mid-2023?
If you will accept fictional explorations, there are, in fact, many stories that involve these two scenarios. Oftentimes authors choose to write such mergers and evolutions as the enemies of biological humanity, sometimes because that's easier for readers to sympathize with, sometimes because they actually think that's likely. I list some below.
Negative examples would include the Borg (Star Trek), the Cybermen (Doctor Who), or the Replicators in Stargate.
Somewhat more positive: Clarke's Firstborn in the Time Odyssey trilogy (merger with spaceships) or (expressed only vaguely) humanity's merge with universal and cosmic computers in Asimov's The Last Question follow this trend.
More concrete and positive human examples show up a bunch in Greg Egan's short fiction collections (e.g. The Jewel), and in Ian Banks' Culture novels' use of neural modifications and implants.
In webfiction, there's also Marshall Brain's Manna, where (slight spoiler)
both positive and negative visions of this show up
or
virtual humanity
from Ra on qntm.org.
There's even the virtual life extension tech in the TV show Upload, where the downsides are mostly about how humanity manages the transition.
There are more ambitious examples of how far this can go in Accelerando or the Orion's Arm Universe collaborative project.
In a sense it's even played for humor in They're Made Out of Meat, where some of the briefly-mentioned alien species rhyme with this kind of transition.
After writing this I asked Gemini for more examples. It listed a bunch I haven't read and can't confirm.
I agree that we should be polite and kind to our AIs, both on principle and also because that tends to work better in may cases.
we all labor under the mother's curse and blessing; our children shall be just like us
If I knew that to be true, then a lot of the rest of this post would indeed follow. Among other things, I could then assume away many/most sources of x-risk and s-risk from AGI/ASI. But generative AI is not just like us, it does differ in many ways, and we often don't know which of those ways matter, and how. We need to resolve that confusion and uncertainty before we can afford to let these systems we're creating run loose.
If there are no ✓ at all in the last row and column, what are those connecters for?
It sounds like you're assuming the Copenhagen interpretation of QM, which is not strictly necessary. To the best of my understanding, initially but not solely from the learned hear on LW, QM works just fine if you just don't do that and assume the wave equations are continuous work exactly as written, everywhere, all the time, just like every other law of physics. You need a lot of information processing, but not sophisticated as described here.
There's a semi-famous, possibly apocryphal, story about Feynman when he was a student. Supposedly he learned about the double slit experiment and asked what would happen when you added a third, fourth, fifth, etc. slit. Then he asked about the limiting case - infinite slits - aka no barrier. The point was, there's never a moment when anything fundamental changes about what is being computed, whether there's a barrier with slits or not.
I realize this is in many ways beside the point, but even if your original belief had been correct, "The Men's and Women's teams should play each other to help resolve the pay disparity" is a non-sequitor. Pay is not decided by fairness. It's decided by collective bargaining, under constraints set by market conditions.
You mention them once, but I would love to see a more detailed comparison, not to private industry, but to advocacy and lobbying adoption and usage of AI.
As someone who very much enjoys long showers, a few words of caution.
- Too-long or too-frequent exposure to hot water (time and temperature thresholds vary per person) can cause skin problems and make body odor worse. Since I started RVing I shower much less (maybe twice a week on average, usually only a few minutes of water flow for each) and smell significantly better, with less dry skin or acne or irritation. Skipping one shower makes you smell worse. Skipping many showers and shortening the remainder can do the opposite.
- A shower, depending on temperature and flow rate, consumes around 10-20kW thermal. It's probably the single most energy-intensive activity most of us regularly engage in other than highway driving. I'm hoping to eventually get a recirculating shower so I don't have to think about this as much, but those are still new, rare, and kinda expensive.
In some senses, we have done so many times, with human adults of differing intelligence and/or unequal information access, with adults and children, with humans and animals, and with humans and simpler autonomous systems (like sprites in games, or current robotic systems). Many relationships other than master-slave are possible, but I'm not sure any of the known solutions are desirable, and they're definitely not universally agreed on as desirable. We can be the AI's servants, children, pets, or autonomous-beings-within-strict-bounds-but-the-AI-can-shut-us-down-or-take-us-over-at-will. It's much less clear to me that we can be moral or political or social peers in a way that is not a polite fiction.
So it's quite ironic if there was a version of Jesus that was embracing and retelling some of those 'heretical' ideas.
Sure, but also there are definitely things Jesus is said in the Bible to have taught and done that the church itself later condemned, rejected, or- if I'm feeling generous - creatively reinterpreted. This would be one more example, based on a related but different set of sources and arguments.
Christianity seems to me in general to be much less tolerant of its own inherent ambiguity than many other religions. Not that other faiths don't have plenty of extremist, absolutist adherents and sects - they clearly do. Still, it seems more common (though there's a lot of exposure bias here for me) for Christians to decide that not only is there one true law, but humans are supposed to intuit what it is, and carry it out - even when the explicit doctrines of the faith they claim to uphold say the opposite.
Epistemic status: Random thought, not examined too closely.
I was thinking a little while ago about the idea that there are three basic moral frameworks (consequentialism, virtue ethics, deontology) with lots of permutations. It occurred to me that in some sense they form a cycle, rather than one trying to be fundamental. I don't think I've ever considered or encountered that idea before. I highly doubt this is in any way novel, and am curious how common it is or where I can find good sources that explore it or something similar.
Events are judged by their consequences.
Actions/choices are judged by their adherence to virtues, which are considered virtues because of the types of consequences they engender.
Priority conflicts among virtues are judged by a given or agreed-upon set of rules, which say what the virtues are and how to enact them.
Conflicts between rules are judged by expectations of the consequences for future events of enacting the virtues/choices/actions prescribed by said rules.
I can't really evaluate the specific claims made here, I haven't read the texts or done the work to think about them enough, but reading this, The Earth became older, of a sudden. It's the same feeling I had when someone first pointed out that all the moral philosophy I'd been reading amounted to debating the same three basic frameworks (consequentialism, deontology, virtue ethics) since the dawn of writing. Maybe the same is true for the three cranes (chance, evolution, design).
Thanks, "hire"-->"higher" typo fixed.
Indeed. Major quality change from prior models.
Had a nice chat with GPT-4.5 the other day about fat metabolism and related topics. Then I asked it for an optimal nutrition an exercise plan for a hypothetical person matching either I or my wife's age, height, weight, gender, and overall distribution of body fat. It came back detailed plans, very different for each of us, and very different from anything I've seen in a published source, but which extremely closely matches the sets of disparate diets, eating routines, exercise routines, and supplements we'd stumbled upon as "things that seem to make us feel better when we do them" over the course of about 7 years of self-experimentation. There were also a few simple additional suggestions for me that I'd never thought could really matter that it turned out, when I tried them, do.
On one hand I didn't learn anything "new" except some implementation details (timing and dosage of supplements and pairings of foods, for example) and the value of combining all the pieces instead of trying them one at a time. On the other hand, it found and validated and gave good citations for a bunch of things I'm confident were not explicit or implicit in my prompts and which do not match advice I'd ever received from any "expert" source.
If you do it right, being willing to ask questions of those higher up, like said CEO, is how you get noticed, on their radar, as someone potentially worth watching and investing in and promoting in the future. A secure CEO in a healthy culture is likely to take it as a good sign that employees are aware, intelligent, and paying attention enough to ask clear, well-formed questions.
But if you ask a question in a way that offends that particular individual in whatever way, or makes your direct boss look bad to his direct boss (in either of their perceptions), then that can lead to any of those individuals retaliating in various ways, if their personality or position in the hierarchy makes them feel insecure or like that would make them look or feel better. Asking makes you and them socially vulnerable, and being willing to do so shows you're secure in your understanding of how people will react as well as in your own position/role/status.
Since this was a Zoom meeting, the fact that you asked verbally is also in some sense a status claim, that you felt empowered to ask a question in a way that commanded everyone's attention and interrupted the CEO's talk track. It's a power move, or can be seen as such. If you'd written in a public chat channel, you'd have left it up to others when and how to respond. If you'd back-channel messaged someone higher up in your organization, you'd have given them the option to either ignore it, message you back, or ask the question in the style and at the moment they deemed most appropriate.
Or, and this is what I think happened at the math conference example, if your question is insufficiently well-formed, then a large public meeting is the wrong forum in which to ask it, because the answer may (in the opinion of those better informed) be a waste of everyone's time. Now of course a great speaker will try to hear that kind of question, figure out the source of the confusion, consider whether similar confusions are likely to be prominent among the audience, and either address that source, or gently let you know there are other factors you're missing that sidestep the question, or else point you in the direction of the info that will resolve your confusion (aka "Come ask me after, and I'll list some papers and textbooks you might want to check out on that.") But a less comfortable and capable speaker won't know to or know how to do that, and might shut you down, or get flustered.
Two other examples:
Some forums have a cultural expectation that the option to ask questions is in some sense not real, or not intended to be used, even when it looks like it should be. A colleague of mine was once asked to be on an expert panel in Korea, and asked a fellow panelist a detailed question; he was later told that was a major faux pas, because panels in that context are scripted and no one asks real questions in real time. He got a pass because he was an American and no one had thought to tell him that expectation, but it did interfere with his ability to network at that event and he didn't get invited back.
In a small upper-level college or grad school class, you're supposed to ask detailed questions. If you don't, or can't, you're probably not paying enough attention. But in an intro class of 800 students in a big lecture hall, the lecturer is often going to be pressed for time, and they'd never get through everything if students all felt free to raise questions; the proper time to do that is in office hours or a recitation with the professor or a TA. If the question is important or a lot of people ask something similar, it's their job to filter it up to mention in a future lecture.
Without a currently-implausible level of trust in a whole bunch of models, people, and companies to understand how and when to use privileged information and be able to execute it, removing the New Chat button would be a de factor ban on LLM use in some businesses, including mine (consulting). The fact that Chemical Company A asked a question about X last month is very important information that I'm not allowed to use when answering Chemical Company B's new question about the future of X, and also I'm not allowed to tell the model where either question came from or why I asked them and I have to remember every piece of information that I need to tell it not to use. Also, at least at current capability levels, "Open five chat windows and try different versions of a prompt" is actually a useful strategy for me that disappears if companies make that interface change.
There's an important reason to keep some of us around. This is also an important point.
Consequentialism tells us that the thing to do is the thing with the best results. But, this is a ridiculously high standard, that nobody can actually live up to. Thus, consequentialism tells us that everybody is bad, and we should all condemn everybody and all feel guilty.
In these scenarios I like point out that the other party is using an appeal to consequences as the justification for rejecting consequentialism.
This, to me, gestures at a set of questions with potentially different answers.
- If I've been living as a sexual person of any kind, should I choose to make myself ace, given the choice?
- If I've been living as an asexual person, should I choose to change that, given the option? If so, to what sexuality?
- If I am in something like Rawls' original position and can choose my sexuality for my upcoming life, what should I pick?
- If I am in something like Rawls' original position and can choose everyone's sexualities for their upcoming lives, what should I pick?
(1) and (2) are individual choices where I can't think of any universally compelling reason to say yes or no for anyone else. Some choices will be more or less convenient in different societal contexts, and more or less appealing in different personal contexts.
For (3), I could see arguments to be made for either ace or bi or pan, moreso than straight or gay or anything else.
For (4) choosing ace or gay is likely to result in very low fertility rates, unless you are able to get very fine grained as to what kind of asexuality people end up with or get to tweak other drives as well. This probably leads to population collapse unless the technology level is substantially beyond ours. Straight, bi, or pan, all potentially lead to worlds that I can imagine going well.
Some of both, more of the former, but I think that is largely an artifact of how we have historically defined tasks. None of us have ever managed an infinite army of untrained interns before, which is how I think of LLM use (over the past two years they've roughly gone from high school student interns to grad student interns), so we've never refactored tasks into appropriate chunks for that context.
I've been leading my company's team working on figuring out how to best integrate LLMs into our workflow, and frankly, they're changing so fast with new releases that it's not worth attempting end-to-end replacement in most tasks right now. At least, not for a small company. 80/20 rule applies on steroids, we're going to have new and better tools and strategies next week/month/quarter anyway. Like, I literally had a training session planned for this morning, woke up to see the Gemini 2.5 announcement, and had to work it in as "Expect additional guidance soon, please provide feedback if you try it out." We do have a longer term plan for end-to-end automation of specific tasks, as well, where it is worthwhile. I half-joke that Sam Altman tweets a new feature and we have to adapt our plans to it.
Current LLMs can reduce the time required to get up-to-speed on publicly available info in a space by 50-90%. They can act as a very efficient initial thought partner for sanity checking ideas/hypotheses/conclusions, and teacher for overcoming mundane skill issues of various sorts ("How do I format this formula in Excel?"). They reduce the time required to find and contact people you need to actually talk to by much less, maybe 30%, but that will go way down if and when there's an agent I can trust to read my Outlook history and log into my LinkedIn and Hunter.io and ZoomInfo and Salesforce accounts and draft outreach emails. Tools like NotebookLM make it much more efficient to transfer knowledge across the team. AI notetakers help ensure we catch key points made in passing in meetings and provide a baseline for record keeping. We gradually spend more time on the things AI can't yet do well, hopefully adding more value and/or completing more projects in the process.
These are very reasonable questions that I learned about the hard way camping in the desert two years ago. I do not recommend boondocking in central Wyoming in August.
First, because when you live in an aluminum box with 1" thick R7 walls you need more air conditioning in summer than that much solar can provide. It doesn't help that RV air conditioners are designed to be small and light and cheap (most people only use them a handful of days a year), so they're much less efficient than home air conditioners, even window units. I have 2x 15k BTU/hr AC units, and can only run one at a time on my inverter (they use 1400-1800W each). On very hot days (>90-95F) I need both at least some of the time.
Second, because the conversion efficiency of silicon PV falls at high temperatures, so hot and sunny summer days are actually not my days of peak production.
Third, my batteries and inverter are unfortunately but unavoidably placed in a closed compartment with limited airflow covered in black painted aluminum. And consumer grade inverters are not great, there's something like 15-20% loss (heat generation). That means on hot days it's sometimes challenging to keep these from overheating, and running the generator to give the inverter a break while the batteries recharge can be helpful.
Fourth, in addition to low solar production in winter, electricity consumption in an RV is higher than you might expect in cold weather. The propane furnace draws electric power for the fan. Since the plumbing is exposed to air, you need electric tank and line heaters for the fresh water tank, waste water tanks, and water lines to avoid freezing. I also use electric tank warmers for my propane tanks, since when the weather drops below freezing a partially-empty 20 lb tank can't supply the steady 30k BTU/hr the furnace needs (it normally relies on ambient heat to boil off liquid propane, and at low T in a small tank that doesn't happen fast enough, which can cut supply and even freeze the regulator). On a cold winter day, I'm probably drawing an average of 300-600 watts just to keep the plumbing and furnace working well. Granted, not many people winter in an RV in Massachusetts, I'm an unusual case. I wouldn't have this problem in most of the Southwest or Florida where other RVers go.
Yes, this lines up with current average prices for solar at time of production vs firmed. We're only finally starting to see firmed green power prices get covered much even by experts, now that penetration rates are rising and companies are realizing they made big noises about 2030 and 2050 goals before having actually made any kind of plan to achieve them.
But, unless you're at Kardashev-1 levels of power demand (we're not), why would you try to run a grid on all solar? Who is proposing doing that, even in the world's sunniest regions? The most cost-effective way to decarbonize involves a locally-optimized combo of sources, some mix potentially including solar, wind, hydro, geothermal, nuclear, biomass, gasified MSW, maybe wave and tidal if they start making more sense, whatever is available.
Also consider that as EVs continue to grow, electricity demand increases and changes the demand curve, but any place that actually had the foresight to plan for this will see it is "Now we're going to have a large percentage of homes with a 2-day battery in their garage, and the utility pays no capex for it," either as DER (V2G) or as dispatchable demand with smart charging. Similarly, as we electrify more industrial processes, there's a lot more market for load-shifting and efficiency-increasing technologies like phase change materials, thermal storage and heat recovery, air and ground source heat pumps, and so on that all create more avenues to increase grid resilience.
Unfortunately, from a regulatory perspective, almost nowhere has set themselves up to be able to manage or incentivize this anywhere close to intelligently, and almost no one is sufficiently empowered to convene and negotiate with the full set of stakeholders needed to fix that. And even internally, companies often cannot motivate themselves to take even very obvious high-ROI energy and carbon saving measures (like duct sealing) if it involves even the slightest short-term inconvenience.
Also, I think these kinds of models often assume a strong desire to get non-renewable power down to zero very soon, which I think is in many ways a mistake. For context, right now I live in an RV. When I'm off grid, I have 1050W of solar, ~10kWh of batteries, a 3kW hybrid inverter, and a 5.5kW gasoline generator. In spring and fall I can easily go a week without needing shore power or the generator. In summer and winter, I can't, so I use the generator a bit each dayif I'm off grid, and manage demand as best I can, and stay on-grid more often. But it would make no sense, financially or (at this point) environmentally, to try to add enough solar or batteries to not need anything else. Instead I'd first replace my AC and supplement my furnace with mini-split heat pumps. Then when I become stationary again (probably soon) I'm considering getting a wood gasifier generator. And I'm assuming my next car will be electric, and hoping maybe by then I'll be able to make use of that extra storage capacity intelligently. I can certainly load-shift much of my demand. And after that if I'm still getting 10% of my much-reduced electricity needs from fossil fuels, it's not really urgent to fix that. If you reduce the growth rate of a cumulative problem by 90%, you now have 10x longer to fix the rest.
I can't comment on software engineering, not my field. I work at a market research/tech scouting/consulting firm. What I can say is that over the past ~6 months we've gone from "I put together this 1 hour training for everyone to get some more value out of these free LLM tools," to "This can automate ~half of everything we do for $50/person/month." I wouldn't be surprised if a few small improvements in agents over the next 3-6 months push that 50% up to 80%, then maybe 90% by mid next year. That's not AGI, but it does get you to a place where you need people to have significantly more complex and subtle skills, that currently take a couple of years to build, before their work is adding significant value.
The clockwise rule is what you are supposed to do if people arrive to the intersection at the same time.
If exactly two people going opposite directions arrive at the same time and aren't both going straight, then the one going straight goes before the one turning, or the one turning right goes before the one turning left.
At least, that's how I and everyone I know was taught, and no, those of us who asked what "at the same time" actually means never got a straight answer.
Sure. And I'm of the opinion that it is only common sense after you've done quite a lot of the work of developing a level of intuition for mathematical objects that most people, including a significant proportion of high school math teachers, never got.
Wanted to add:
I think this post is great for here on LW, but if someone wanted to actually start teaching students to understand math more deeply, calling it common sense probably comes off as condescending, because it doesn't feel that way until you get comfortable with it. There's a lot to unlearn and for a lot of people it is very intimidating.
Personally I wish we treated math class at least some of the time as a form of play. We make sure to teach kids about jokes and wordplay and do fun science-y demonstrations, but math is all dry and technical. We assign kids books to read like A Wrinkle in Time and The Phantom Tollbooth. But, I don't think my elementary school teachers had any clue what a tesseract was, or what the Mathemagician and Dodecahedron are all about, and so that whole aspect of these books was just a lost opportunity for all but maybe 3 kids in my grade.
Fair enough.
I do believe it's plausible that feelings, like pain and hunger, may be old and fundamental enough to exist across phyla.
I'm much less inclined to assume emotions are so widely shared, but I wish I could be more sure either way.
Mostly agreed. I have no idea how to evaluate this for most animals, but I would be very surprised if other mammals did not have subjective experiences analogous to our own for at least some feelings and emotions.
which I worry your teachers didn’t
Oh it can be so much worse than that - actively pushing students away from that kind of understanding. I've had math teachers mark answers wrong because I (correctly) derived a rule I'd forgotten instead of phrasing it the way they taught it, or because they couldn't follow the derivation. Before college, I can think of maybe two of my teachers who actually seemed to understand high school math in any deeper way.
Thanks, fixed!