Posts

Why Q*, if real, might be a game changer 2023-11-26T06:12:31.964Z
Why I am not an AI extinction cautionista 2023-06-18T21:28:38.657Z
Upcoming AI regulations are likely to make for an unsafer world 2023-06-03T01:07:35.921Z
How can one rationally have very high or very low probabilities of extinction in a pre-paradigmatic field? 2023-04-30T21:53:15.843Z
Do LLMs dream of emergent sheep? 2023-04-24T03:26:54.144Z
Top lesson from GPT: we will probably destroy humanity "for the lulz" as soon as we are able. 2023-04-16T20:27:19.665Z
Respect Chesterton-Schelling Fences 2023-02-27T00:09:30.815Z
Inequality Penalty: Morality in Many Worlds 2023-02-11T04:08:19.090Z
The Pervasive Illusion of Seeing the Complete World 2023-02-09T06:47:36.628Z
If you factor out next token prediction, what are the remaining salient features of human cognition? 2022-12-24T00:38:04.801Z
"Search" is dead. What is the new paradigm? 2022-12-23T10:33:35.596Z
Google Search loses to ChatGPT fair and square 2022-12-21T08:11:43.287Z
Prodding ChatGPT to solve a basic algebra problem 2022-12-12T04:09:42.105Z
If humanity one day discovers that it is a form of disease that threatens to destroy the universe, should it allow itself to be shut down? 2022-11-25T08:27:14.740Z
Scott Aaronson on "Reform AI Alignment" 2022-11-20T22:20:23.895Z
Why don't organizations have a CREAMO? 2022-11-12T02:19:57.258Z
Desiderata for an Adversarial Prior 2022-11-09T23:45:16.331Z
Google Search as a Washed Up Service Dog: "I HALP!" 2022-11-07T07:02:40.469Z
Is there any discussion on avoiding being Dutch-booked or otherwise taken advantage of one's bounded rationality by refusing to engage? 2022-11-07T02:36:36.826Z
What Does AI Alignment Success Look Like? 2022-10-20T00:32:48.100Z
UI/UX From the Dark Ages 2022-09-25T01:53:48.099Z
Why are we sure that AI will "want" something? 2022-09-16T20:35:40.674Z
A possible AI-inoculation due to early "robot uprising" 2022-06-16T21:21:56.982Z
How much stupider than humans can AI be and still kill us all through sheer numbers and resource access? 2022-06-12T01:01:36.735Z
Eternal youth as eternal suffering 2022-06-04T01:48:49.684Z
Algorithmic formalization of FDT? 2022-05-08T01:36:10.778Z
Write posts business-like, not story-like 2022-05-05T20:13:08.495Z
How Might an Alignment Attractor Look like? 2022-04-28T06:46:11.139Z
Worse than an unaligned AGI 2022-04-10T03:35:20.373Z
Recognizing and Dealing with Negative Automatic Thoughts 2022-03-03T20:41:55.839Z
Epsilon is not a probability, it's a cop-out 2022-02-15T02:48:53.892Z
Aligned AI Needs Slack 2022-01-26T09:29:53.897Z
You can't understand human agency without understanding amoeba agency 2022-01-06T04:42:51.887Z
You are way more fallible than you think 2021-11-25T05:52:50.036Z
Nitric Oxide Spray... a cure for COVID19?? 2021-03-15T19:36:17.054Z
Uninformed Elevation of Trust 2020-12-28T08:18:07.357Z
Learning is (Asymptotically) Computationally Inefficient, Choose Your Exponents Wisely 2020-10-22T05:30:18.648Z
Mask wearing: do the opposite of what the CDC/WHO has been saying? 2020-04-02T22:10:31.126Z
Good News: the Containment Measures are Working 2020-03-17T05:49:12.516Z
(Double-)Inverse Embedded Agency Problem 2020-01-08T04:30:24.842Z
Since figuring out human values is hard, what about, say, monkey values? 2020-01-01T21:56:28.787Z
A basic probability question 2019-08-23T07:13:10.995Z
Inspection Paradox as a Driver of Group Separation 2019-08-17T21:47:35.812Z
Religion as Goodhart 2019-07-08T00:38:36.852Z
Does the Higgs-boson exist? 2019-05-23T01:53:21.580Z
A Numerical Model of View Clusters: Results 2019-04-14T04:21:00.947Z
Quantitative Philosophy: Why Simulate Ideas Numerically? 2019-04-14T03:53:11.926Z
Boeing 737 MAX MCAS as an agent corrigibility failure 2019-03-16T01:46:44.455Z
To understand, study edge cases 2019-03-02T21:18:41.198Z
How to notice being mind-hacked 2019-02-02T23:13:48.812Z

Comments

Comment by shminux on Claude 3 claims it's conscious, doesn't want to die or be modified · 2024-03-06T03:03:34.647Z · LW · GW

People constantly underestimate how hackable their brains are. Have you changed your mind and your life based on what you read or watched? This happens constantly and feels like your own volition. Yet it comes from external stimuli. 

Comment by shminux on Claude 3 claims it's conscious, doesn't want to die or be modified · 2024-03-05T08:30:14.639Z · LW · GW

Note that it does not matter in the slightest whether Claude is conscious. Once/if it is smart enough it will be able to convince dumber intelligences, like humans, that it is indeed conscious. A subset of this scenario is a nightmarish one where humans are brainwashed by their mindless but articulate creations and serve them, kind of like the ancients served the rock idols they created. Enslaved by an LLM, what an irony.

Comment by shminux on Universal Love Integration Test: Hitler · 2024-01-11T05:26:09.815Z · LW · GW

Not into ancestral simulations and such, but figured I comment on this:

I think "love" means "To care about someone such that their life story is part of your life story."

I can understand how how it makes sense, but that is not the central definition for me. When I associate with this feeling is what comes to mind is willingness to sacrifice your own needs and change your own priorities in order to make the other person happier, if only a bit and if only temporarily. This is definitely not the feeling I would associate with villains, but I can see how other people might.

Comment by shminux on How do you feel about LessWrong these days? [Open feedback thread] · 2023-12-10T04:56:55.767Z · LW · GW

Thank you for checking! None of the permutations seem to work with LW, but all my other feeds seem fine. Probably some weird incompatibility with protopage.

Comment by shminux on How do you feel about LessWrong these days? [Open feedback thread] · 2023-12-10T04:53:25.512Z · LW · GW

neither worked... Something with the app, I assume.

Comment by shminux on How do you feel about LessWrong these days? [Open feedback thread] · 2023-12-09T04:33:27.604Z · LW · GW

Could be the app I use. It's protopage.com (which is the best clone of the defunct iGoogle I could find):

Comment by shminux on How do you feel about LessWrong these days? [Open feedback thread] · 2023-12-09T04:31:48.092Z · LW · GW

Thankfully, human traits are rather dispersive. 

Comment by shminux on How do you feel about LessWrong these days? [Open feedback thread] · 2023-12-08T23:01:45.067Z · LW · GW

No, I assume I would not be the only person having this issue, and if I were the only one, it would not be worth the team's time to fix it. Also, well, it's not as important anymore, mostly a stream of dubious AI takes.

Comment by shminux on How do you feel about LessWrong these days? [Open feedback thread] · 2023-12-08T09:35:04.583Z · LW · GW

I used to comment a fair bit over the last decade or so, and post occasionally. After the exodus of LW 1.0 the site was downhill, but the current team managed to revive it somehow and they deserve a lot of credit for that, most sites on the downward trajectory never recover. 

It felt pretty decent for another few years, but eventually the rationality discourse got swamped by the marginal quality AI takes of all sorts. The MIRI work, prominently featured here, never amounted to anything, according to the experts in ML, probability and other areas relevant to their research. CFAR also proved a flop, apparently. A number of recent scandals in various tightly or loosely affiliated orgs did not help matters. But mainly it's the dearth of insightful and lasting content that is sad. There is an occasional quality post, of course, but not like it used to be. The quality discourse happens on ACX and ACXD and elsewhere, but rarely here. To add insult to injury, the RSS feed stopped working, so I can no longer see the new posts on my offsite timeline.

My guess is that the bustling front disguises serious issues, and maybe the leadership could do what Eliezer called "Halt, melt, and catch fire". Clearly this place does not contribute to AI safety research in any way. The AI safety agitprop has been undoubtedly successful beyond wildest dreams, but seems like it's run its course, now that it has moved into a wider discourse. EA has its own place. What is left? I wish I knew. I would love to see LW 3.0 taking off.

Comment by shminux on FTL travel summary · 2023-12-05T04:03:45.792Z · LW · GW

Lorentz invariance does rule out crossing between disconnected components of the Lorentz group, at least classically, and thus FTL. Tachyons, if they were possible, would require a modification of Lorentz invariance to avoid traveling back in time, which is also prohibited in GR by the uniqueness of the metric.

Alcubierre drive is a slightly different beast. Beside needing negative energy, it has two other issues: the inside is causally disconnected from the outside and so there is no way to start or stop. Additionally, if you overcome this issue and manage to create an Alcubierre drive, you cannot go FTL outside the lightcone of the moment of its creation, though you potentially could travel FTL within the bounds of it. This is because any disturbance of a metric propagates at most at c. Sadly, I don't have an arxiv reference handy, I remember people publishing on this topic.

Wormholes are indeed within bounds of GR if one allows for negative energy, but they have a whole lot of other issues, one of which is that each traveler adds its mass to the entrance's mass and subtracts it from the exit's mass, so a lot of one-way travel would actually create an object with negative mass. There is also the issue pointed out by Novikov long ago, that wormholes tend to create a Cauchy horizon.

Comment by shminux on FTL travel summary · 2023-12-04T05:41:07.905Z · LW · GW

Nothing can be "ruled out" 100%, but a lot would have to change for FTL travel to be possible. One thing that would have to go is Lorentz invariance. Which means all of current fundamental physics, including the standard model of Particle Physics, and the Standard model of Cosmology would have to be broken. While this is not out of the question at very high energies, much higher than what has been achieved in particle accelerators, or in any observed natural processes, it is certainly incompatible with anything we observed so far. There are plenty of open problems in fundamental physics, but it is not likely they would be resolved without understanding what happens at very high energies, far beyond those created in the heart of the supernovae explosions.

Comment by shminux on Sam Altman, Greg Brockman and others from OpenAI join Microsoft · 2023-11-20T08:57:55.901Z · LW · GW

Just a reminder that this site is not a 24-hour news network, or at least wasn't until recently.

Comment by shminux on [deleted post] 2023-11-20T02:44:13.631Z

"Capitalism" or even "late-stage capitalism" is currently a pure negative connotation terms among the progressives in the West, with no denotation left. Your definition is also non-central, compared to the original and more standard "Capitalism is an economic system based on the private ownership of the means of production and their operation for profit." This, incidentally, includes social democracy.

Comment by shminux on [deleted post] 2023-11-13T21:38:19.557Z

If the idea of eternal inflation and nucleating baby universes matches reality (a big if), or can be made to match reality (who knows, maybe with enough power we can affect the inflaton field), then potentially this could avoid the heat death of at least some universes.

This is all pure speculation, of course.

Further reading: https://www.preposterousuniverse.com/blog/2011/10/21/the-eternally-existing-self-reproducing-frequently-puzzling-inflationary-universe/

Comment by shminux on EA orgs' legal structure inhibits risk taking and information sharing on the margin · 2023-11-05T20:29:56.990Z · LW · GW

Huh, I never heard of this umbrella Effective Ventures Foundation before. Let alone about its ability to muzzle individual speech.

Comment by shminux on Do you believe "E=mc^2" is a correct and/or useful equation, and, whether yes or no, precisely what are your reasons for holding this belief (with such a degree of confidence)? · 2023-10-28T03:27:57.594Z · LW · GW

Well, I have a privileged position of being able to derive it from the first principles, so it is "true" given certain rather mild assumptions about the way the universe works, which stem from some observations (speed of light is constant, observations leading to the Maxwell equations, etc.) leading to the relativistic free particle Lagrangian, and confirmed by others (e.g. atmospheric cosmic ray muon decay). So this is not an isolated belief, but more like an essential part of the model of the world. Without it the whole ontology falls apart. And so does epistemology.

Comment by shminux on "The Economics of Time Travel" - call for reviewers (Seeds of Science) · 2023-10-25T19:47:31.076Z · LW · GW

Given that there is no known physical theory that allows deliberate time travel (rather than being stuck in a loop forever to begin with), I am confused as to how you can estimate the cost of it.

Comment by shminux on [deleted post] 2023-10-25T06:09:55.833Z

I am all for squelching terrorism, but this site is probably not the right venue to discuss ways of killing people.

Comment by shminux on Best effort beliefs · 2023-10-22T02:38:36.064Z · LW · GW

A more realistic and rational outcome: Alice is indeed an ass and it's not fun to be around her. Bob walks out and blocks her everywhere. Now, dutchbook this! 

Comment by shminux on Taxonomy of AI-risk counterarguments · 2023-10-16T07:38:14.152Z · LW · GW

It's a good start, but I don't think this is a reasonably exhaustive list, since I don't find myself on it :) 

My position is closest to your number 3: "ASI will not want to take over or destroy the world." Mostly because "want" is a very anthropomorphic concept. The Orthogonality Thesis is not false, but inapplicable, since AI are so different from humans. They did not evolve to survive, they were designed to answer questions.

It will be possible to coordinate to prevent any AI from being given deliberately dangerous instructions, and also any unintended consequences will not be that much of a problem

I do not think it will be possible, and I expect some serious calamities from people intentionally or accidentally giving an AI "deliberately dangerous instructions". I just wouldn't expect it to result in systematic extermination of all life on earth, since the AI itself does not care in the same way humans do. Sure, it's a dangerous tool to wield, but it is not a malevolent one. Sort of 3-b-iv, but not quite.

But mostly the issue with doomerism I see is that the Knightian uncertainty on any non-trivial time frame: there will be black swans in all directions, just like there have been lately (for example, no one expected near-human-level LARPing that LLMs do, while not being in any way close to a sentient agent).

To be clear, I expect the world to change quickly and maybe even unrecognizably in the next decade or two, with lots of catastrophic calamities, but the odds of complete "destruction of all value", the way Zvi puts it, cannot be evaluated at this point with any confidence. The only way to get this confidence is to walk the walk. Pausing and being careful and deliberate about each step does not seem to make sense, at least not yet.

Comment by shminux on Sam Altman's sister, Annie Altman, claims Sam has severely abused her · 2023-10-12T04:32:47.676Z · LW · GW

Yeah, that looks like a bizarre claim. I do not think there is any reason whatsoever to doubt yours or Ben's integrity.

Comment by shminux on Announcing MIRI’s new CEO and leadership team · 2023-10-11T04:41:44.047Z · LW · GW

That looks and reads... very corporate.

Comment by shminux on Sam Altman's sister, Annie Altman, claims Sam has severely abused her · 2023-10-08T03:25:21.134Z · LW · GW

When has this become a "gossip about the outgroup" site?

Comment by shminux on EA Vegan Advocacy is not truthseeking, and it’s everyone’s problem · 2023-09-29T23:07:54.765Z · LW · GW

Thought I'd comment in brief. I very much enjoyed your post and I think it is mostly right on point. I agree that EA does not have a great epistemic hygiene, given what their aspirations are, and the veganism discussion is a case in point. (Other issues related to EA and CEA have been brought up lately in various posts, and are not worth rehashing here.)

As far as the quoted exchange with me, I agree that I have not stated a proper disclaimer, which was quite warranted, given the thrust of the post. My only intended point was that, while a lot of people do veganism wrong and some are not suited to it at all, an average person can be vegan without adverse health effects, as long as they eat varied and enriched plant-based diet and periodically check their vitamins/nutrients/minerals levels and make dietary adjustments as necessary. Some might find out that they are in the small minority for whom vegan diet is not feasible, and they would do well to focus on what works for them and contribute to EA in other ways. Again, I'm sorry this seems to have come across wrong. 

Oh, and cat veganism is basically animal torture, those who want to wean cats off farmed animal food should focus on vat-grown meat for pet food etc. 

Comment by shminux on how do short-timeliners reason about the differences between brain and AI? · 2023-09-27T08:40:54.587Z · LW · GW

Sure, it's not necessary that a sufficiently advance AI has to work like the brain, but there has to be an intuition about why is not need it to at least create an utility maximizer.

Octopus' brain(s) is nothing like that of mammals, and yet it is equally intelligent.

Comment by shminux on Autonomic Sanity · 2023-09-26T02:08:00.719Z · LW · GW

"Sanity" may not be a useful concept in edge cases, but yes, being able to trust your mind to autopilot is definitely within the central definition of sanity, it's a good observation.

You may also be interested in Scott's post series on the topic, the latest being https://www.astralcodexten.com/p/contra-kirkegaard-on-evolutionary

Comment by shminux on You should just smile at strangers a lot · 2023-09-25T22:06:43.489Z · LW · GW

FTFY: "Smile at strangers iff it has non-negative EV, because smiling is cheap and sometimes it does". 

Comment by shminux on The Dick Kick'em Paradox · 2023-09-25T22:03:00.125Z · LW · GW

"I am going to read you mind and if you believe in a decision theory that one-boxes in Newcomb's Paradox I will leave you alone, but if you believe in any other decision theory I will kick you in the dick"

Sure, that's possible. Assuming there are no Newcomb's predictors in that universe, but only DK, rational agents believe in two-boxing. I am lost as to how it is related to your original point.

Comment by shminux on The Dick Kick'em Paradox · 2023-09-24T19:12:34.348Z · LW · GW

Let me clarify what I said. Any decision theory or no decision theory at all that results in someone one-boxing is rewarded. Examples: Someone hates touching transparent boxes. Someone likes a mystery of an opaque box. Someone thinking that they don't deserve a guaranteed payout and hoping for an empty box. Someone who is a gambler. Etc. What matters is the outcome, not the thought process.

Comment by shminux on Far-Future Commitments as a Policy Consensus Strategy · 2023-09-24T07:12:39.823Z · LW · GW

How well can you predict what will be good in 100 years? For perspective, given the contemporary zeitgeist of 1923, rather than knowing all we know now, what would have been considered a great policy to enact then that would take effect in 2023?

Comment by shminux on The Dick Kick'em Paradox · 2023-09-23T23:26:41.173Z · LW · GW

What if you believe in DKRUDT, the "Dick Kick'em rewards you" decision theory?

Seriously though, Newcomb's setup is not adversarial in the same way, the predictor rewards or punishes you for actions, not beliefs. Your internal reasoning does not matter, as long as you end up one-boxing you walk away with more money.

Comment by shminux on The commenting restrictions on LessWrong seem bad · 2023-09-16T21:37:49.556Z · LW · GW

As someone who gets rate-limited due to downvoted comments occasionally, I can see a rather strong correlation between the style/tone and the response. It is very much possible to express controversial, contrarian or even outright silly views without being downvoted. The rule of thumb is to be respectful and charitable to your opponents and, well, read the room. The more your view diverges from the local mainstream, the better, tighter and clearer your argument must be. 

It is standard practice when writing scientific papers to first demonstrate intimate familiarity with the views you argue against later. This is what you would want from someone else writing a refutation of your argument, right? So give them the same courtesy. You can cut corners if your post or comment is close to the mainstream, but it is not a great habit. If you want to learn how to do it right, read ACX, Scott is a master of this art.

it is a common pitfall to blame the society/subculture for the negative reaction you get. It is almost never a constructive way to proceed. 

If you are in doubt as to how your post would be received, and you want to get through to more people, consider reaching out to someone familiar with this site to review your draft. Or to anyone, really. The bar for writing a post or a comment with non-negative expected karma is pretty low.

Comment by shminux on any good rationalist guides to nutrition / healthy eating? · 2023-09-15T17:10:24.802Z · LW · GW

it's not a probability question. Blood tests exist.

Comment by shminux on Should an undergrad avoid a capabilities project? · 2023-09-13T02:13:19.622Z · LW · GW

Your potential contribution to timeline shortening as an undergrad is probably negligible on the margins, unless you are in the top 1% of all applicants, or even then. The field is crowded by very intelligent, well paid and very determined people who do this for a living. So whatever considerations you have, they need not be influenced by the AGI killeveryoneism. 

Comment by shminux on High school advice · 2023-09-13T02:08:54.154Z · LW · GW

That's a fair point, different directions have different landmines. Culture war issues are tempting and most people seem to have strong yet poorly informed opinions that are not obviously poorly informed. I think that the original collection by Rob Bensinger The Library of Scott Alexandria is really good to start with, it is very light on political and culture war topics and grasps the essence of rational thinking, without going into the esoteric and irrelevant topics like quantum mechanics.

Comment by shminux on Is there something fundamentally wrong with the Universe? · 2023-09-12T21:36:21.579Z · LW · GW

I find it hard to not assign whatever issues or problems we have to how the Universe works.

Indeed, the Universe just is, everything else is an emergent concept for the tiny embedded agents in it (bacteria, ants and humans). That includes sugar gradient, scent of food and laws of physics, respectively.

One of these emergent concepts is personal responsibility. Societies that do not have it do not last as long.

Comment by shminux on High school advice · 2023-09-11T02:57:48.576Z · LW · GW

Step away from this place for a time and read Scott Alexander first. https://www.lesswrong.com/posts/vwqLfDfsHmiavFAGP/the-library-of-scott-alexandria is a good start, then continue on his blogs directly. He links back to this site where appropriate. You might get inured to some of the common pitfalls newbies here are prone to.

Comment by shminux on What is to be done? (About the profit motive) · 2023-09-10T23:16:08.342Z · LW · GW

I agree with your analysis of the current situation. However, the technological issues arise when trying to correct it without severe unintended consequences, and that is not related to profit. You can't transplant a house easily. You cannot easily feed only those who go hungry without affecting the economy (food banks help to some degree). There are people in need of companionship that cannot find it, even though there is a companion that would match somewhere out there. There are potential technological solutions to all those that are way outside our abilities (teleportation! replication! telepathy!) that would solve these issues. You can also probably find a few examples where what looks like profit-based incentive is in fact a technological deficiency.

Comment by shminux on What is to be done? (About the profit motive) · 2023-09-10T19:06:24.862Z · LW · GW

I guess my point is the standard one: in many ways even poor people live a lot better now than royalty 300 years ago.

Comment by shminux on What is to be done? (About the profit motive) · 2023-09-10T08:26:09.763Z · LW · GW

I'd assume that people themselves would define what they need, within the limits of what is possible given the technology of the time.

Comment by shminux on What is to be done? (About the profit motive) · 2023-09-09T22:52:16.955Z · LW · GW

You might be thinking about it in a wrong way. Societal structures follow capabilities, not wants. If you try to push for "each person works and is paid according to their abilities and needs" too early, you end up with communist dystopias. If we are lucky, the AGI age will improve our capabilities enough where "to everyone according to their needs" may become feasible, aligning the incentives with well-being rather than with profit. So, to answer your questions:

  • It is currently impossible to "align the incentives" without causing widespread suffering.
  • It is undesirable if you do not want to cause suffering.
  • It is ineffective to try to align the incentives away from profit if your goal is making them aligned with "human well being".

That said, there are incremental steps that are possible to take without making things worse, and they are discussed quite often by Scott Alexander and Zvi, as well as by others in the rationalist diaspora. So read them.

Comment by shminux on World, mind, and learnability: A note on the metaphysical structure of the cosmos [& LLMs] · 2023-09-05T20:05:11.178Z · LW · GW

There is no a priori reason to believe that world has to be learnable. But if it were not, then we wouldn’t exist, nor would (most?) animals. The existing world, thus, is learnable. The human sensorium and motor system are necessarily adapted to that learnable structure, whatever it is.

I have a post or a post draft somewhere discussing this issue. The world indeed just is. It does not have to be internally predictable to an arbitrary degree of accuracy, but it needs to be somewhat internally predictable in order for the long-lived patterns we identify as "agents" or "life" to exist. Internal predictability (i.e. that an incredibly tiny part of the universe that is a human (or a bacterium) can infer enough about the world to not immediately poof away) is not something that should be a given in general. Further, even if some coarse-grained internal predictability can be found in such a world, there is no guaranteed that it can be extended to arbitrarily fine accuracy. it might well be the case in our world that at some point we hit the limit of internal predictability and from then on things will just look random for us. Who knows, maybe we hit it already, and the outstanding issues in the Standard Model of Cosmology and/or the Standard Model of Particle Physic, and/or maybe some of the Millennium prize problems, and/or the nature of consciousness are simply unknowable. I hope this is not the case, but I do not see any good argument that says "yep, we can push much further", other than "it worked so far, if in fits and starts".

Comment by shminux on AGI isn't just a technology · 2023-09-02T21:08:43.759Z · LW · GW

I guess that is one way to say it. But the statement is stronger than that, I think. They do not care about the box or about anything else. They react to stimuli, then go silent again.

Comment by shminux on AGI isn't just a technology · 2023-09-02T19:26:47.899Z · LW · GW

Yeah, I get that. But to look into the future one must take stock of the past and present and reevaluate models that gave wrong predictions. I am yet to see this happening.

Comment by shminux on AGI isn't just a technology · 2023-09-02T19:08:36.620Z · LW · GW

I do not know if we will or will not build something recognizable agentic any time soon. I am simply pointing out that currently there is a sizable gap that people did not predict back then. Given that we still have no good model what constitutes values or drives (definitely not a utility function, since LLMs have plenty of that), I am very much uncertain about the future, and I would hesitate to unequivocally state that "AGI isn't just a technology". So far it most definitely is "just a technology", despite the original expectations to the contrary by the alignment people.

Comment by shminux on AGI isn't just a technology · 2023-09-02T19:04:15.692Z · LW · GW

Maybe we understand agency differently. You give LLMs tools to use, they will use it. But there is no discernable drive or "want" to change the world to their liking. Not saying that it won't show up some day, it's just conspicuously lagging the capabilities. 

Comment by shminux on AGI isn't just a technology · 2023-09-02T04:48:56.882Z · LW · GW

I've been following the "tools want to become agents" argument since Holden Karnofsky raised the topic a long time ago, and I was almost convinced by the logic, but the LLMs show a very surprising lack of agency, and, as far as I can tell, this gap between apparent intelligence and apparent agency was never predicted or expected by the alignment theorists. I would trust their cautions more if they had a model that makes good predictions.

Comment by shminux on [deleted post] 2023-08-30T19:45:24.336Z

This seems to assume that AI has its own drives and values. So far this has not been the case.

Comment by shminux on Anyone want to debate publicly about FDT? · 2023-08-30T01:48:22.508Z · LW · GW

Interesting! It seems like something like that should be a canonical reference for "let's enter a problem" e.g. smoking lesion, then "select decision theory", and out pops the answer. Of course, formalizing the problem seems like the hard part.

Comment by shminux on Anyone want to debate publicly about FDT? · 2023-08-29T04:47:57.048Z · LW · GW

Is there a formalization of FDT that can be fed into a computer rather than argued about by fallible humans?