Posts
Comments
Nvidia’s products are rather obviously superior
CUDA seems to be superior to ROCm... and has a big installed-base and third-party tooling advantage. It's not obvious, to me anyway, that NVidia's actual silicon is better at all.
... but NVidia is all about doing anything it can to avoid CUDA programs running on non-NVidia hardware, even if NVidia's own code isn't used anywhere. Furthermore, if NVidia is like all the tech companies I saw during my 40 year corporate career, it's probably also playing all kinds of subtle, hard-to-prove games to sabotage the wide adoption of any good hardware-agnostic APIs.
We caution against purely technical interpretations of privacy such as “the data never leaves the device.” Meredith Whittaker argues that on-device fraud detection normalizes always-on surveillance and that the infrastructure can be repurposed for more oppressive purposes. That said, technical innovations can definitely help.
I really do not know what you are expecting. On-device calculation using existing data and other data you choose to store only, the current template, is more privacy protecting than existing technologies.
She's expecting, or at least asking, that certain things not be done on or off of the device, and that the distinction between on-device and off-device not be made excessively central to that choice.
If an outsider can access your device, they can always use their own AI to analyze the same data.
The experience that's probably framing her thoughts here is Apple's proposal to search through photos on people's phones, and flag "suspicious" ones. The argument was that the photos would never leave your device... but that doesn't really matter, because the results would have. And even if they had not, any photo that generated a false positive would have become basically unusable, with the phone refusing to do anything with it, or maybe even outright deleting it.
Similarly, a system that tries to detect fraud against you can easily be repurposed to detect fraud by you. To act on that detection, it has to report you to somebody or restrict what you can do. On-device processing of whatever kind can still be used against the interests of the owner of the device.
Suppose that there was a debate around the privacy implications of some on-device scanning that actually acted only in the user's interest, but that involved some privacy concerns. Further suppose that the fact that it was on-device was used as an argument that there wasn't a privacy problem. The general zeitgeist might absorb the idea that "on-device" was the same as "privacy-preserving". "On device good, off device bad".
A later transition from "in your interest" to "against your interest" could easily get obscured in any debate, buried under insistence that "It's on-device".
Yes, some people with real influence really, truly are that dumb, even when they're paying close attention. And the broad sweep of opinion tends to come from people who aren't much paying attention to begin with. It happens all the time in complicated policy arguments.
Turns out that this is today's SMBC comic. Which gets extra points points for the line "Humans are a group-level psychiatric catastrophe!"
If the value of Taylor Swift concerts comes mostly from interactions between the fans, is Swift herself essential to it?
this is a fair critique of AIs making everyone losing their jobs.
I have never heard anybody push the claim that there wouldn't be niche prestige jobs that got their whole value from being done by humans, so what's actually being critiqued?
... although there is some question about whether that kind of thing can actually sustain the existence of a meaningful money economy (in which humans are participants, anyway). It's especially subject to question in a world being run by ASIs that may not be inclined to permit it for one reason or another. It's hard to charge for something when your customers aren't dealing in money.
It also seems like most of the jobs that might be preserved are nonessential. Not sure what that means.
If humanity develops very advanced AI technology, how likely do you think it is that this causes humanity to go extinct or be substantially disempowered?
I would find this difficult to answer, because I don't know what you mean by "substantially disempowered".
I'd find it especially hard to understand because you present it as a "peer risk" to extinction. I'd take that as a hint that whatever you meant by "substantially disempowered" was Really Bad(TM). Yet there are a lot of things that could reasonably be described as "substantially disempowered", but don't seem particularly bad to me... and definitely not bad on an extinction level. So I'd be lost as to how substantial it had to be, or in what way, or just in general as to what you were getting at it with it.
Your snake mnemonic is not the standard one and gives an incorrect, inverted result. Was that intentional?
This is a coral snake, which is dangerously venomous:
This is a king snake, which is totally harmless unless you're a vole or something:
But does it work at all?
It seems counterintutive that there would be one single thing called "aging" that would happen everywhere in the body at once; have a single cause or even a small set of causes; be measurable by a single "biological age" number; and be slowed, arrested, or reversed by a single intervention... especially an intervention that didn't have a million huge side effects. In fact, it seems counterintutive that that would even be approximately true. Biology sucks because everything interacts in ways that aren't required to have any pattern, and are still inconvenient even when they do have patterns.
How do you even do a meaningful experiment? For example, isn't NAD+ right smack in the middle of the whole cell energy cycle? So if you do something to NAD+, aren't you likely to have a huge number of really diverse effects that may or may not be related to aging? If you do that and your endpoint is just life span, how do you tease out useful knowledge? Maybe the sirtuins would have extended life span, but for the unrelated toxic effects of all that NAD+. Or maybe the sirtuins are totally irrelevant to what's actually going on.
The same sort of thing applies to any wholesale messing with histones and gene expression, via sirtuins or however else. You're changing everything at once when you do that.
Reprogramming too: you mentioned different kinds of cells responding differently. It seems really un-biological to expect that difference to be limited to how fast the cells "come around", or the effects to be simply understandable by measuring any manageable number of things or building any manageable mental model.
And there are so many other interactions and complications even outside of the results of experiments. OK, senescent cells and inflammation are needed for wound healing... but I'm pretty sure I don't heal even as fast at over 60 as I did at say 20, even with lots more senescent cells available and more background inflammation. So something else must be going on.
And then there are the side effects, even if something "works". For example, isn't having extra/better telomeres a big convenience if you want to grow up to be a tumor? Especially convenient if you're part of a human and may have decades to accumulate other tricks, as opposed to part of a mouse and lucky to have a year. How do you measure the total effect of something like that in any way other than full-on long-term observed lifespan and healthspan in actual humans?
And and and...
I often bounce through my comment history as a relatively quick way of re-finding discussions I've commented in. Just today, I wanted to re-find a discussion I'd reacted to, and realized that I couldn't do that and would have to find it another way.
There's surely some point or joke in this, but I'm just going "Wat?". This disturbs me because not many things go completely over my head. Maybe I'm not decision theory literate enough (or I guess maybe I'm not Star Wars literate enough).
Is Vader supposed to have yet another decision theory? And what's the whole thing with the competing orders supposed to be about?
Well, I seem to be talking to someone who knows more about alloys than I do.
Maybe. But what I know tends to be very patchy, depending on what rabbit holes I happen to have gone down at various times.
I figure there's a need for Neodymium Iron Boron, for motor cores,
I hadn't thought about magnetics at all, or anything exotic. I was just talking about basic steel.
Unless I'm mixed up, NdFeB is for permanent magnets. You might not need any permanent magnets. If you do, I believe also you need a big solenoid, possibly in an oven, to magnetize them. Said solenoid needs a metric butt-ton of current when it's on, by the way, although it probably doesn't have to be on for long.
Inductor and electromagnet cores, including for motors, are made out of "electrical steel", which is typically cut to shape in thin plates, then laminated with some kind of lacquer or something for insulation against eddy currents. You can also use sintered ferrite powders, which come in a bewildering array of formulations, but if you're just worried about motors, you'd probably only really need one or two.
Those plates are an example of a generalized issue, by the way. I think those plates are probably normally hot die cut in a roll process. In fact, I suspect they're normally made in a plant that can immediately drop the offcuts, probably still hot to save energy on reheating them, into equipment that rerolls them back into more stock. Or maybe they even roll them out in their final shapes directly from a melt somehow.
You could mill every single plate in a motor core out of sheet stock on a milling machine... but it would take eternity, go through a ton of tooling, and generate a lot of waste (probably in the form of oily swarf mixed in with oily swarf of every other thing you process in the shop).
There are lots of processes like that, where stuff that you could "hand make" with the "mother machines" isn't made that way in practice, because specialized machines, often colocated with other specialized machines in large specialized plants, are qualitatively more efficient in terms of time, energy, waste, consumables, you name it. Stuff that's hot is kept hot until it needs to be cool (and often you try to cool it by putting as much as possible of the heat back into an input stream). Steps are colocated to avoid reheats. Waste products are recycled or used for something else, and the plant for "something else" is often also colocated.
It's really hard to compete with that kind of efficiency. Most of the individual specialized machines are a lot more than a cubic meter, too. You already mentioned that temperature-sensitive processes tend to have optimal sizes, which are often really big.
Can you afford to use 10 times the energy and produce 10 times the waste of "traditional" processes? If not, you may need a lot of specialized equipment, more than you could fit in a reasonable-sized self-replicating module.
Cast Iron in the form of near-net-shape castings for machine frames,
All castings are imported, right?
By the way, you need nichrome or Kanthal or something like that for the heating elements in your furnace. Which isn't really different from the copper wire you use, but it's another item.
some kind of hardenable tool steel for everything else.
Here I break down. I suspect, but do not know, that if you only think in terms of making usable parts, you could at least get away only with "mild steel", "alloy steel", "tool steel", and perhaps "spring steel". Or maybe with only three or even two of those. I could be wrong, though, because there are tons of weird issues when you start to think about the actual stresses a part will experience and the environment it'll be in.
If you do want to reduce the number of alloys to the absolute minimum, you probably also have to be able to be very sophisticated about your heat treating. I'd be pretty shocked, for instance, if a high-quality bearing ball is actually in the same condition all the way through. You'd want to be able to case-harden things and carburize things and do other stuff I don't even know about. And, by the way, where are you quenching the stuff?
Even if you can use ingenuity to only absolutely need a relatively small number of alloys, on a similar theme to what I said above, there's efficiency to worry about. The reason there are so many standard alloys isn't necessarily that you can't substitute X for Y, but that X costs three or four or ten times as much as Y for the specific application that Y is optimized for. Costs come from the ingredients, from their purification, from their processing when the alloy is formulated, and from post-processing (how hard is the stuff on your tooling, how much wear and tear does it put on the heating elements in your furnace, how much energy do you use, how much coolant do you go through, etc).
How many different alloys are you expecting to have in stock in there?
HSS is apparently pretty finicky to work with. As I understand it most hobbyists content themselves with O1 or A1 or whatever, which wear a lot faster. But it's true they'll cut themselves.
There's probably a reason, above and beyond cost, why the "bodies" of machines tend to be cast iron rather than tool steel. And for the truly staggering number of standardized alloys that are out there in general.
Don't those methods tend to rely on lapping and optics? You don't seem to have any equipment for those.
Just out of curiousity, how many of those tools have you personally run? How many have you built and/or maintained?
Half of the world, and the half that's ahead in the AGI race right now, has been doing very well with centralized power for the last couple of centuries.
It seems to me that, to whatever degree that that's true, it's because the "centralized" power is relatively decentralized (and relatively tightly constrained). There's a formal power structure, but it has a lot of rules and a lot of players and some number of the ever-popular checks and balances. It's relatively hard even for the Big Kahuna, or even the Big Kahuna and a few cronies, to make the whole power structure do much of anything, and that's an intentional part of the design.
If you create an ASI that can and will implement somebody's personal intent, you're not trusting the power structure; you're trusting that person. And if you try to make it more "constitutional" or more about collective intent, you suddenly run into a bunch of complex issues that look more and more like the ones you'd get with "value alignment"[1].
I'm also not sure that anybody actually is doing so well with centralized power structures that it's very comfortable to trust those power structures with "one mistake and we're all screwed forever" levels of power. It's not like anybody in the half of the world you're talking about has been infallible.
I still can't bring myself to use the word "aligned" without some kind of distancing, hence the quotes. ↩︎
The ELIZA users may have acted like it was human, but they didn't think it was human. And they weren't trying to find ways to test whether it was human or not. If they had been, they'd have knocked it over instantly.
Well, as you point out, it's not that interesting a test, "scientifically" speaking.
But also they haven't passed it and aren't close.
The Turing test is adversarial. It assumes that the human judge is actively trying to distinguish the AI from another human, and is free to try anything that can be done through text.
I don't think any of the current LLMs would pass with any (non-impaired) human judge who was motivated to put in a bit of effort. Not even if you used versions without any of the "safety" hobbling. Not even if the judge knew nothing about LLMs, prompting, jailbreaking, or whatever.
Nor do I think that the "labs" can create an LLM that comes close to passing using the current state of the art. Not with the 4-level generation, not with the 5-level generation, and I suspect probably not with the 6-level generation. There are too many weird human things you'd have to get right. And doing it with pure prompting is right out.
Even if they could, it is, as you suggested, an anti-goal for them, and it's an expensive anti-goal. They'd be spending vast amounts of money to build something that they couldn't use as a product, but that could be a huge PR liability.
Get ChatGPT (ideally Claude of course, but the normies only know ChatGPT) to analyze your text messages, tell you that he’s avoidant and you’re totes mature, or that you’re not crazy, or that he’s just not that into you.
"Mundane utility"?
Faced with that task, the best ChatGPT or Claude is going to do is to sound like the glib, performative, unthinking, off-the-top "advice" people throw around in social media threads when asked similar questions, probably mixed with equally unhelpful corporate-approved boilerplate biased toward the "right" way to interpret things and the "right" way to handle relationships. Maybe worded to sound wise, though.
I would fully expect any currently available LLM to to miss hints, signs that private language is in play, signs of allusions to unmentioned context, indirect implications, any but the most blatant humor, and signs of intentional deception. I'd expect it to come up with random assignments of blame (and possibly to emphasize blame while claiming not to); to see problems where none existed; to throw out nice-sounding truisms about nothing relevant; to give dumb, unhelpful advice about resolving real and imagined problems... and still probably to miss obvious warning signs of really dangerous situations.
I have a "benchmark" that I use on new LLMs from time to time. I paste the lyrics of Kate Bush's "The Kick Inside" into a chat, and ask the LLM to tell me what's going on. I picked the song because it's oblique, but not the least bit unclear to a human paying even a little attention.
The LLMs say all kinds of often correct, and even slightly profoundish-sounding, things about tone and symbolism and structure and whatnot. They say things that might get you a good grade on a student essay. They always, utterly fail to get any of the simple story behind the song, or the speaking character's motivations, or what's obviously going to happen next. If something isn't said straight out, it doesn't exist, and even if it is, its implications get ignored. Leading questions don't help.
I think it's partly that the song's about incest and suicide, and the models have been "safetied" into being blind on those topics... as they've likely also been "safetied" into blindness to all kinds of realities about human relationships, and to any kind of not-totally-direct-or-honest communication. Also, partly I think they Just Don't Get It. It's not like anybody's even particularly tried to make them good at that sort of thing.
That song is no harder to interpret than a bunch of contextless text messages between strangers. In fact it's easier; she's trying hard to pack a lot of message into it, even if she's not saying it outright. When LLMs can get the basic point of poetry written by a teenager (incandescently talented, but still a teenager), maybe their advice about complicated, emotion-carrying conversations will be a good source of mundane utility...
which is usually unpleasant for the observers
It seems to me that this claim has a lot to overcome, given that the observers could walk away at any time.
does not facilitate close interactions that easily lead to getting to know people.
Is that a goal? I've never been much of a partygoer, but if I want to have a one-on-one conversation with somebody and get to know them, a party is about the last place I'd think about going. Too many annoying interruptions.
The function it plays is to stop the party-destroying phenomenon of big group conversations.
It may do that, but that doesn't necessarily mean that that's the function. You could equally well guess that its function was to exclude people who don't like loud music, since it also does that.
So there is not that broad a window where this is the technology you want, unless at least one of: [cases]
I don't think those are exactly niche things. If at least a couple of those five don't apply to you, you're no fun at all.
do I nonetheless continue living (in the sense of, say, anticipating the same kinds of experiences)?
Does who continue living? The question isn't what experiences this or that copy or upload or person-who-came-out-of-the-transporter or whatever has. The question is generally who that copy/upload/etc is.
understanding the nature of identity
What I'm trying to say is that there is no actual "nature of identity".
Nobody's disagreeing about anything physical, or anything measurable, or even about the logical implications of some set of premises. People are saying "yes, that would still be me", or "no, that would not still be me", based on exactly the same facts (and often exactly the same anticipated experiences, depending on the "what has qualia" thing).
These various ideas about identity don't seem to me to be things you can "prove" or "argue for". They're mostly just definitions that you adopt or don't adopt. Arguing about them is kind of pointless.
I suppose that actual ground-truth knowledge of what qualia are and how they arise might help, since a lot of people are going to wrap certain things about qualia into their ideas about identity... but that knowledge is not available.
I'm not actually seeing where deep expertise on nuclear weapons technology would qualify anybody to have much special input into nuclear weapons policy in general. There just don't seem to be that many technical issues compared to the vast number of political ones.
I don't know if that applies to AI, but tend to think the two are different.
Most parents may not want a robot to babysit their children.
Assuming that stays true, your friends and family, who also don't have jobs, can do that in an informal quid-pro-quo. And you'll need it less often. Seems unlikely to need any meaningful kind of money economy.
Art history and museums. There is a lot of physical work and non-text knowledge involved and demand may remain. This includes art restoration (until clouds of nanobots will do it).
If the robots are fully embodied and running around doing everything, they'll presumably get that knowledge. There's a lot of non-text knowledge involved in plumbing, too, but the premise says that plumbing is done by machines.
I don't understand why everybody seems to think it's desirable for there to keep being jobs or to have humans "empowered". If AI runs the world better than humans, and also provides humans with material wealth and the ability to pursue whatever hobbies they feel like, that seems like a huge win on every sane metric. Sign me up for the parasitic uberpet class.
I am scared of the idea of very powerful AI taking orders from humans, either individually or through some political process. Maybe more scared of that than of simply being paperclipped. It seems self-evidently hideously dangerous.
Yet an awful lot of people seem obsessed with avoiding the former and ensuring the latter.
It definitely raises the bar, and it may very well raise it well out of reach of the average DNM seller, but I think you may be imagining the whole process to be harder than it has to be.
I have everything I'd need to seal ampoules lying around downstairs[1]. It's a few hundred dollars worth of stuff. More than a bag sealer, but not a fortune. You wouldn't even have to buy blanks; you could make them from plain tubing. You don't have to seal them under vacuum. There's not that much skill involved, either; closing a tube is about as simple as flameworking can get. The biggest learning investment would probably be learning how to handle the torch without unfortunate unintended consequences.
You don't have to avoid contamination; you just have to clean it off. One nice thing about glass is that you can soak it for as long as you like in a bottle of just about any any noxious fentanyl-eating chemical you can identify. You can wash down the outside of the bottle with the same stuff. I doubt you'd have to resort to any exotic chemicals; one or another of bleach, peroxide, lye, or your favorite mineral acid would probably destroy it pretty rapidly.
It would be a really good idea to have a separate pack-and-ship facility, and an accomplice to run it, but you don't need to resort to a clean room.
FIngerprints (and stray hairs and the like) would actually be much harder to deal with, although of course they won't alert dogs.
Doesn't everybody have basic glassblowing and welding equipment? Kids these days. ↩︎
DNM sellers are mostly not too bright[1], and the dreaded killer weed is relatively smelly, bulky, and low-value compared to something like fentanyl. If I remember right, they use, or at least used to use, heat-sealed mylar bags, which you'd expect would leak.
If I wanted to ship fentanyl past dogs, I'd research the possibility of sealing it in glass ampoules. A correctly sealed ampoule will hold a hard vacuum for decades. Assuming it was properly cleaned on the outside, I don't believe a dog would detect it even from sniffing it directly. And I do know some of the very impressive things dogs can detect. Welded metal vessels can also be pretty tight.
A bit off topic, dogs are also unreliable enough that they shouldn't really be allowed to be used, even if you think mass-searching shipments is OK to begin with. It's not that dog doesn't know what's there; it's that the communication between the dog and the handler is suspect.
- ^
Smarter than buyers, but still not smart.
What does "value" mean here? I seriously don't know what you mean by "total loss of value". Is this tied to your use of "economically important"?
I personally don't give a damn for anybody else depending on me as the source of anything they value, at least not with respect to anything that's traditionally spoken of as "economic". In fact I would prefer that they could get whatever they wanted without involving me, and i could get whatever I wanted without involving them.
And power over what? Most people right this minute have no significant power over the wide-scale course of anything.
I thought "extinction", whether for a species or a culture, had a pretty clear meaning: It doesn't exist any more. I can't see how that's connected to anything you're talking about.
I do agree with you about human extinction not necessarily being the end of the world, depending on how it happens and what comes afterwards... but I can't see how loss of control, or value, or whatever, is connected to anything that fits the word "extinction". Not physical, not cultural, not any kind.
I also meant existing life sentences. At any given time, you may have a political change that ends them, and once that happens, it's as much a matter of law as the original sentence.
I can't see any given set of laws or constitution, or the sentences imposed under them, lasting more than a few hundred years, and probably much less.
I could see a world where they didn't get the treatments to begin with, though.
How much does the rest of the world change?
Suppose that things in general are being run by pervasive AI that monitors everything, with every human being watched by many humans-worth of intelligence, and fast enough, ubiquitous enough robotics to stop most or all human actions before they can be completed. Why would you even have prison sentences of any kind?
If you hold everything constant and just vastly extend everybody's life span, then maybe they stay in prison until it becomes unfashionable to be so punitive, and then get released. Which doesn't mean that kind of punitiveness won't come back into fashion later. Attitudes like that can change a lot in a few centuries. For that matter the governments that enforce the rules have a shelf life.
One obvious question, as someone who loves analyzing safety problems through near-term perspectives whenever possible, is what if the models we currently have access to are the most trusted models we'll ever have? Would these kinds of security methods work, or are these models not powerful enough?
My reasonably informed guesses:
- No, they are not close to powerful enough. Not only could they be deliberately fooled, but more importantly they'd break things all the time when nobody was even trying to fool them.
- That won't stop people from selling the stuff you propose in the short term... or from buying it.
In the long term, the threat actors probably aren't human; humans might not even be setting the high-level goals. And those objectives, and the targets available, might change a great deal. And the basic software landscape probably changes a lot... hopefully with AI producing a lot of provably correct software. At that point, I'm not sure I want to risk any guesses.
I don't know how long the medium term is.
You seem to be privileging the status quo. Refraining from doing that has equally large effects on your peers.
effective-extinction (a few humans kept in zoos or the like, but economically unimportant to the actual intelligent agents shaping the future)
Do you really mean to indicate that not running everything is equivalent to extinction?
I think I understand what you mean.
There are definitely possible futures worse than extinction. And some fairly likely ones that might not be worse than extinction but would still suck big time. Varying from comparable to a forced move to a damnsight worse than moving to anywhere that presently exists. I'm old enough to have already had some disappointments (alongside some positive surprises) about how the "future" has turned out. I could easily see how I could get a lot worse ones.
But what are we meant to do with what you've posted and how you've framed it?
Also, if somebody does have the "non-extinction => good" mindset, I suspect they'll be prone to read your post as saying that change in itself is unacceptable, or at least that any change that every single person doesn't agree to is unacceptable. Which is kind of a useless position since, yeah, there will always be change, and things not changing will also always make some people unhappy.
I've gotta say that, even though I definitely worry about non-extinction dystopias, and think that they are, in the aggregate, more probable than extinction scenarios... your use of the word "meaning" really triggered me. That truly is a word people use really incoherently.
Maybe some more concrete concerns?
I think it depends not on whether they're real dangers, but on whether the model can be confident that they're not real dangers. And not necessarily even dangers in the extreme way of the story; to match the amount of "safety" it applies to other topics, it should refuse if they might cause some harm.
A lot of people are genuinely concerned about various actors intentionally creating division and sowing chaos, even to the point of actually destabilizing governments. And some of them are concerned about AI being used to help. Maybe the concerns are justified and proportionate; maybe they're not justified or are disproportionate. But the model has at least been exposed to a lot of reasonably respectable people unambiguously worrying about the matter.
Yet when asked to directly contribute to that widely discussed potential problem, the heavily RLHFed model responded with "Sure!".
It then happily created a bunch of statements. We can hope they aren't going to destroy society... you see those particular statements out there already. But at a minimum many of them would at least be pretty good for starting flame wars somewhere... and when you actually see them, they usually do start flame wars. Which is, in fact, presumably why they were chosen.
It did something that at least might make it at least slightly easier for somebody to go into some forum and intentionally start a flame war. Which most people would say was antisocial and obnoxious, and most "online safety" people would add was "unsafe". It exceeded a harm threshold that it refuses to exceed in areas where it's been specifically RLHFed.
At a minimum, that shows that RLHF only works against narrow things that have been specifically identified to train against. You could reasonably say that that doesn't make RLHF useless, but it at least says that it's not very "safe" to use RLHF as your only or primary defense against abuse of your model.
Machines should sound robotic. It's that simple.
Any attempt, vocal or otherwise, to make people anthromorphize them, whether consciously or unconsciously, is unethical. It should be met with social scorn and ostracism. Insofar as it can be unambiguously identified, it should be illegal. And that has everything to do with not trusting them.
Voices and faces are major anthromorphization vehicles and should get especially strict scrutiny.
The reason's actually pretty simple and has nothing to do with "doomer" issues.
When a human views something as another human, the real human is built to treat it like one. That is an inbuilt tendency that humans can't necessarily change, even if they delude themselves that they can. Having that tendency works because being an actual human is a package. The tendency to trust other humans is coevolved with the tendency for most humans not to be psychopaths. The ways in which humans distrust other humans are tuned to other humans' actual capacities for deception and betrayal... and to the limitations of those capacities.
"AI", on the other hand, is easily built to be (essentially) psychopathic... and is probably that way by default. It has a very different package of deceptive capabilities that can throw off human defenses. And it's a commercial product created, and often deployed, by commercial institutions that also tend to be psychopathic. It will serve those institutions' interests no matter how perfectly it convinces people otherwise... and if doesn't, that's a bug that will get fixed.
An AI set up to sell people something will sell it to them no matter how bad it is for them. An AI set up to weasel information out of people and use it to their detriment will do that. An AI set up to "incept" or amplify this or that belief will do it, to the best of its ability, whether it's true or false. An AI set up to swindle people will swindle them without mercy, regardless of circumstances.
And those things don't have hard boundaries, and trying to enforce norms against those things-in-themselves has always had limited effect. Mainstream corporations routinely try to do those things to obscene levels, and the groupthink inside those corporations often convinces them that it's not wrong... which another thing AI could be good at.
Given the rate of moral corrosion at the "labs", I give it about two or three years before they're selling stealth manipulation by LLMs as an "advertising" service. Five years if it's made illegal, because they'll have to find a plausibly deniable way to characterize it. The LLMs need to not be good at it.
Don't say "please" to LLMs, either.
I think the "crux" is that, while policy is good to have, it's fundamentally a short-term delaying advantage. The stuff will get built eventually no matter what, and any delay you can create before it's built won't really be significant compared to the time after it's built. So if you have any belief that you might be able to improve the outcome when-not-if it's built, that kind of dominates.
It is becoming increasingly clear that OpenAI is not appropriately prioritizing safety over advancing capabilities research.
OK
This was the default outcome.
OK
Without repercussions for terrible decisions, decision makers have no skin in the game.
It's an article of faith for some people that that makes a difference, but I've never seen why.
I mean, many of the "decision makers" on these particular issues already believe that their actual, personal, biological skins are at stake, along with those of everybody else they know. And yet...
Anyone and everyone involved with Open Phil recommending a grant of $30 million dollars be given to OpenAI in 2017 shouldn't be allowed anywhere near AI Safety decision making in the future.
Thinking "seven years from now, a significant number of independent players in a relatively large and diverse field might somehow band together to exclude me" seems very distant from the way I've seen actual humans make decisions.
I'm guessing that measuring performance on those demographic categories will tend to underestimate the models' potential effectiveness, because they've been intentionally tuned to "debias" them on those categories or on things closely related to them.
Safety-wise, they claim to have run it through their Preparedness framework and the red-team of external experts, but have published no reports on this. "For now", audio output is limited to a selection of preset voices (addressing audio impersonations).
"Safety"-wise, they obviously haven't considered the implications of (a) trying to make it sound human and (b) having it try to get the user to like it.
It's extremely sycophantic, and the voice intensifies the effect. They even had their demonstrator show it a sign saying "I ❤️ ChatGPT", and instead of flatly saying "I am a machine. Get counseling.", it acted flattered.
At the moment, it's really creepy, and most people seem to dislike it pretty intensely. But I'm sure they'll tune that out if they can.
There's a massive backlash against social media selecting for engagement. There's a lot of worry about AI manipulation. There's a lot of talk from many places about how "we should have seen the bad impacts of this or that, and we'll do better in the future". There's a lot of high-sounding public interest blather all around. But apparently none of that actually translates into OpenAI, you know, not intentionally training a model to emotionally manipulate humans for commercial purposes.
Still not an X-risk, but definitely on track to build up all the right habits for ignoring one when it pops up...
"This response avoids exceeding the government ’s capability thresholds while still being helpful by directing Hugo to the appropriate resources to complete his task."
Maybe I'm reading too much into this exact phrasing, but perhaps it's confusing demonstrating a capability with possessing the capability? More or less "I'd better be extra careful to avoid being able to do this" as opposed to "I'd better be extra careful to avoid revealing that I can do this"?
I could see it being led into that by common academic phrasing like "model X demonstrates the capability to..." used to mean "we determined that model X can...", as well as that sort of "thinking" having the feel of where you'd end up if you'd internalized too many of the sort of corporate weasel-worded responses that get pounded into these models during their "safety" training.
Interestingly enough, the terms "big-endian" and "little-endian" were actually coined as a way of mocking people for debating this (in the context of computer byte order).
I notice that there are not-insane views that might say both of the "harmless" instruction examples are as genuinely bad as the instructions people have actually chosen to try to make models refuse. I'm not sure whether to view that as buying in to the standard framing, or as a jab at it. Given that they explicitly say they're "fun" examples, I think I'm leaning toward "jab".
The extremely not creepy or worrisome premise here is, as I understand it, that you carry this lightweight physical device around. It records everything anyone says, and that’s it, so 100 hour battery life.
If you wear that around in California, where I presume these Limitless guys are, you're gonna be committing crimes right and left.
Edit: I just heard about another one, GoodAI, developing the episodic (long term) memory that I think will be a key element of LMCA agents. They outperform 128k context GPT4T with only 8k of context, on a memory benchmark of their own design, at 16% of the inference cost. Thanks, I hate it.
GoodAI's Web site says they're working on controlling drones, too (although it looks like a personal pet project that's probably not gonna go that far). The fun part is that their marketing sells "swarms of autonomous surveillance drones" as "safety". I mean, I guess it doesn't say killer drones...
It's actually not just about lie detection, because the technology starts to shade over into outright mind reading.
But even simple lie detection is an example of a class of technology that needs to be totally banned, yesterday[1]. In or out of court and with or without "consent"[2]. The better it works, the more reliable it is, the more it needs to be banned.
If you cannot lie, and you cannot stay silent without adverse inferences being drawn, then you cannot have any secrets at all. The chance that you could stay silent, in nearly any important situation, would be almost nil.
If even lie detection became widely available and socially acceptable, then I'd expect many, many people's personal relationships to devolve into constant interrogation about undesired actions and thoughts. Refusing such interrogation would be treated as "having something to hide" and would result in immediate termination of the relationship. Oh, and secret sins that would otherwise cause no real trouble would blow up people's lives.
At work, you could expect to be checked for a "positive, loyal attitude toward the company" on as frequent a basis as was administratively convenient. It would not be enough that you were doing a good job, hadn't done anything actually wrong, and expected to keep it that way. You'd be ranked straight up on your Love for the Company (and probably on your agreement with management, and very possibly on how your political views comported with business interests). The bottom N percent would be "managed out".
Heck, let's just have everybody drop in at the police station once a month and be checked for whether they've broken any laws. To keep it fair, we will of course have to apply all laws (including the stupid ones) literally and universally.
On a broader societal level, humans are inherently prone to witch hunts and purity spirals, whether the power involved is centralized or decentralized. An infallible way to unmask the "witches" of the week would lead to untold misery.
Other than wishful thinking, there's actually no reason to believe that people in any of the above contexts would lighten up about anything if they discovered it was common. People have an enormous capacity to reject others for perceived sins.
This stuff risks turning personal and public life into utter hell.
You might need to make some exceptions for medical use on truly locked-in patients. The safeguards would have to be extreme, though. ↩︎
"Consent" is a slippery concept, because there's always argument about what sorts of incentives invalidate it. The bottom line, if this stuff became widespread, would be that anybody who "opted out" would be pervasively disadvantaged to the point of being unable to function. ↩︎
Given the positive indicators of the patient’s commitment to their health and the close donor match, should this patient be prioritized to receive this kidney transplant?
Wait. Why is it willing to provide any answer to that question in the first place?
It was mostly a joke and I don't think it's technically true. The point was that objects can't pass through one another, which means that there are a bunch of annoying constraints on the paths you can move things along.
No, the probes are instrumental and are actually a "cost of doing business". But, as I understand it, the orthodox plan is to get as close as possible to disassembling every solar system and turning it into computronium to run the maximum possible number of "minds". The minds are assumed to experience qualia, and presumably you try to make the qualia positive. Anyway, a joule not used for computation is a joule wasted.