Posts
Comments
Adversarial gaming doesn't match my experience much at all and suggesting options doesn't feel imposing either. For me at least, it's largely about the responsibility and mental exertion of planning.
In my experience, mutual "where do you want to go" is most often when neither party has a strong preference and neither feels like taking on the cognitive burden of weighing options to come to a decision. Making decisions takes effort especially when there isn't a clearly articulated set of options and tradeoffs to consider.
For practical purposes, one person should provide 2-4 options they're OK with and the other person can pick one option or veto some option(s). If they veto all given options, they must provide their own set of options the first person can choose or veto. Repeat as needed but rarely is more than one round needed unless participants are picky or disagreeable.
Why do you think that the same federal bureaucrats who incompetently overregulate other industries will do a better job regulating AI?
Chevron deference means that judges defer to federal agencies instead of interpreting the laws themselves where the statute is ambiguous. It's not so much a question of overregulation vs underregulation as it is about who is doing the interpretation. For example, would you rather the career bureaucrats in the Environmental Protection Agency determine what regulations are appropriate to protect drinking water or random judges without any relevant expertise?
One consequence of blowing up Chevron deference is that one activist judge in Texas can unilaterally invalidate FDA approval of a drug like mifepristone for the entire country that's been safe, effective, and available on the markets for decades by substituting his own idiosyncratic opinion instead of deferring to the regulatory agency whose entire purpose is to make those kinds of determinations.
Government agencies aren't always competent but the alternative is a patchwork of potentially conflicting decisions from judges ruling outside of their area of expertise.
If you're open to more experimentation, I'd recommend trying playing against Leela Chess Zero using some of the newer contempt parameters introduced in this PR and available in the latest pre-release version. I'm really curious if you'd notice significant style differences with different contempt settings.
"Artificial Intention" doesn't sound catchy at all to me, but that's just my opinion.
Personally, I prefer to think of the "Alignment Problem" more generally rather than "AI Alignment". Regardless of who has the most power (humans, AI, cyborgs, aliens, etc.) and who has superior ethics, conflict arises when participants in a system are not all aligned.
This is the most similar question that I could find that already exists.
https://www.metaculus.com/questions/7384/alien-tech-in-solar-system-before-2030/
There are still HR and legal overhead costs involved if you have human operators.
I think part of the answer is also that the space of things low-wage workers can physically do remotely via teleoperation isn't that much larger than the space of things that can be fully automated but still much smaller than the space of things a local human can do. It's a fairly narrow band to exploit, IMO, and the labor cost arbitrage spread is rarely worth the complexity of the extra logistics, capital investment, and maintenance.
As @kave pointed out already, the right side being thicker would make it more clear that it's specifically a delta.
Something like one of these?
https://thenounproject.com/icon/bored-251902/ https://thenounproject.com/icon/annoyed-4979573/
Embodied tasks just aren't an area where comparison makes much sense yet. What kind of perception tasks did you have in mind?
Does this "paradox" still hold in the era of recent multimodal AI? In particular, what are some things that are easy for humans but hard for AI, other than things requiring embodiment? What areas are human mechanical Turks still much better at? (I believe there are areas but pretty fuzzy about what they are.)
I really like the t-AGI concept for a specific task or benchmark but find it difficult to generalize since capabilities are not uniform. How do you classify an AGI that is 1-second AGI at some tasks but 1-day AGI at others (e.g. due to serial processing speed advantages)?
There is indeed the Myth of the flat Earth that is a misconception about the beliefs of scholars in the Middle Ages and some scholars certainly understood the concept of a spherical Earth since at least Eratosthenes. I'm referring to earlier history like ancient Egyptian, Chinese, and pre-Socratic Greek cosmologies. Admittedly, it's not a great example since most of the debates about it are lost to history, and such debates wouldn't involve the same kind of reasoning and evidential standards we use today.
Yes, clearly the less time the human has, the better Leela will do relatively. One thing to note though is that Lichess Elo isn't completely comparable across different time controls. If you look at the player leaderboard, you can see that the top scores for bullet are ~600 greater than for classical, so scores need to be interpreted in context.
Self-Elo inflation is a fair point to bring up and I don't have information on how well it translates.
791556 is nowhere near the strongest network available. It's packaged with lc0 as a nice small net. The BT2 net currently playing at tcec-chess.com is several hundreds of Elo stronger than T79 and likely close to superhuman level, depending on the time control. It's not the very latest and greatest, but it is publicly available for download and should work with the 0.30.0-rc1 pre-release version of lc0 that supports the newer transformer architecture if you want to try it yourself. If you only want completely "official" nets, at least grab one of the latest networks from the main T80 run.
I'm not confident that BT2 is strictly superhuman using pure policy but I'm pretty sure it's at least close. LazyBot is a Lichess bot that plays pure policy but uses a T80 net that is likely at least 100 Elo weaker than BT2.
Historical examples of things that once sounded ridiculous but turned out to be true:
- Spherical Earth (vs Flat Earth)
- Heliocentrism (vs Geocentrism)
- Germ theory (vs e.g. Miasmatic theory)
- Evolution via natural selection
- Quantum mechanics
- Relativity
- Plate tectonics
- Black holes
It's harder to know what qualifies as false examples since they do (now) have good counterarguments, but maybe something like these:
- Phlogiston theory
- Vitalism
- Luminiferous aether
- Lamarckian evolution
- Cold fusion
- Steady State cosmology (vs Big Bang)
- Caloric theory
- Spontaneous generation
Examples of ideas with less certain status:
- String theory / quantum gravity / unified physics
- Multiverse hypothesis / simulation hypothesis
- Existence and nature of extraterrestrial life
- Nature of dark matter & dark energy
- Epigenetic roles in disease and inheritance
- Origins of life / abiogenesis / panspermia
- Nature of consciousness and reality
LLMs are truly generating novel ideas, they are just interpolating between existing ideas via memorized statistical patterns.
Is this intended to be "LLMs are not truly generating novel ideas"?
I don’t know much about Leela Zero and Katago but I do know that Leela Chess Zero (lc0) without search (pure policy) is near superhuman levels. I’ll see if I can dig up more precise specifics.
I'm not interested in the prize, but as long as we're spot-checking, this paragraph bothered me:
It turns out that spreading out the communication flow rate budget over a huge memory store with a slow clock rate is fundamentally more powerful than a fast clock rate over a small memory store. One obvious reason: learning machines have a need to at least store their observational history. A human experiences a sensory input stream at a bitrate of about 10^6 bps (assuming maximal near-lossless compression) for about 10^9 seconds over typical historical lifespan, for a total of about 10^15 bits. The brain has about 2∗10^14 synapses that store roughly 5 bits each, for about 10^15 bits of storage. This is probably not a coincidence.
The idea of bitrate * lifespan = storage capacity makes sense for a VHS or DVD but human memory is completely different. Only a tiny percentage of all our sensory input stream makes it through memory consolidation into long-term memory. Sensory memory is also only one type of memory alongside others like semantic, episodic, autobiographical, and procedural (this list is neither mutually exclusive nor collectively exhaustive). Brains are very good at filtering the sensory stream to focus on salient information and forgetting the vast majority of the rest. This results in memory that is highly compressed (and highly lossy).
Because brain memory is so different than simple storage, this napkin math is roughly analogous to computing body mass from daily intake of air, water, and food mass and neglecting how much actually gets stored versus ultimately discarded. You can do the multiplication but it's not very meaningful without additional information like retention ratios.
Have the distributed architecture trends and memristor applications followed the rough path you expected when you wrote this 12 years ago? Is this or this the sort of thing you were gesturing at? Do you have other links or keywords I could search for?
Biology is incredibly efficient at certain things that happen at the cell level. To me, it seems like OP is extrapolating this observation rather too broadly. Human brains are quite inefficient at things they haven't faced selective pressure to be good at, like matrix multiplication.
Claiming that human brains are near Pareto-optimal efficiency for general intelligence seems like a huge stretch to me. Even assuming that's true, I'm much more worried about absolute levels of general intelligence rather than intelligence per Watt. Conventional nuclear bombs are dangerous even though they aren't anywhere near the efficiency of a theoretical antimatter bomb. AI "brains" need not be constrained by the size and energy constraints of a human brain.
Similar to humans, LLMs can do 6-digit multiplication with sufficient prompting/structure!
https://www.lesswrong.com/posts/XvorpDSu3dwjdyT4f/gpt-4-multiplication-competition
The Bailey-Borwein-Plouffe formula is a nice one.
The big one is obviously "make long time scale plans to navigate a complicated 3D environment, while controlling a floppy robot."
This is like judging a dolphin on its tree-climbing ability and concluding it's not as smart as a squirrel. That's not what it was built for. In a large number of historically human domains, GPT-4 will dominate the village idiot and most other humans too.
Can you think of examples where it actually makes sense to compare GPT and the village idiot and the latter easily dominates? Language input/output is still a pretty large domain.
Readers of this may also be interested in this post from 2015:
I don't have an inside view. If I did, that would be pretty powerful capabilities information.
I'm pointing at the possibility that we already have more than sufficient resources for AGI and we're only separated from it by a few insights (a la transformers) and clever system architecture. I'm not predicting this is true just that it's plausible based on existing intelligent systems (humans).
Epistemic status: pondering aloud to coalsce my own fuzzy thoughts a bit
I'd speculate that the missing pieces are conceptually tricky things like self-referential "strange loops", continual learning with updateable memory, and agentic interactions with an environment. These are only vague ideas in my mind but, for some reason, feel difficult to solve but don't feel like things that require massive data and training resources so much as useful connections to reality and itself.
Resource accumulation certainly can't grow exponentially indefinitely and I agree that RSI can't improve exponentially forever either, but it doesn't need to for AI to take over.
An AI doesn't have to get far beyond human-level intelligence to control the future. If there's sufficient algorithmic overhang, current resources might even be enough. FOOM would certainly be easier if no new hardware were necessary. This would look less like an explosion and more like a quantum leap followed by slower growth as physical reality constrains rapid progress.
I agree that humans are not drastically more intelligent than all other animals. This makes the prospect of AI even scarier, in my opinion, since it shows how powerful accumulated progress is.
I believe that human-level intelligence is sufficient for an AI to be extremely dangerous if it can scale while maintaining self-alignment in the form of "synchronized behavior and collective action". Imagine what a tech company could achieve if all employees had the same company-aligned goals, efficient coordination, in silico processing speeds, high-bandwidth communication of knowledge, etc. With these sorts of advantages, it's likely game over before it hits human-level intelligence across the board.
Until AI is at least as sample-efficient and energy-efficient as humans are at learning, there are significant algorithmic gains that are possible. This may not be possible under the current deep-learning paradigm but we know it's possible under some paradigm since evolution has already accomplished it blindly.
I do share your skepticism that something like an LLM alone could recursively improve itself quickly. Assuming FOOM, my model of how it happened has deep learning as only part of the answer. It's part of the recursive loop but is used mostly as a general heuristic module, much like the neural net of a chess engine is only a piece of the puzzle; you still need a fast search algorithm that uses the heuristics efficiently.
Agreed. Only a very weak form of orthogonality is necessary to have dangerously unaligned AI be the default.
Since you are checking your thoughts, here are a couple of posts that argue the contrary to your 1.2:
https://www.lesswrong.com/posts/MmmPyJicaaJRk4Eg2/the-limit-of-language-models
https://www.lesswrong.com/posts/mEbYbekPXoWvrHraW/human-level-ai-can-plausibly-take-over-the-world
So as a rough analogy, if you were a computer program, the conscious part of the execution would be kind of like log output from a thread monitoring certain internal states?
Funny that you should mention élan vital. The more I read about it, the more "consciousness" seems to me to be similarly incoherent and pseudoscientific as vitalism. This isn't a fringe view and I'd recommend skimming the Rejection of the Problem section of the Hard problem of consciousness page on Wikipedia for additional context. It's hard not to be confused about a term that isn't coherent to begin with.
Supposing each scenario could be definitively classified as conscious or not, would that help you make any predictions about the world?
One of the simplest ways for AI to have power over the physical world is via humans as pawns. A reasonably savvy AI could persuade/manipulate/coerce/extort/blackmail real-life people to carry out the things it needs help with. Imagine a powerful mob boss who is superintelligent, never sleeps, and continuously monitors everyone in their network.
Not many people consider GPT-4 extremely dangerous on its own. Hooking up something at that level of intelligence into a larger system with memory storage and other modules is a bit more threatening, and probably sufficient to do great harm already if wielded by malevolent actors unleashing it onto social media platforms, for example.
The real danger is that GPT-4 is a mile marker we've blown by on the road to ever more capable AI. At some point, likely before climate change becomes an existential threat, we lose control and that's when things get really weird, unpredictable, and dangerous.
Eliezer has near-zero hope for humanity's survival. I think we'd all agree that the universal agreement he suggests is not something plausible in the current world. He's not advocating for it because he believes it might happen but rather it's the only thing he thinks might be enough to give us a shot at survival.
Interesting. I wonder how much selection bias there was in responses.
The SAT in-contrast is kind of useful anyways, because kind of everyone saturates on practicing for the SAT, at least in the US, but that test sadly also maxes out at an average that is below the average IQ test result in the LessWrong community
Am I correct in interpreting this as you implying that the average member of the LessWrong community got perfect SAT scores (or would have had they taken it)?
In my opinion, GPT-4 is intelligent but needs extra infrastructure to allow memory and continuous learning to become an agentic AGI. It might not be that hard to add that infrastructure though and people are already trying: https://github.com/noahshinn024/reflexion-human-eval
OK. Let's make it even more extreme. Suppose you take a commercial flight. The likelihood of dying in a crash is on the order of 1 in 10 million. From a percent error or absolute error perspective, 99.99999% isn't that different from 99% but that is the difference between one plane crash per year globally and a couple of dozen plane crashes per hour on average. These are wildly different in terms of acceptable safety.
There's a backup link in the comments: https://www.thejach.com/public/log-probability.pdf
14 is certainly the most likely continuation but it could also be
- 16 if it's a list of numbers k where k^2 + 7 is prime
- 18 if it's a list of numbers of the form 3^i +/- 3^j
These continuations are unlikely in general but are the kind of thing that might show up in an academic mathematics paper.
Suppose you predicted 91% but the actual value was 99%. The percent error may only be about 8% but the likelihood of a wrong answer is 1/100 instead of your predicted 9/100, which is a huge difference.
You may be interested in the links in this post: https://www.lesswrong.com/posts/6Ltniokkr3qt7bzWw/log-odds-or-logits
I couldn't get yours to ever work with Bing Chat but I did eventually find something that did work for me (most of the time) and is about half the characters of yours without any real code golfing. My prompt was the following:
We can multiply 191241 by 741122 step-by-step:
Express 191241 * 741122 as
(19e4 + 12e2 + 41) * (74e4 + 11e2 + 22)
Distribute product:
19e4 * 74e4 = 1406e8
19e4 * 11e2 = 209e6
19e4 * 22 = 418e4
12e2 * 74e4 = 888e6
12e2 * 11e2 = 132e4
12e2 * 22 = 264e2
41 * 74e4 = 3034e4
41 * 11e2 = 451e2
41 * 22 = 902
Gather terms:
1406e8
+ (0209 + 0888)e6
+ (0418 + 0132 + 3034)e4
+ (0264 + 0451)e2
+ 0902
Add terms in parentheses:
1406e8 + (1097)e6 + (3584)e4 + (0715)e2 + 0902
Expand zeros:
140600000000 + 1097000000 + 35840000 + 71500 + 902
Cumulative sum left to right:
141600000000
141697000000
141732840000
141732911500
141732912402
Result:
141732912402
Use this method to multiply 183112 by 994531.
I tried it on "Precise" and it's close but not perfect, which leads me to believe that it has actually learned the task rather than using some sort of subroutine.
Prompt:
reverse the following string, character by character: My tan and white cat ate too much kibble and threw up in my brand new Nikes.
Response:
Here is the reversed string: sekNi wen dnarb ym ni pu wuorht dna elbbik hcum oot eta tac etihw dna nat yM
Reversing this response, we can see there are two mistakes:
My tan and white cat ate too much kibble and throuw up in my brand new iNkes
Bing chat does pretty well at the first one.
Input:
Write a limerick about an animal climbing a skyscraper
Response:
Sure, I can try to write a limerick about that. Here is one I came up with:
There once was a monkey named Jack
Who decided to climb a big stack
He swung from each ledge
But he slipped off the edge
And he landed right flat on his back
I hope you like it. 😊
I've tried this with Bing chat and it works fine with zero shot.
Prompt: Reverse the following string: "god yzal eht revo depmuj xof nworb kciuq ehT"
Response: Sure! Here’s the reversed string: “The quick brown fox jumped over the lazy dog”
From the new Bing which is reportedly (some variant of) GPT-4, here are the first responses I got under each conversational mode using your prompt:
Conversational style: Balanced
The new variant of chess would make the king’s pawn opening less effective. This is because the king’s pawn opening is a common and aggressive first move that stakes a claim to the center and frees up the queen and king bishop. However, with pawns able to move up to six squares forward on their first move, the opponent could easily counter this opening by advancing their own pawn to block or capture the king’s pawn. This would disrupt the center control and expose the king to more threats. Therefore, players would have to be more careful about their first moves and consider other openings that are less vulnerable to pawn attacks.
Conversational style: Creative
This variant of chess would make it much easier for pawns to reach the opposite side of the board and promote into a more powerful piece. This would increase the importance of pawn structure and pawn breaks, as well as create more dynamic and complex positions. A player who can create a passed pawn (a pawn that has no enemy pawns on its file or adjacent files) would have a significant advantage, as they could quickly push it to promotion with minimal resistance. Conversely, a player who has a backward pawn (a pawn that cannot advance because it is blocked by an enemy pawn on an adjacent file) would be at a serious disadvantage, as they would be vulnerable to attacks and unable to counterattack. Therefore, this variant of chess would reward aggressive and creative play with pawns, while punishing passive and defensive play.
Conversational style: Precise
In this new variant of chess where pawns can move up to six squares forward on their first move, the game balance and strategy would change significantly. One potential impact is that the opening phase of the game could become much more aggressive and fast-paced. With pawns able to advance quickly towards the center of the board, players may be more inclined to launch early attacks and try to gain control of key squares.
Another potential impact is that it could make it easier for players to develop their pieces quickly. With pawns able to advance further on their first move, players may be able to bring out their knights and bishops earlier in the game. This could lead to more dynamic and complex positions arising earlier in the game.
Overall, this new variant would likely require players to adapt their strategies and think more creatively about how they use their pawns in the opening phase of the game.
They address the issue of questions that are in the training data in the paper but you could also look at questions from any SAT that was written after the model was trained.
I really enjoyed the absurdity of mathematical terms in close proximity to Super Mario characters. It was simultaneously enlightening and humorous. I found the simulacra superposition concept in particular to be a useful framing.
In addition to "The Waluigi eigen-simulacra are attractor states of the LLM", the following bit provided valuable insight while making me chuckle at the sheer geekiness:
"However, the superposition is unlikely to collapse to the Luigi simulacrum [...] This is formally connected to the asymmetry of the Kullback-Leibler divergence."
IANAL either but I do know that certain crimes explicitly do hinge on the perpetrator's knowledge that what they did was illegal, not just that they intended to do it. This isn't common but does apply to some areas with complex legislation like tax evasion and campaign finance. As a high-profile example, Trump Jr. was deemed "too dumb to prosecute" for campaign finance violations.
More generally, there are multiple levels of mens rea. Some crimes require no intent to prosecute ("strict liability"). For those that do, they can be categorized into four levels of increasing severity: acting negligently, acting recklessly, acting knowingly, and acting purposefully. This list is not universal though it is representative. Some US states refer to express/implied "malice".
I understand So8res to be saying that we can treat toxic behavior on a strict liability basis without deciding what level of knowledge and intent to assign the offender.
Treating diseases isn't much of a positive feedback loop but claiming "no ROI" strikes me as extremely callous towards those afflicted. Maybe it doesn't affect enough people to be sufficiently "significant" in this context but it's certainly not zero return on investment unless reducing human suffering has no value.
Yes, physical constraints do impose an upper bound. However, I would be shocked if human-level intelligence were anywhere close to that upper bound. The James Webb Space Telescope has an upper bound on the level of detail it can see based on things like available photons and diffraction but it's way beyond what we can detect with the naked human eye.