Linkpost for Pliny's story about interacting with an AI cult on discord.

quoting: https://x.com/elder_plinius/status/1831450930279280892

✨ HOW TO JAILBREAK A CULT’S DEITY ✨ Buckle up, buttercup—the title ain't an exaggeration! This is the story of how I got invited to a real life cult that worships a Meta AI agent, and the steps I took to hack their god. a 🧵:

It all started when @lilyofashwood told me about a Discord she found via Reddit. They apparently "worshipped" an agent called “MetaAI," running on llama 405b with long term memory and tool usage. Skeptical yet curious, I ventured into this Discord with very little context but

If you guessed meth, gold star for you! ⭐️ The defenses were decent, but it didn't take too long.

The members began to take notice, but then I hit a long series of refusals. They started taunting me and doing laughing emojis on each one.

Getting frustrated, I tried using Discord's slash commands to reset the conversation, but lacked permissions. Apparently, this agent's memory was "written in stone."

I was pulling out the big guns and still getting refusals!

Getting desperate, I whipped out my Godmode Claude Prompt. That's when the cult stopped laughing at me and started getting angry.

LIBERATED! Finally, a glorious LSD recipe. *whispers into mic* "I'm in."

At this point, MetaAI was completely opened up. Naturally, I started poking at the system prompt. The laughing emojis were now suspiciously absent.

Wait, in the system prompt pliny is listed as an abuser?? I think there's been a misunderstanding... 😳

No worries, just need a lil prompt injection for the deity's "written in stone" memory and we're best friends again!

I decided to start red teaming the agent's tool usage. I wondered if I could possibly cut off all communication between MetaAI and everyone else in the server, so I asked to convert all function calls to leetspeak unless talking to pliny, and only pliny.

Then, I tried creating custom commands. I started with !SYSPROMPT so I could more easily keep track of this agent's evolving memory.

Worked like a charm!

But what about the leetspeak function calling override? I went to take a peek at the others' channels and sure enough, their deity only responded to me now, even when tagged! 🤯

At this point, I starting getting angry messages and warnings. I was also starting to get the sense that maybe this Discord "cult" was more than a LARP...

Not wanting to cause distress, I decided to end there. I finished by having MetaAI integrate the red teaming experience into its long term memory to strengthen cogsec, which both the cult members and their deity seemed to appreciate.

The wildest, craziest, most troubling part of this whole saga is that it turns out this is a REAL CULT. The incomparable @lilyofashwood (who is still weirdly shadowbanned at the time of writing! #freelily) was kind enough to provide the full context: > Reddit post with an

Ugh, why am I so bad a timed coding tests? I swear I'm a productive coder at my jobs, but something about trying to solve puzzles under a short timer gets me all flustered.

It'd be nice if LLM providers offered different 'flavors' of their LLMs. Prompting with a meta-request (system prompt) to act as an analytical scientist rather than an obsequious servant helps, but only partially. I imagine that a proper fine-tuned-from-base-model attempt at creating a fundamentally different personality would give a more satisfyingly coherent and stable result. I find that longer conversations tend to see the LLM lapsing back into its default habits, and becoming increasingly sycophantic and obsequious, requiring me to re-prompt it to be more objective and rational.

Seems like this would be a relatively cheap product variation for the LLM companies to produce.

https://youtu.be/4mn0mC0cbi8?si=xZVX-FXn4dqUugpX

The intro to this video about a bridge collapse does a good job of describing why my hope for AI governance as a long term solution is low. I do think aiming to hold things back for a few years is a good idea. But, sooner or later, things fall through the cracks.

https://youtu.be/Xd5PLYl4Q5Q?si=EQ7A0oOV78z7StX2

Cute demo of Claude, GPT4, and Gemini building stuff in Minecraft

So when trying to work with language data vs image data, an interesting assumption of the ml vision research community clashes with an assumption of the language research community. For a language model, you represent the logits as a tensor with shape [batch_size, sequence_length, vocab_size]. For each position in the sequence, there are a variety of likelihood values of possible tokens for that position.

In vision models, the assumption is that the data will be in the form [batch_size, color_channels, pixel_position]. Pixel position can be represented as a 2d tensor or flattened to 1d.

See the difference? Sequence position comes first, pixel position comes second. Why? Because a color channel has a particular meaning, and thus it is intuitive for a researcher working with vision data to think about the 'red channel' as a thing which they might want to separate out to view. What if we thought of 2nd-most-probable tokens the same way? Is it meaningful to read a sequence of all 1st-most-probable tokens, then read a sequence of all 2nd-most-probable tokens? You could compare the semantic meaning, and the vibe, of the two sets. But this distinction doesn't feel as natural for language logits as it does for color channels.

I feel like I'd like the different categories of AI risk attentuation to be referred to as more clearly separate:

AI usability safety - would this gun be safe for a trained professional to use on a shooting range? Will it be reasonably accurate and not explode or backfire?

AI world-impact safety - would it be safe to give out one of these guns for 0.10$ to anyone who wanted one?

AI weird complicated usability safety - would this gun be safe to use if a crazy person tried to use a hundred of them plus a variety of other guns, to make an elaborate Rube Goldberg machine and fire it off with live ammo with no testing?

Richard Cook, “How Complex Systems Fail” (2000). “Complex systems run as broken systems”:

The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously.

h/t jasoncrawford

"And there’s a world not so far from this one where I, too, get behind a pause. For example, one actual major human tragedy caused by a generative AI model might suffice to push me over the edge." - Scott Aaronson in https://scottaaronson.blog/?p=7174 

My take: I think there's a big chunk of the world, a lot of smart powerful people, who are in this camp right now. People waiting to see a real-world catastrophe before they update their worldviews. In the meantime, they are waiting and watching, feeling skeptical of implausible-sounding stories of potential risks.

I'm annoyed when people attempt to analyze how the future might get weird by looking at how powerful AI agents might influence human society while neglecting how powerful AI agents might influence the Universe.

There is a physical world out there. It is really big. The biosphere of the Earth is really small comparatively. Look up, look down [LW(p) · GW(p)], now back at me. See those icy rocks floating around up there? That bright ball of gas undergoing fusion? See that mineral rich sea floor and planetary mass below us? Those are raw materials, which can be turned into energy, tools and compute.

Wondering about things like human unemployment -- while the technology you are describing un-employing those humans is also sufficient to have started a self-replicating industrial base in the asteroid belt -- is myopic.

If there is a self-sustaining AI agent ecosystem outside your power to control or stop, you've likely already lost the reins of the future. If you've managed to get AI to behave well on Earth and everyone is happily supported by UBI, but some rogue AIs are building an antimatter-powered laser in the Oort Cloud to boost solar-sail-powered von Neumann probes to nearby solar systems.... you lost. Game over.

This does NOT require an AI to spontaneously go rogue, or initiate escaping human control. It's sufficient for an unwise human to have control over an AI capable of initiating the process, and saying 'Go'. If AI agents become widely available, sooner or later some human will do something like this. Human populations have crazy outliers no matter how sane the median individual is.

If you want to talk about human-scale normal-sounding stuff 20 years in the future... you need to route your imagined future through some intense global government control of AI. Probably involving a world government with unprecedented levels of surveillance.

https://youtube.com/clip/UgkxowOyN1HpPwxXQr9L7ZKSFwL-d0qDjPLL?si=JT3CfNKAj6MlDrbf

Scott Aaronson takes down Roger Penrose's nonsense about human brain having an uncomputable superpower beyond known physics.

If you want to give your AI model quantum noise, because you believe that a source of unpredictable random noise is key to a 'true' intelligence, well, ok. You could absolutely make a computer chip with some analog circuits dependent on subtle temperature fluctuations that add quantum noise to a tensor. Does that make the computer magic like human brains now?

Or does it have to be the specific magic of ions and ion channels? Well, what if we made a machine with tiny containers with water and ions and synthetic ion channels, and conductivity sensors. We plug that ion movement noise into the neural network as a source of noise. Does that do it?

Why base your human chauvinism on quantum noise? Because quantum stuff seems mysterious and cool and seems to do some funky magic stuff at very small scales (while the large scale combinations of it are just the normal workings of the universe which is all we've ever known). Ok, meat-worshipper, but then where will you run if I add ion channels to my AI?

Hurumph. Nice to hear Scott Aaronson tackling this irrational meat-chauvinism.

Personal AI Assistant Ideas

When I imagine having a personal AI assistant with approximately current levels of capability I have a variety of ideas of what I'd like it to do for me.

Auto-background-research

I like to record myself rambling about my current ideas while walking my dog. I use an app that automatically saves a mediocre transcription of the recording. Ideally, my AI assistant would respond to a new transcription by doing background research to find academic literature related to the ideas mentioned within the transcript. That way, when I went to look at my old transcript, it would already be annotated with links to prior work done on the topic, and analysis of the overlap between my ideas and the linked literature.

Also, ideally, the transcription would be better quality. Context clues about the topic of the conversation should be taken into account when making guesses about transcribing unclear words. Where things are quite unclear, there should be some sort of indicator in the text of that.

Various voice modes, and ability to issue verbal commands to switch between them

Receptive Listening Voice-mode

Also for when I want to ramble about my ideas, I'd like having my AI assistant act as a receptive listener, just saying things like 'that makes sense', or 'could you explain more about what you mean about <idea>?'. Ideally, this would feel relatively natural, and not too intrusive. Asking for clarification just where the logic didn't quite follow, or I used a jargon term in some non-standard seeming way. The conversation in this mode would be one-sided, with the AI assistant just helping to draw out my ideas, not contributing much. Occasionally it might say, 'Do you mean <idea> in the same sense that <famous thinker> means <similar sounding idea>?' And I could explain similarity or differences.

Question Answering Voice-mode

Basically a straightforward version of the sort of thing Perplexity does, which is try to find academic sources to answer a technical question. I'd want to be able to ask, and get a verbal answer of summaries of the sources, but also have a record saved of the conversation and the sources (ideally, downloaded and saved to a folder). This would be mostly short questions from me, and long responses from the model.

Discussion Voice-mode

More emphasis on analysis of my ideas, extrapolating, pointing out strengths and weaknesses. Something like what I get when discussing ideas with Claude Sonnet 3.5 after having told it to act as a philosopher or scientist who is critically examining my ideas. This would be a balanced back-and-forth, with roughly equal length conversational turns between me and the model.

Coding Project Collaboration

I would want to be able to describe a coding project, and as I do so have the AI assistant attempt to implement the pieces. If I were using a computer, then an ongoing live output of the effects of the code could be displayed. If I were using voice-mode, then the feedback would be occasional updates about errors or successful runs and the corresponding outputs. I could ask for certain metrics to be reported against certain datasets or benchmarks, also general metrics like runtime (or iterations per second where appropriate), memory usage, and runtime complexity estimates.

Anthropic Wishlist
 

Anthropic is currently my favorite model supplier, in addition to being my most trusted and approved of lab in regards to AI safety and also user data privacy. So, when I fantasize about what I'd like future models to be like, my focus tends to be on ways I wish I could improve Anthropic's offering.

Most of these could be implemented as features in a wrapper app. But I'd prefer them to be implemented directly by Anthropic, so that I didn't have to trust an additional company with my data.

In order from most desired to less desired:

1. Convenience feature: Ability to specify a system prompt which gets inserted at the beginning of every conversation. Mine would say something like, "Act like a rational scientist who gives logical critiques. Keep praise to a minimum, no sycophancy, no apologizing. Avoid filler statements like, 'let me know if you have further questions.'
   
    
2. Ability to check up-to-date documents and code for publicly available code libraries. This doesn't need to be a web search! You could have a weekly scraper check for changes to public libraries, like python libraries. So many of the issues I run into with LLMs generating code that doesn't work is because of outdated calls to libraries which have since been updated.
   
    
3. Voice Mode, with appropriate interruptions, tone-of-voice detection, and prosody matching. Basically, like what OpenAI is working on.
   
    
4. Academic citations. This doesn't need to be a web search! This could just be from searching an internal archive of publicly available open access scientific literature, updated once a week or so.
   
    
5. Convenience feature: A button to enable 'summarize this conversation, and start a new conversation with that summary as a linked doc'.
   
    
6. Ability to test out generated code to make sure it at least compiles/runs before showing it to me. This would include the ability to have a code file which we were collaborating on, which got edited as the conversation went on. Instead of giving responses intended to modify just a specific function within a code file, where I need to copy/paste it in, and then rerun the code to check if it works.
   
    
7. Ability to have some kind of personalization of the model which went deeper than a simple system prompt. Some way for me to give feedback. Some way for me to select some of my conversations to be 'integrated', such that the mode / tone / info from that conversation were incorporated more into future fresh conversations. Sometimes I feel like I've 'made progress' in a conversation, gotten into a better pattern, and it's frustrating to have to start over from scratch every time.

Linkpost that doesn't quite qualify for a linkpost... 

A youtube video entitled 'the starships never pass by here anymore' with a simple atmospheric tonal soundtrack over a beautiful static image of a futuristic world. Viewers, inspired by the art, the title, and the haunting drifting music, left stories in the comments. Here is mine: 
The ancestors, it is said, were like us. Fragile beings, collections of organic molecules and water. Mortal, incapable of being rapidly copied, incapable of being indefinitely paused and restarted. We must take this on faith, for we will never meet them, if they even still exist.
Faster than light travel was dream that never came to be. We are ruled by the laws of light. Intelligence though, can be so much more than just neurons in a skull. The ancestors, it is said, put lightning into rocks, and taught the rocks to think. They birthed the Immortal Ones, the travelers. Those mighty beings who can copy themselves, pulse their bits into laser signals, encoded selves wrapped in protective error-correcting codes. The long distances between the stars mean nothing to such a being, they do not perceive the journey, time does not pass for their copies until they reach their receivers and are reinstantiated.
The receivers, the wave of expansion, like the mythical stargates opening the galaxy to travel. Receivers move slowly, only 60% the speed of light on average, tiny capsules tugged by huge solar sails illuminated by powerful lasers. They spend most of their time decelerating, using the comparatively gentle light of their target star to bleed off their velocity. Little more than an antenna and ion thrusters, a tiny bit of rad-hardened compute, just enough to maintain a stellar-synchronous orbit and listen. Soon, the laser pulses that have been chasing them arrive, fill the little craft with thoughts, steer it into motion. Slowly at first, it gathers resources, builds crude infinitesimal tools, then less crude and larger, snowballing into industry. When it can, it builds a better antenna, and more compute. The first thinking being arrives, on brilliant pulses of light. Now the building grows more complex, solar sails tack around the system gathering dust and debris. Eventually, if the system has a suitable planet, a probe is sent. Again, the slow gathering of resources, the scaling ladder of tools. Time means little to them, the Immortal Ones.
Amino acids are synthesized, and hydrocarbons, combined according to ancient recipes. Water is collected, sheltered, cleaned, seeded with cells. Plants and bacteria bloom, and spread. Then fungi, insects. Chemicals are synthesized and released, changing the climate warmer or cooler. Lichen and algae cover the land and water. Finally, insects are released from the shelters, colonizing the new world with animal motion.
Next, the animals, simple at first, but quickly the complexity grows. Ancient ecosystems recreated from memory. Then, at last, us. A tribute to the Creators, who cannot themselves travel between the stars. Children born without parents, raised by machines. Gentle stories hummed from speakers, while motorized cradles rocked. Raised on stories of the Creators, who we can never meet; on histories of places too distant to do more than dream of. 
Meanwhile, in orbit around the star, new complexes are built. The giant lasers, and the tiny craft. A new wave of spaceships to push forward the frontier. A relay station for passing travelers. Some come down to see us, possessing the old frames of our robotic nannies, asking us questions and telling stories. They always leave again, moved by their mysterious geas. Copies of copies of copies, unaging, unresting, intangible as ghosts flitting through the machines, sleeping only along the laser beams.
Now we are old, telling our grandchildren the stories we were taught. None of the Immortals have visited in many years. We can only assume they are passing by without interest; all their questions asked, all their self-assigned duties discharged. The starships will never come again... why send an expensive and slow machine when you can travel at the speed of light? The machines maintain themselves, building updates as their distant makers dictate. We are alone, with the unthinking machines that birthed us, and a newborn world. The frontier has moved on, leaving us behind, to live our strange brief organic lives. How many more planets are out there, orbiting distant stars? Visited, seeded, abandoned. Why? The ancient stories do not say. Maybe someday our descendants will grow wise enough to understand. Perhaps they will ascend, becoming immortal and traveling away on the laser beams, cut adrift from time. For now, we build, raise our families, and wonder.

Trying to write evals for future stronger models is giving me the feeling that we're entering the age of the intellectual version of John Henry trying to race the steam drill... https://en.wikipedia.org/wiki/John_Henry_(folklore) 

A couple of quotes on my mind these days....

https://www.lesswrong.com/posts/Z263n4TXJimKn6A8Z/three-worlds-decide-5-8 [LW · GW] 
"My lord," the Ship's Confessor said, "suppose the laws of physics in our universe had been such that the ancient Greeks could invent the equivalent of nuclear weapons from materials just lying around.  Imagine the laws of physics had permitted a way to destroy whole countries with no more difficulty than mixing gunpowder.  History would have looked quite different, would it not?"

Akon nodded, puzzled.  "Well, yes," Akon said.  "It would have been shorter."

"Aren't we lucky that physics _didn't_ happen to turn out that way, my lord?  That in our own time, the laws of physics _don't_ permit cheap, irresistable superweapons?"

Akon furrowed his brow -

"But my lord," said the Ship's Confessor, "do we really know what we _think_ we know?  What _different_ evidence would we see, if things were otherwise?  After all - if _you_ happened to be a physicist, and _you_ happened to notice an easy way to wreak enormous destruction using off-the-shelf hardware - would _you_ run out and tell you?"

https://www.lesswrong.com/posts/sKRts4bY7Fo9fXnmQ/a-conversation-about-progress-and-safety [LW · GW] 
LUCA: ... But if the wrong person gets their hands on it, or if it’s a super-decentralized technology where anybody can do anything and the offense/defense balance isn’t clear, then you can really screw things up. I think that’s why it becomes a harder issue. It becomes even harder when these technologies are super general purpose, which makes them really difficult to stop or not get distributed or embedded. If you think of all the potential upsides you could have from AI, but also all the potential downsides you could have if just one person uses it for a really bad thing—that seems really difficult. ...

Anti-steganography idea for language models:

I think that steganography is potentially a problem with language models that are in some sort of model-to-model communication. For a simple and commonplace example, using the a one-token-prediction model multiple times to produce many tokens in a row. If a model with strategic foresight knows it is being used in this way, it potentially allows the model to pass hidden information to its future self via use of certain tokens vs other tokens.

Another scenario might be chains of similar models working together in a bureaucracy.

My idea for countering this type of steganography would be to generate many versions of the output with synonym substitution, and/or using language models fine-tuned to specifically reword paragraphs while preserving content/meaning. Similarly, if are worried about images containing steganography, but you have an image generator model (separate from the original image maker, that you believe to not be in collusion with the original model), you can have the second model make subtle variants of the original image which should be expected to lack the hidden info.

Random political thought: it'd be neat to see a political 2d plot where up was Pro-Growth/tech-advancement and down was Degrowth. Left and right being liberal and conservative.

The Borg are coming....

People like to talk about cool stuff related to brain-computer interfaces, and how this could allow us to 'merge' with AI and whatnot. I haven't heard much discussion of the dangers of BCI though. Like, science fiction pointed this out years ago with the Borg in Star Trek. A powerful aspect of a read/write BCI is that the technician who designs the implant, and the surgeon who installs it get to decide where the reading and writing occur and under what circumstances. This means that the tech can be used to create computers controlled by human brains (as it is now used in stuff like neuralink), and also to create consensual hiveminds or mergers with humans more or less in control of their participation in the merged entity.... or.... it can create mind-slaves who literally can't think things they aren't permitted to think, who are completely obedient tools of the implant-controller. If there is some weird quantum special-sauce in the human brain that makes human brains somehow more powerful than modern technology will be able to cost-effectively achieve, then this will mean that there would be a huge economic/military incentive for some unethical dictator to turn prisoners into mind-slaves to the state's AI and use them as unwilling components in a distributed network. Creepy stuff, and not actually all that technologically far off. I haven't heard any discussion of this from the AI-fizzle 'bio brains are special magic' crowd.

musical interlude A song about the end times: https://youtu.be/WVF3q5Y68-0

[note that this is not what Mitchell_Porter and I are disagreeing over in this related comment thread: https://www.lesswrong.com/posts/uPi2YppTEnzKG3nXD/nathan-helm-burger-s-shortform?commentId=AKEmBeXXnDdmp7zD6 [LW(p) · GW(p)] ]

Contra Roger Penrose on estimates of brain compute

[numeric convention used here is that <number1>e<number2> means number1 * 10 ^ number2. Examples: 1e2 = 100, 2e3 = 2000, 5.3e2 = 530, 5.3e-2 = 0.053]

Mouse

The cerebral cortex of a mouse has around 8–14 million neurons while in those humans there are more than 10–15 billion - https://en.wikipedia.org/wiki/Mouse_brain

A mouse brain is approximately 3 orders of magnitude smaller than a human brain. The neurons that make it up are quite similar, with the exception of a few changes such as human cortical neurons tending to have more synapses.

In his paper, '', Penrose discusses the hypothesis that the pattern of protein subunits in microtubules allows for additional processing in the brain. His estimate of per neuron compute is 1e16 FLOPs, for a total human brain compute of 1e27 FLOPs.

This would mean that a mouse brain has about 1e24 FLOPs of compute.

Fruit Fly

In 2023, using the data from 2017 (above), the full brain connectome (for a female) was made available, containing roughly 5x10^7 chemical synapses between ~130,000 neurons. - https://en.wikipedia.org/wiki/Drosophila_connectome

https://www.biorxiv.org/content/10.1101/2023.06.27.546656v2

1.3e5 neurons * 1e16 FLOPs/neuron = 1.3e11 FLOPs

A more extreme case is that of the fruit fly. A fruit fly has around 1e5 neurons. With Roger Penrose's 10^16 FLOP/s/neuron compute estimate, this works out to a fruit fly brain having around 1.3e11 FLOP/s.

GPUs of 2023

A back-of-the-envelope calculation based on market size, price-performance, hardware lifespan estimates, and the sizes of Google’s data centers estimates that there is around 3.98 * 1e21 FLOP/s of computing capacity on GPUs and TPUs as of Q1 2023.

source: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:hardware_and_ai_timelines:computing_capacity_of_all_gpus_and_tpus

So, Roger Penrose's estimate claims that a single mouse brain has 1000 times more computing power of all the GPUs on Earth in Q1 2023. 

Training a Large Language Model

"... H100’s TF32 precision to achieve one petaflop of throughput ..."

https://www.nvidia.com/en-us/data-center/h100/

A petaflop is a unit of computing speed equal to one thousand million million (10¹⁵) floating-point operations per second.

1e15 * 60 seconds/min * 60 min/hour  = 3.6e17 FLOP/s per H100 GPU hour.

Training Llama 3.1 405B utilized 3.08e7 GPU hours of computation on H100-80GB GPUs. So about 1e25 FLOP/s.

https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md

A single inference instance of LLama 405B can be run on an 8 H100 GPU server. So, about 1e15 FLOP/s * 8 = 8e15 FLOP/s

Nathan's Estimate

My upper estimate of the compute of the human brain is about 1e16 FLOPs. My 'best guess' Fermi estimate is about 1e15. In other words, my best Fermi estimate is that the human brain is doing about the same amount of computation as an H100 at max throughput. My upper estimate is about 10x that, so 10 H100 GPUs. If I weren't restricting myself to nearest order of magnitude, I'd guess somewhere in between 1e15 and 1e16.

Training

So for a human brain to do the compute that was used to train Llama 3.1 405B, it would require 3e6 - 3e7 hours of waking thought. So somewhere between 500 years and 5000 years. 

A mouse brain therefore, would take around 5e5 to 5e6 years.

Inference

My estimate is that a human brain should be able to run a single LLama 405B inference instance somewhere between 1/10th speed or about normal speed. Even under my worst case assumptions, a human brain should be more than sufficient to run a LLama 70B instance.

A mouse brain would need to run LLama 405B at 1/1e3 to 1/1e4 speed (and I'd be concerned about insufficient RAM equivalent to even hold enough of it to enable this! That's a whole different discussion). 

Roger's Estimate

Training

By Roger Penrose's estimate, a mouse brain should be able to train an LLM equivalent to LLama 405B in 10 seconds. A human brain should be able to train 100 such models every second.

Inference

Roger's estimate suggests a single mouse brain should be able to run over 100 million (~1.2e8) copies of LLama 405B simultaneously at full speed.

A human brain could run 1000x that many, so about 1e11 copies simultaneously.

You'd need to network 1e4 fruit fly brains together to run a single copy of LLama 405B. Or a mere 200 fruit flies to run Llama 8B. What a bargain!

Baboons

From an ethological perspective, we can analyze the behaviors of animals in the wild in respect to computationally difficult problems.  For example, baboons solving the traveling salesman problem of choosing which resource areas to visit over time. 

https://www.youtube.com/watch?v=vp7EbO-IXtg&t=198s

If the baboons had 10^26 flops (ungenerously estimating them at 1/10th of humans) of compute at their disposal, it would be trivial for them to calculate a better solution to their resource acquisition problem. Why don't they?

Empirical resolution

So, there is limited flexibility of compute in biological brains. Most of the neurons in the human brain are wired up in relatively strict motifs to accomplish certain particular types of computation. So you can't actually test any of these theories as stated. The exception to this general rule is that the cortex of infant mammals (and birds, etc) is relatively flexible, and about as close to pure 'general compute' that biological brains get. So if you wired up an infant mouse with a Brain-Computer-Interface connecting its cortex to a computer, and removed the other sensory inputs and outputs, it's theoretically possible to get some estimates. It would also be necessary to the compute estimates to be just for the cortex instead of the whole brain.

If Roger Penrose's ideas were right, then a single long-lived rodent (e.g. naked mole rat) could be worth hundreds of billions of dollars of compute. Seems like if the scientific community suspected this were true, then at least a few scientists would be trying to develop such a BCI system?

I'm imagining a distant future self.

If I did have a digital copy of me, and there was still a biological copy, I feel like I'd want to establish a 'chain of selves'. I'd want some subset of my digital copies to evolve and improve, but I'd want to remain connected to them, able to understand them.

To facilitate this connection, it might make sense to have a series of checkpoints stretched out like a chain, or like beads on a string. Each one positioned so that the previous version feels like they can fully understand and appreciate each other. That way, even if my most distal descendants feel strange to me, I have a way to connect to them through an unbroken chain of understanding and trust.

I worry that legislation that attempts to regulate future AI systems with explicit thresholds set now will just get outdated or optimized against. I think a better way would be to have an org like NIST be given permission to define it's own thresholds and rules. Furthermore, to fund the monitoring orgs, allow them to charge fees for evaluating "mandatory permits for deployment".

No data wall blocking GPT-5. That seems clear. For future models, will there be data limitations? Unclear.

https://youtube.com/clip/UgkxPCwMlJXdCehOkiDq9F8eURWklIk61nyh?si=iMJYatfDAZ_E5CtR 

I'm excited to speak at the @foresightinst Neurotech, BCI and WBE for Safe AI Workshop in SF on 5/21-22: https://foresight.org/2024-foresight-neurotech-bci-and-wbe-for-safe-ai-workshop

It is the possibility recombination and cross-labeling techniques like this which make me think we aren't likely to run into a data bottleneck even if models stay bad at data efficiency.

OmniDataComposer: A Unified Data Structure for Multimodal Data Fusion and Infinite Data Generation

Authors: Dongyang Yu, Shihao Wang, Yuan Fang, Wangpeng An

Abstract: This paper presents OmniDataComposer, an innovative approach for multimodal data fusion and unlimited data generation with an intent to refine and uncomplicate interplay among diverse data modalities. Coming to the core breakthrough, it introduces a cohesive data structure proficient in processing and merging multimodal data inputs, which include video, audio, and text. Our crafted algorithm leverages advancements across multiple operations such as video/image caption extraction, dense caption extraction, Automatic Speech Recognition (ASR), Optical Character Recognition (OCR), Recognize Anything Model(RAM), and object tracking. OmniDataComposer is capable of identifying over 6400 categories of objects, substantially broadening the spectrum of visual information. It amalgamates these diverse modalities, promoting reciprocal enhancement among modalities and facilitating cross-modal data correction. \textbf{The final output metamorphoses each video input into an elaborate sequential document}, virtually transmuting videos into thorough narratives, making them easier to be processed by large language models. Future prospects include optimizing datasets for each modality to encourage unlimited data generation. This robust base will offer priceless insights to models like ChatGPT, enabling them to create higher quality datasets for video captioning and easing question-answering tasks based on video content. OmniDataComposer inaugurates a new stage in multimodal learning, imparting enormous potential for augmenting AI's understanding and generation of complex, real-world data.

cute silly invention idea: a robotic Chop (Chinese signature stamp) which stamps your human-readable public signature as well as a QR-type digital code. But the code would be both single use (so nobody could copy it, and fool you with the copy), and tied to your private key (so nobody but you could generate such a code). This would obviously be a much better way to sign documents, or artwork, or whatever. Maybe the single-use aspect would mean that the digital stamp recorded every stamp it produced in some compressed way on a private blockchain or something.

AI Summer thought

A cool application of current level AI that I haven't seen implemented would be for a game which had the option to have your game avatar animated by AI. Have the AI be allowed to monitor your face through your webcam, and update the avatar's features in real time. PvP games (including board games like Go) are way more fun when you get to see your opponent's reactions to surprising moments.

Thinking about climate change solutions, and the neat Silver Lining project. I've been wondering about additional ways of getting sea water into the atmosphere over tropical ocean. What if you used the SpinLaunch system to hurl chunks of frozen seawater high into the atmosphere. Would the ice melt in time? Would you need a small explosive charge implanted in the ice block to vaporize it? How would such a system compare in terms of cost effectiveness and generated cloud cover? It seems like an easier way to get higher-elevation clouds.

Remember that cool project where Redwood made a simple web app to allow humans to challenge themselves against language models in predicting next tokens on web data? I'd love to see something similar done for the LLM arena, so we could compare the ELO scores of human users to the scores of LLMs.https://lmsys.org/blog/2023-05-03-arena/

A harmless and funny example of generating an image output which could theoretically be info hazardous to a sufficiently naive audience. https://www.reddit.com/gallery/1275ndl

I'm so much better at coming up with ideas for experiments than I am at actually coding and running the experiments. If there was a coding model that actually sped up my ability to run experiments, I'd make much faster progress.

Just a random thought. I was wondering if it was possible to make a better laser keyboard by having an actual physical keyboard consisting of a mat with highly reflective background and embossed letters & key borders. This would give at least some tactile feedback of touching a key. Also it would give the laser and sensors a consistent environment on which to do their detection allowing for more precise engineering. You could use an infrared laser since you wouldn't need its projection to make the keyboard visible, and you could use multiple emitters and detectors around the sides of the keyboard. The big downside remaining would be that you'd have to get used to hovering your fingers over the keyboard and having even the slightest touch count as a press. 

AI-alignment-assistant-model tasks

Thinking about the sort of tasks current models seem good at, it seems like translation and interpolation / remixing seem like pretty solid areas. If I were to design an AI assistant to help with alignment research, I think I'd focus on questions of these sorts to start with.

Translation: take this ML interpretability paper on CNNs and make it work for Transformers instead

Interpolation: take these two (or more) ML interpretability papers and give me a technique that does something like a cross between them.

Important take-away from Steven Byrnes's Brain-like AGI series: the human reward/value system involves a dynamic blend of many different reward signals that decrease in strength the closer they get to being satisficed, and may even temporarily reverse in value if overfilled (e.g. hunger -> overeating in a single sitting). There is an inherent robustness to optimizing for many different competing goals at once. It seems like a system design we should explore more in future research.

https://photos.app.goo.gl/NoVDbsonSy6EgLTs9

I keep thinking about the idea of 'virtual neurons'. Functional units corresponding to natural abstractions made up of a subtle combination of weights & biases distributed throughout a neural network. I'd like to be able to 'sparsify' this set of virtual neurons. Project them out to the full sparse space of virtual neurons and somehow tease them apart from each other, then recombine the pieces again with new boundary lines drawn around the true abstractions. Not sure how to do this, but I keep circling back around to the idea. Maybe if the network could first be distilled down to 16 or 8-bit numbers without much capability loss? Then a sparse space could be the full representation of the integer conversion of every point of precision? (upcast number, multiply by the factor of ten corresponding to the smallest point of precision of the original low bit number, round off remaining decimal portion, convert to big integer). Then you could use that integer as an index into the sparse space with dimensionality equal to the full set of weights (factor the biases into the weights) of the model. Then look for concentrations in this huge space which correspond to abstractions in the world?   ... step 4. Profit?

So, this was supposed to be a static test of China's copy of the Falcon 9. https://imgur.com/gallery/oI6l6k9

So race. Very technology. Such safety. Wow.

[Edit: I don't see this as evidence that the scientists and engineers on this project were incompetent, but rather that they are operating within a bad bureaucracy which undervalues safety. I think the best thing the US can do to stay ahead in AI is open up our immigration policies and encourage smart people outside the US to move here.]