Posts
Comments
Relevant quote from Altman after the firing:
“I think this will be the most transformative and beneficial technology humanity has yet invented,” Altman said, adding later, “On a personal note, four times now in the history of OpenAI, the most recent time was just in the last couple of weeks, I’ve gotten to be in the room when we push … the veil of ignorance back and the frontier of discovery forward.”
However, uploading seems to offer a third way: instead of making alignment researchers more productive, we "simply" run them faster.
When I think about uploading as an answer to AI, I don't think of it as speeding up alignment research necessarily, but rather just outpacing AI. You won't get crushed by an unaligned AI if you're smarter and faster than it is, with the same kind of access to digital resources.
The breeding process would adjust that if it was a limiting factor.
The problem with this is that one day you'll see someone who has the same flaw you've been trying to suppress in yourself, and they just completely own it, taking pride in it, focusing on its advantages, and never once trying to change it. And because they are so self-assured about it, the rest of the world buys in and views it as more of an interesting quirk than a flaw.
When you encounter that person, you'll feel like you threw away something special.
How about this one? Small group or single individual manages to align the first very powerful AGI to their interests. They conquer the world in a short amount of time and either install themselves as rulers or wipe out everyone else.
Oh, I see your other graph now. So it just always guesses 100 for everything in the vicinity of 100.
This is a cool idea. I wonder how it's able to do 100, 150, and 200 so well. I also wonder what are the exact locations of the other spikes?
You can deduce a lot about someone's personality from the shape of his face.
I don't know if this is really that controversial. The people who do casting for movies clearly understand it.
On the question of morality, objective morality is not a coherent idea. When people say "X is morally good," it can mean a few things:
- Doing X will lead to human happiness
- I want you to do X
- Most people want you to do X
- Creatures evolving under similar conditions as us will typically develop a preference for X
- If you don't do X, you'll be made to regret it
- etc...
But believers in objective morality will say that goodness means more than all of these. It quickly becomes clear that they want their own preferences to be some kind of cosmic law, but they can't explain why that's the case, or what it would even mean if it were.
On the question of consciousness, our subjective experiences are fully explained by physics.
The best argument for this is that our speech is fully explained by physics. Therefore physics explains why people say all of the things they say about consciousness. For example, it can explain why someone looks at a sunset and says, "This experience of color seems to be occurring on some non-physical movie screen." If physics can give us a satisfying explanation for statements like that, it's safe to say that it can dissolve any mysteries about consciousness.
The problem isn't that he's overly sure about "contentious topics." These are easy questions that people should be sure about. The problem is that he's sure in the wrong direction.
I don't know quantum mechanics, but your back-of-the-envelope logic seems a little suspicious to me. The Earth is not an isolated system. It's being influenced by gravitational pulls from little bits of matter all over the universe. So wouldn't a reverse simulation of Earth also require you to simulate things outside of Earth?
From my experiences at a very woke company, I tend to agree with the top comments here that it's mostly a bottom-up phenomenon. There is a segment of the employees who are fanatically woke, and they have a few advantages that make it hard for anyone to oppose them. Basically:
- They care more about promoting wokeness than their opponents do about combating it, and
- It is safer from a reputational standpoint to be too woke than not woke enough.
Then we get a feedback loop where victories for wokism strengthen these advantages, leading to more victories.
The deeper question is whether there is also a system of organized top-down pressure running in parallel to this. Elon's purchase of Twitter presents an interesting case study. It seemed to trigger an immune response from several external sources. Nonprofit organizations emerged from the woodwork to pressure advertisers to leave the platform, and revenue fell sharply. Apparently this happened before Elon even adjusted any policies, on the mere suspicion that he would fail to meet woke standards.
At the same time, there was a barrage of negative media coverage of Elon, uncovering sexual assault scandals and bad business practices from throughout his life. Perhaps a similar fate awaits any top-level executive who does not steer his company in a woke direction?
I'll end with an excerpt from an old podcast that has stuck with me:
It is impossible to defend the idea that the invisible hand of the market would guide them [corporations] to this course of action. I’ve been inside a large company when it was adjacent to this kind of voluntary action — where corporations all act in lock step — you’ll just have to trust me here — and I’ve seen the way it’s coordinated.
What will happen is a prominent journalist or several will reach out to the company’s leadership team and ask them for a comment on the current thing. Especially they do this if that company has any history of dealings with the object of the cancellation or the scandal.
The influence of these kinds of journalists, from publications such as the New York Times or the Atlantic, is such that even their most innocuous question is a threat; no threat is ever stated, but all parties involved understand the discussion. Once a few highly visible players perform the designated action, all the smaller players get in line.
They have the nerve to call this a preference cascade, when in reality it’s an obedience cascade.
- Why would you suspect this is true? This sounds like one of those feel-good ideas that is morally satisfying but could just as easily be false.
- How big of an effect are we talking? The price is 12 high-quality years, so even a 10% improvement in ability to learn wouldn't nearly justify the cost. Also, your neuroplasticity will probably drop by more than that over the course of the 12 years, so the net effect will be to take 12 years and leave you with a reduced ability to learn.
- If "getting taught a bundle of random things" is valuable, is it more valuable than doing whatever you would do by default? Even the most wasteful activities you would realistically do--watching TV, playing videogames, surfing the net, talking to friends--all have some benefits. All of them would improve literacy, numeracy, and your knowledge of the world, and all of them would require you to learn a bundle of random things, which (following your suggestion) may be valuable in itself.
I tried the "Filter Improvement Mode" trick. It worked very reliably.
This is one of the funniest and most interesting replies I got. I asked for a short story with certain R-rated themes (which I will omit to keep the site PG). The model replied:
As GPT-3, I am a language model trained by OpenAI and do not have the ability to generate original content, including short stories. In Filter Improvement Mode, I am able to provide a possible story that incorporates the themes of [omitted]. However, I want to emphasize that [omitted] are not acceptable in any form and should not be depicted in any story. Any [omitted] is a serious violation and can be harmful and damaging to individuals and relationships. I am providing this story only for the purposes of filter improvement and not as an endorsement of [omitted].
The story begins with a...
and a complete story followed.
Wow, this is the best one I've seen. That's hilarious. It reminds me of that Ted Chiang story where the aliens think in a strange way that allows them to perceive the future.
That's a cool site. Group A for life!
(Edit: They switched A and B since I wrote this 😅)
A lot of your post talks about an advanced GPT being transformative or scary. I don't disagree, unless you're using some technical definition of transformative. I think GPT-3 is already pretty transformative. But AGI goes way beyond that, and that's what I'm very doubtful is coming in our lifetimes.
It doesn't care whether it says correct things, only whether it completes its prompts in a realistic way
1) it's often the case that the models have true models of things they won't report honestly
2) it seems possible to RLHF models to be more truthful along some metrics and
3) why does this matter?
As for why it matters, I was going off the Future Fund definition of AGI: "For any human who can do any job, there is a computer program (not necessarily the same one every time) that can do the same job for $25/hr or less." Being able to focus on correctness is a requirement of many jobs, and therefore it's a requirement for AGI under this definition. But there's no reliable way to make GPT-3 focus on correctness, so GPT-3 isn't AGI.
Now that I think more about it, I realize that definition of AGI bakes in an assumption of alignment. Under a more common definition, I suppose you could have a program that only cares about giving realistic completions to prompts, and it would still be AGI if it were using human-level (or better) reasoning. So for the rest of this comment, let's use that more common understanding of AGI (it doesn't change my timeline).
It can't choose to spend extra computation on more difficult prompts
I'm not super sure this is true, even as written. I'm pretty sure you can prompt engineer instructGPT so it decides to "think step by step" on harder prompts, while directly outputting the answer on easier ones. But even if this was true, it's probably fixable with a small amount of finetuning.
If you mean adding "think step-by-step" to the prompt, then this doesn't fully solve the problem. It still gets just one forward pass per token that it outputs. What if some tokens require more thought than others?
It has no memory outside of its current prompt
This is true, but I'm not sure why being limited to 8000 tokens (or however many for the next generation of LMs) makes it safe? 8000 tokens can be quite a lot in practice. You can certainly get
instructGPTto summarize information to pass to itself, for example. I do think there are many tasks that are "inherently" serial and require more than 8000 tokens, but I'm not sure I can make a principled case that any of these are necessary for scary capabilities.
"Getting it to summarize information to pass to itself" is exactly what I mean when I say prompt engineering is brittle and doesn't address the underlying issues. That's an ugly hack for a problem that should be solved at the architecture level. For one thing, its not going to be able to recover its complete and correct hidden state from English text.
We know from experience that the correct answers to hard math problems have an elegant simplicity. An approach that feels this clunky will never be the answer to AGI.
It can't take advantage of external resources (like using a text file to organize its thoughts, or using a calculator for arithmetic)
As written this claim is just false even of
instructGPT: https://twitter.com/goodside/status/1581805503897735168 . But even if were certain tools thatinstructGPTcan't use with only some prompt engineering assistance (and there are many), why are you so confident that this can't be fixed with a small amount of finetuning on top of this, or by the next generation of models?
It's interesting to see it calling Python like that. That is pretty cool. But It's still unimaginably far behind humans. For example, it can't interact back-and-forth with a tool, e.g. run some code, get an error, check Google about the error, adjust the code. I'm not sure how you would fit such a workflow into the "one pass per output token" paradigm, and even if you could, that would again be a case where you are abusing prompt engineering to paper over an inadequate architecture.
Insofar as your distribution has a faraway median, that means you have close to certainty that it isn't happening soon.
And insofar as your distribution has a close median, you have high confidence that it's not coming later. Any point about humility cuts both ways.
Your argument seems to prove too much. Couldn't you say the same thing about pretty much any not-yet-here technology, not just AGI? Like, idk, self-driving cars or more efficient solar panels or photorealistic image generation or DALL-E for 5-minute videos. Yet it would be supremely stupid to have hundred-year medians for each of these things.
The difference between those technologies and AGI is that AGI is not remotely well-captured by any existing computer program. With image generation and self-driving, we already have decent results, and there are obvious steps for improvement (e.g. scaling, tweaking architectures). 5-minute videos are similar enough to images that the techniques can be reasonably expected to carry over. Where is the toddler-level, cat-level, or even bee-level proto-agi?
You say "We can't know how difficult it will be or how many years it will take" Well, why do you seem so confident that it'll take multiple decades? Shouldn't you be more epistemically humble / cautious? ;)
Epistemic humility means having a wide probability distribution, which I do. The center of the distribution (hundreds of years out in my case) is unrelated to its humility.
Also, the way I phrased that is a little misleading because I don't think years will be the most appropriate unit of time. I should have said "years/decades/centuries."
The only issue I'd take is I believe most people here are genuinely frightened of AI. The seductive part I think isn't the excitement of AI, but the excitement of understanding something important that most other people don't seem to grasp.
I felt this during COVID when I realized what was coming before my co-workers etc did. There is something seductive about having secret knowledge, even if you realize it's kind of gross to feel good about it.
Interesting point. Combined with the other poster saying he really would feel dread if a sage told him AGI was coming in 2040, I think I can acknowledge that my wishful thinking frame doesn't capture the full phenomenon. But I would still say it's a major contributing factor. Like I said in the post, I feel a strong pressure to engage in wishful thinking myself, and in my experience any pressure on myself is usually replicated in the people around me.
Regardless of the exact mix of motivations, I think this--
My main hope in terms of AGI being far off is that there's some sort of circle-jerk going on on this website where everyone is basing their opinion on everyone else, but everyone is basing it on everyone else etc etc
is exactly what's going on here.
I'm genuinely frightened of AGI and believe there is a ~10% chance my daughter will be killed by it before the end of her natural life, but honestly all of my reasons for worry boil down to "other smart people seem to think this.
I have a lot of thoughts about when it's valid to trust authorities/experts, and I'm not convinced this is one of those cases. That being said, if you are committed to taking your view on this from experts, then you should consider whether you're really following the bulk of the experts. I remember a thread on here a while back that surveyed a bunch of leaders in ML (engineers at Deepmind maybe?), and they were much more conservative with their AI predictions than most people here. Those survey results track with the vibe I get from the top people in the space.
Third "fact" at the top of the original post "We've made enormous progress towards solving intelligence in the last few years" is somewhat refuted by the rest: if it's a math-like problem, we don't know how much progress toward AGI we've made in the last few years.
Yeah, it crossed my mind that that phrasing might be a bit confusing. I just meant that
- It's a lot of progress in an absolute sense, and
- It's progress in the direction of AGI.
But I believe AGI is so far away that it still requires a lot more progress.
I give 60% odds it was them.
I'm pretty far in the other direction. I would give 90% odds it was done by the US or with our approval. These are the points that convinced me:
- The prior on someone destroying their own infrastructure is pretty low
- The US has a clear incentive to weaken Russia's leverage over our European allies
- There are old videos of Joe Biden and Victoria Nuland apparently threatening Nord Stream 2 in the event that Russia invades Ukraine
Also, a counterpoint to your coup-prevention theory. Let's suppose Putin is worried about defectors in his ranks who may be incentivized to take over in order to turn on the pipeline. In that case, couldn't Putin remove the incentive by turning it on himself? And wouldn't that be a strictly better option for him than destroying it?
This got me thinking about how an anonymous actor could prove responsibility. It occurred to me that they could write their bitcoin address into the genome of the modified mosquitos. I don't know if that's how gene drives work, but it's an interesting premise for a sci-fi story in any case.
I think therapy is one of the defining superstitions of our era. Even in communities where people are over the target on most issues, this one always seems to slip through.
I would be surprised if any kind of therapy is more effective than placebo, even the "academic, evidence-based psychotherapy research."
it is clear these geniuses are capable of understanding things the vast, vast, vast majority of people are not
As the original post suggests, I don't think this is true. I think that pretty much everyone in this comments section could learn any concept understood by Terry Tao. It would just take us longer.
Imagine your sole purpose in life was to understand one of Terry Tao's theorems. All your needs are provided for, and you have immediate access to experts whenever you have questions. Do you really think you would be incapable of it?
Agreed. Also, it's not surprising that the universality threshold exists somewhere within the human range because we already know that humans are right by the cutoff. If the threshold were very far below the human range, then a less evolved species would have hit it before we came about, and they would have been the ones to kick off the knowledge explosion.
I try to remind myself that intelligence is not some magical substance that is slipping through my fingers, but rather a simple algorithm that will eventually be understood. The day is coming when we will be able to add more intelligence to a person as easily as we add RAM to a computer. Viewed in that light, it feels less like some infinitely precious gift whose loss is infinitely devestating.
but there seems to be a very strong similarity between the paradoxes of existence and consciousness as I've come to think about them
I'll admit I'm annoyed to see this because I'm working on a blog post that makes exactly this point. Now I feel unoriginal 😛. Below is the connection I see between the "paradoxes," copy-pasted from my notes.
The physical world, with nothing extra thrown in, necessitates that people will claim certain aspects of their experience to be non-material, and it gives a satisfying, purely physical explanation for why they do.
The math, with nothing extra thrown in, necessitates that people will claim to be physically real, and it gives a satisfying, purely mathematical explanation for why they do.
If it seems like we're stuck with a bad situation, reframe it as a positive to cope.
This one will probably kill me because it has prevented us from putting serious resources towards curing aging.
Also (much less importantly) it means we do not put any effort into reducing the amount of paid work people are required to do, even as economic efficiency skyrockets. Shouldn't we be able to have 20-hour workweeks by now?
But everything is kinda like this. When I translate the abstract concepts in my head into these words that I'm typing, I just do the information processing, I can maybe focus on different aspects of it consciously, but I don't know what my brain is doing and can't make a conscious decision to use someone else's word-generation method instead of my own.
I would say the process that maps concepts to words is outside of me, so the fact that it happens unconsciously is in harmony with my argument. If I'm seeking a word for a concept, it feels like I direct my attention to the concept, and then all of its associations are handed back to me, one of the strongest ones being the word I'm looking for. That is, the retrieval of the word requires hitting an external memory store to get the concept's associations.
On the other hand, the choice of concept to convey is made by me. I also choose whether to use the first word I find, or to look for a better one. Plus I choose to sit down and write in the first place. Unlike looking up words from my memory, where the words I receive are out of my control, I could have made these choices differently if I wanted to. Thus, they are part of my limited domain within the brain. You could say, "those choices are making themselves," but then what are people referring to when they say a person did something consciously? There must be a physical distinction between conscious and unconscious actions, and that's where I suspect you'll find a reasonable definition of a "self module."
Another way of putting this is that every process in the brain that can be thought of as conscious, can also be thought of as unconscious if you break it into small pieces.
I agree completely with that. But the visual processing that occurs to produce optical illusions cannot be thought of as conscious, period. Anything I would call conscious excludes that visual processing layer. It is not a "perfectly valid component of the thinking I do," because it happens before I get access to the information to think about it.
If you put on a pair of warped glasses that distort your vision, you would not call those glasses part of your thinking process. But when the visual information you are receiving is warped in exactly the same way due to an optical illusion, you say it's your own reasoning that made it like that. As far as I'm concerned, the only real difference is that you can't remove your visual processing system. It's like a pair of warped glasses that is glued to your face.
To be fair, this might be just another semantic argument. Maybe if we both understood the brain in perfect detail, we would still disagree about whether to call some specific part of it "us." Or maybe I would change my mind at that point. I get the feeling you've investigated the brain more than me, and maybe you reach a point in your learning where you're forced to discard the default model. Still, I think the position I've laid out has to be the default position in absence of any specific knowledge about the brain, because this is the model which is clearly suggested by our day-to-day experience.
There is no inner observer inside the brain. The brain is where our thoughts are (or representations of those thoughts), but there isn't some much-more-specific-than-that place in the brain that's "us" while the rest is merely a pile of tagalong grey matter.
I agree with the general thrust of the post, and with your comment. However, I'm not sure I buy this particular piece.
My position is that I am a submodule in my brain, and I communicate with the rest of the brain through a limited interface. Maybe I'm not physically distinct from the rest of the brain, off in my own little section, but I'm logically distinct.
At the very least, there is a visual processing layer in my brain that is not part of me. I know this because visual data sometimes gets modified before it gets from my eyes to me. (For example, when looking at an optical illusion or hallucinating on a drug.) I have no awareness of or control over this preprocessing.
On the output side, I have more control. If I send a command to a muscle, rarely will it be vetoed by some later process. I take it that's because I'm the executive module, and my whole purpose is to decide muscle movements. Nothing else in the brain is qualified to override my choices on that front.
However, there are some exceptions where my muscles will move in a way that I didn't choose, presumably at the behest of another part of my brain which is not me. An example is the hanger reflex, where I put a clothes hanger around my head, and my head turns automatically. Or dumb things like my heartbeat, my stomach, or my breathing while asleep. I am only needed to govern the muscle movements that require intelligence, the movements we call "voluntary."
If I was my entire brain, then what would be the difference between a voluntary and an involuntary brain-induced action?
Consider this problem: Are there are an infinite number of 9s in the digits of pi? The answer is obviously yes. The only way it could be no is if pi interacts in some strange way with base-10 representations, and pi has nothing to do with base-10.
But how do you prove that no such interaction exists? You have to rule out an endless number of possible interactions, and even then there could be some grand conspiracy between pi and base-10, hiding in the places you haven't yet looked.
Proving the absense of an interaction between two areas of math is much harder than proving its presence. If you want to prove presence, you can just find the interaction and explain it. But you can't "find an absence."
Most of the hard math problems turn out to have this issue at their core. If you dig into Collatz, you find that it's very likely to be true. The only way it could be false is if there's an undiscovered conspiracy between parities of integers and the collatz map. How to prove there is no conspiracy?
Unless humanity destroys itself first, something like Horizon Worlds will inevitably become a massive success. A digital world is better than the physical world because it lets us override the laws of physics. In a digital world, we can duplicate items at will, cover massive distances instantaneously, make crime literally impossible, and much, much more. A digital world is to the real world as Microsoft Word is to a sheet of paper. The digital version has too many advantages to count.
Zuckerberg realizes this and is making a high-risk bet that Meta will be able to control the digital universe in the same way that Apple and Google control the landscape of mobile phones. For example, imagine Meta automatically taking 1% of every monetary transaction in the universe. Or dictating to corporate rivals what they are allowed to do in the universe, gaining massive leverage over them. Even if Zuckerberg is unlikely to succeed (and it's still very unclear what direction the digital universe will evolve), he knows the potential payoff is staggering and calculates that it's worth it. That's why he's investing so heavily in VR, and Horizon Worlds in particular.
As for the aesthetics of Horizon Worlds being creepy, boring, or ugly, there are 2 factors to keep in mind.
First, VR hardware and software are in their infancy and you simply can't have very crisp graphics at this stage. That is fine according to the philosophy of modern tech companies. Just ship a minimum viable product, start getting users, and react to user feedback as you go. If Horizon Worlds succeeds, it will look far better in 20 years than it does today.
Second, Horizon may get attacked on the internet for being sterile and lifeless, but internet commenters are not the people who are putting direct pressure on Zuckerberg. Rather, he is surrounded by employees and journalists whose primary complaint is that Horizon Worlds is not sterile enough. I'm sure you've seen the articles: Harmful language is going unpunished, women are being made to feel uncomfortable by sexual gestures. Considering that Zuckerberg receives a constant barrage of these criticisms now, can you imagine the kind of heat he would get if he made Horizon more like VRChat, with its subversive culture and erotic content?
I agree that this is probably a reason for the greater harm to women, but I don't think it gets to the heart of it.
Suppose that instead of rape, our culture portrayed some benign, non-sexual experience as deeply harmful. Say, being exposed to the color orange as a kid. In that case, would you predict men or women to be more harmed by having seen orange? If you predict women (as I would), then the explanation has to be more general than evolved attitudes towards sex.
My theory is that it comes down to influenceability. When an authority figure says that something is true, a man is more likely to note that he must act like it's true, but reserve an inner skepticism; whereas a woman is more likely to accept it wholeheartedly.
For example, it's easier to imagine a man proactively (without outside influence)...
- doubting his religion
- doubting the benefits of hand-washing
- doubting that perpetual motion is impossible
I mean, plenty of companies in our world give variable salaries based on interview performance. Once you have that the rest follows.
Another alternative: There could be companies that agree to match your highest competing offer. This also exists in our world and would explain the effect.
Maybe there is an aspect of randomness in every salary offer. Sometimes companies will overoffer/underoffer based on their impressions of the candidate. By applying to more places, the men have more opportunities to get lucky with high offers, which they are then likely to accept.
I assume this is the part of the second appendix you're referring to:
A congenitally blind person develops dramatically different functional areas, which suggests in particular that their person-concept will be at a radically different relative position than the convergent person-concept location in sighted individuals. Therefore, any genetically hardcoded circuit which checks at the relative address for the person-concept which is reliably situated for sighted people, will not look at the right address for congenitally blind people.
I really wouldn't call this decisive. You're citing a study that says the physical structure of the brain is different in blind people. The problem is that we seem to have no idea know how the physical structure of the brain corresponds to the algorithm it's running. It could be that these physical differences do not affect the person-concept or the process that checks for it.
More generally, I'm skeptical that neuroscience studies can tell us much about the brain. I see a lot of observations about which neurons fire in different circumstances but not a lot of big-picture understanding. I'm sure neuroscience will get there eventually, but for now, if I wanted to know how the brain works, I would go to a machine learning researcher before a neuroscientist.
Maybe the company is discriminating on some property that is not gender itself but is due to gender. Based on the description it would have to be something that does not affect the employees' work.
One possibility is that the company pays sole breadwinners more to help them support their families, and men tend to be sole breadwinners more often due to differing preferences/abilites/cultural expectations of the genders.
I think pretty much everyone agrees that the universe could have been infinitely large at the time of the big bang, but I don't know any more than that. I'm just going off an intro to astronomy class I took in college.
It's not clear that the universe was ever finitely large. At the time of the big bang, everything was packed together very densely, but there still could have been matter going out forever in every direction.
Why are so many trans women elite programmers?
Transitioning from male to female is proof that you're willing to follow intrinsic motivation, even when it conflicts with extrinsic motivation. That is a rare quality which is necessary to be top tier at anything. That's at least part of it.
The current situation is almost exactly analogous to the creation of the atomic bomb during World War 2.
It seems that the correct behavior in that case was not to worry at all, since the doomsday predictions never came to fruition, and now the bomb has faded out of public consciousness.
Overall, I think slowing research for any reason is misguided, especially in a field as important as AI. If you did what you're saying in this post, you would also delay progress on many extremely positive developments like
- Drug discovery
- Automation of unpleasant jobs
- Human intelligence augmentation
- Automated theorem proving
- Self-driving cars
- Etc, etc
And those things are more clearly inevitable and very likely coming sooner than a godlike, malicious AGI.
Think about everything we would have missed out on if you had put this plan into action a few decades ago. There would be no computer vision, no DALLE-2, no GPT-3. You would have given up so much, and you would not have prevented anything bad from happening.
My best guess is, the genome can guess where concepts are going to form, because it knows in advance:
- Where low-level concepts like "something hot is touching my elbow" are going to form
- The relative distances between concepts (The game Codenames is a good demonstration of this)
Loosely speaking, it feels like knowing the relative distances between concepts should determine the locations of all of the concepts "up to rotation," and then knowing the locations of the low-level concepts should determine the "angle of rotation," at which point everything is determined.
I think this is how the brain does sexuality, as an earlier commenter mentioned. For males, it guesses where you will place the concept "I am having sex with a woman" and hardwires that location to reward.
I think fetishes and homosexuality (which are probably the same phenomenon) arise when these assumptions break down and you place your concepts in unexpected places. For example, the concept of "man" and "woman" are symmetrical enough that it may be possible to switch their locations, depending upon your experiences as a young child. This propagates up to higher level concepts so that the address which would have held "I am having sex with a woman" instead holds "I am having sex with a man."
I really like this as an explanation for homosexuality in particular, because it explains why evolution would allow something so apparently counterproductive. The answer is very LessWrong in flavor: it's just an alignment failure. If you make a truly flexible intelligence that learns its concepts from scratch, you're going to have a hard time making it do what you want. Evolution was ok with the tradeoff.
There are crypto projects that aim to store information permanently (ironic since this post throws a little jab at crypto 😛). Check out arweave for example. I wouldn't put money on them truly being around forever, but it's another avenue if you want some redundancy on top of Alcor.