The way I understand it the homunculus is part of self. So if you put the wanting in the homunculus, it's also inside self. I don't know about you, but my self concept has more than wanting. To be fair, he homunculus concept is also a bit richer than wanting (I think?) but less encompassing than the full self (I think?).
Reading this post is so strange. I've already read the draft, so it's not even new to me, but still very strange.
I do not recognise this homunculus concept you describe.
Other people reading this, do you experience yourself like that? Do you resonate with the intuitive homunculus concept as described in the post?
I my self have a unified self (mostly). But that's more or less where the similarity ends.
For example when I read:
in my mind, I think of goals as somehow “inside” the homunculus. In some respects, my body feels like “a thing that the homunculus operates”, like that little alien-in-the-head picture at the top of the post,
my intuitive reaction is astonishment. Like, no-one really think of themselves like that, right? It's obviously just a metaphor, right?
But that was just my first reaction. I know enough about human mind variety to absolutely believe that Steve has this experience, even though it's very strange to me.
Similarly, as Johnstone points out above, for most of history, people didn’t know that the brain thinks thoughts! But they were forming homunculus concepts just like us.
Why do you assume they where forming homunculus concepts? Since it's not veridical, they might have a very different self model.
I'm from the same culture as you and I claim I don't have homunculus concept, or at least not one that matches what you describe in this post.
(Neuroscientists obviously don’t use the term “homunculus”, but when they talk about “top-down versus bottom-up”, I think they’re usually equating “top-down” with “caused by the homunculus” and “bottom-up” with “not caused by the homunculus”.)
I agree that the homunculus-theory is wrong and bad, but I still think there is something to top-down vs bottom-up.
It's related to what you write later
Another part of the answer is that positive-valence S(X) unlocks a far more powerful kind of brainstorming / planning, where attention-control is part of the strategy space. I’ll get into that more in Post 8.
I think conscious control (aka top-down) is related to conscious thoughts (in the global work space theory sense) which is related to using working memory, to unlock more serial compute.
That said, if those sorts of concepts are natural in our world, then it’s kinda weird that human minds weren’t already evolved to leverage them.
A counter possibility to this that comes to mind:
There might be concepts that is natural in our world, but which are only useful for a mind with much more working memory, or other compute recourses than the human mind.
If weather simulations use concepts that are confusing and un-intuitive for most humans, this would be evidence for something like this. Weather is something that we encounter a lot, and is important for humans, especially historically. If we haven't developed some natural weather concept, it's not for lack of exposure or lack of selection pressure, but for some other reason. That other reason could be that we're not smart enough to use the concept.
Yesterday was the official application deadline for leading a project at the next AISC. This means that we just got a whole host of project proposals.
If you're interested in giving feedback and advise to our new research leads, let me know. If I trust your judgment, I'll onboard you as an AISC advisor.
Also, it's still possible to send us a late AISC project proposals. However we will prioritise people how applied in time when giving support and feedback. Further more, we'll prioritise less late applications over more late applications.
For me the dancer was spinning contraclockwise and would not change. With your screwing trick I could change rotation, and where now stably stuck in the clockwise direction. Until I screwed in the other direction. I've now done this back and forth a few times.
I have two hypothesises for what is going on. I'm leaning towards 1, but very unsure.
1)
king - man + woman = queen
is true for word2vec embeddings but not in LLaMa2 7B embeddings because word2vec has much fewer embedding dimensions.
LLaMa2 7B has 4096 embedding dimensions.
This paper uses a variety of word2vec with 50, 150 and 300 embedding dimensions.
Possibly when you have thousands of embedding dimensions, these dimensions will encode lots of different connotations of these words. These connotations will probably not line up with the simple relation [king - man + woman = queen], and therefore we get [king - man + woman ≠ queen] for high dimensional embeddings.
2)
king - man + woman = queen
Isn't true for word2vec either. If you do it with word2vec embeddings you get more or less the same result I did with LLaMa2 7B.
(As I'm writing this, I'm realising that just getting my hands on some word2vec embeddings and testing this for myself, seems much easier than to decode what the papers I found is actually saying.)
projection of "▁queen" on span( "▁king", "▁man", "▁woman") is ['▁king', '▁King', '▁woman', '▁queen', '▁rey', '▁Queen', 'peror', '▁prince', '▁roi', '▁König']
"▁queen" is the closest match only if you exclude any version of king and woman. But this seems to be only because "▁queen" is already the 2:nd closes match for "▁king". Involving "▁man" and "▁woman" is only making things worse.
I then tried looking up exactly what the word2vec result is, and I'm still not sure.
Wikipedia sites Mikolov et al. (2013). This paper is for embeddings from RNN language models, not word2vec, which is ok for my purposes, because I'm also not using word2vec. More problematic is that I don't know how to interpret how strong their results are. I think the relevant result is this
We see that the RNN vectors capture significantly more syntactic regularity than the LSA vectors, and do remarkably well in an absolute sense, answering more than one in three questions correctly.
which don't seem very strong. Also I can't find any explanation of what LSA is.
I also found this other paper which is about word2vec embeddings and have this promising figure
But the caption is just a citation to this third paper, which don't have that figure!
I've not yet read the two last papers in detail, and I'm not sure if or when I'll get back to this investigation.
If someone knows more about exactly what the word2vec embedding results are, please tell me.
I don't think seeing it as a one dimensional dial, is a good picture here.
The AI has lots and lots of sub-circuits, and many* can have more or less self-other-overlap. For “minimal self-other distinction while maintaining performance” to do anything, it's sufficient that you can increase self-other-overlap in some subset of these, without hurting performance.
* All the circuits that has to do with agent behaviour, or beliefs.
I think the main important lesson is to not get attached to early ideas. Instead of banning early ideas, if anything comes up, you can just write tit down, and set it aside. I find this easier than a full ban, because it's just an easier move to make for my brain.
(I have a similar problem with rationalist taboo. Don't ban words, instead require people to locally define their terms for the duration of the conversation. It solves the same problem, and it isn't a ban on though or speech.)
The other important lesson of the post, is that, in the early discussion, focus on increasing your shared understanding of the problem, rather than generating ideas. I.e. it's ok for ideas to come up (and when they do you save them for later). But generating ideas is not the goal in the beginning.
Hm, thinking about it, I think the mechanism of classical brainstorming (where you up front think of as many ideas as you can) is to exhaust all the trivial, easy to think of, ideas, as fast as you can, and then you're forced to think deeper to come up with new ideas. I guess that's another way to do it. But I think this is method is both ineffective and unreliable, since it only works though a secondary effect.
. . .
It is interesting to comparing the advise in this post with the Game Tree of Aliment or Builder/Breaker Methodology also here. I've seen variants of this exercise popping lots of places in the AI Safety community. Some of them pare probably inspired by each other, but I'm pretty sure (80%) that this method have been invented several times independently.
I think that GTA/BBM works for the same reason the advice in the post works. It also solves the problem of not getting attached, and also as you keep breaking your ideas and explore new territory, you expand your understanding or the problem. I think an active ingrediens in this method is that the people playing this game knows that alignment is hard, and go in expecting their first several ideas to be terrible. You know the exercise is about noticing the flaws in your plans, and learn from your mistakes. Without this attitude, I don't think it would work very well.
I think the main important lesson is to not get attached to early ideas. Instead of banning early ideas, if anything comes up, you can just write tit down, and set it aside. I find this easier than a full ban, because it's just an easier move to make for my brain.
(I have a similar problem with rationalist taboo. Don't ban words, instead require people to locally define their terms for the duration of the conversation. It solves the same problem, and it isn't a ban on though or speech.)
The other important lesson of the post, is that, in the early discussion, focus on increasing your shared understanding of the problem, rather than generating ideas. I.e. it's ok for ideas to come up (and when they do you save them for later). But generating ideas is not the goal in the beginning.
Hm, thinking about it, I think the mechanism of classical brainstorming (where you up front think of as many ideas as you can) is to exhaust all the trivial, easy to think of, ideas, as fast as you can, and then you're forced to think deeper to come up with new ideas. I guess that's another way to do it. But I think this is method is both ineffective and unreliable, since it only works though a secondary effect.
. . .
It is interesting to comparing the advise in this post with the Game Tree of Aliment or Builder/Breaker Methodology also here. I've seen variants of this exercise popping lots of places in the AI Safety community. Some of them pare probably inspired by each other, but I'm pretty sure (80%) that this method have been invented several times independently.
I think that GTA/BBM works for the same reason the advice in the post works. It also solves the problem of not getting attached, and also as you keep breaking your ideas and explore new territory, you expand your understanding or the problem. I think an active ingrediens in this method is that the people playing this game knows that alignment is hard, and go in expecting their first several ideas to be terrible. You know the exercise is about noticing the flaws in your plans, and learn from your mistakes. Without this attitude, I don't think it would work very well.
This already strongly suggests some connection between induction heads and in-context learning, but beyond just that, it appears this window is a pivotal point for the training process in general: whatever's occurring is visible as a bump on the training curve (figure below). It is in fact the only place in training where the loss is not convex (monotonically decreasing in slope).
I can see the bump, but it's not the only one. The two layer graph has a second similar bump, which also exists in the one layer model, and I think I can also see it very faintly in the three level model. Did they ignore the second bump because it only exists in small models, while their bump continues to exist in bigger models?
I've recently crossed into being considered senior enough as an organiser, such that people are asking me for advise on how to run their events. I'm enjoying giving out advise, and it also makes me reflet on event design in new ways.
I think there are two types of good events.
Purpose driven event design.
Unconference type events
I think there is a continuum between these two types, but also think that if you plot the best events along this continuum, you'll find a bimodal distribution.
Purpose driven event design
When you organise one of these, you plan a journey for your participant. Everything is woven into a specific goal that is active by the end of the event. Everything fits together.
These can defiantly have a purpose (e.g. exchanging of ideas) but the purpose will be less precise than for the previous type, and more importantly, the purpose does not strongly drive the event design.
There will be designed elements around the edges, e.g. the opening and ending. But most of the event design just goes into supporting the unconference structure, which is not very purpose specific. For most of the event, the participants will not follow a shared journey, currented by the organisers, instead everyone is free to pick their own adventure.
Some advise from The Art of Gathering works for unconference type events, e.g. the importance of pre-event communication, opening and ending. But a lot of the advise don't work, which is why I noticed this division in the first place.
Strengths and weaknesses of each type
Purpose driven events are more work to do, because you actually have to figure out the event design, and then you probably also have to run the program. With unconferences, you can just run the standard unconference format, on what ever theme you like, and let your participants do most of the work of running the program.
An unconference don't require you to know what the specific purpose of the event is. You can just bring together an interesting group of people and see what happens. That's how you get Burning Man or LWCW.
However if you have a specific purpose you want to active, you're much more likely to succeed if you actually design the event for that purpose.
There are lots of things that a unconferences can't do at all. It's a very broadly applicable format, but not infinitely so.
I feel a bit behind on everything going on in alignment, so for the next weeks (or more) I'll focus on catching up on what ever I find interesting. I'll be using my short form, to record my though.
I make no promises that reading this is worth anyone's time.
Linda's alignment reading adventures part 1
What to focus on?
I do have some opinions on what aliment directions are more or less promising. I'll probably venture in other directions too, but my main focus is going to be around what I expect an alignment solution to look like.
I think that to have an aligned AI it is necessary (but not sufficient) that we have shared abstractions/ontology/concepts/ (what ever you want to call it) with the AI.
I think the way to make progress on the above is to understand what ontology/concepts/abstraction our current AIs are using, and the process that shapes these abstraction.
I think the way to do this is though mech-interp, mixed with philosophising and theorising. Currently I think the mech-interp part (i.e. look at what is actually going on in a network) is the bottleneck, since I think that philosophising with out data (i.e. agent foundations) has not made much progress lately.
Conclusion:
I'll mainly focus on reading up on mech-interp and related areas such as dev-interp. I've started on the interp section of Lucius's aliment reading list.
I should also read some John Wentworth, since his plan is pretty close to the path I think is most promising.
But also, how interesting is this. Basically they removed the cheese observation, it made the agent act as if there where no cheese. This is not some sophisticated steering technique that we can use to align the AIs motivation.
I discussed this with Lucius who pointed out, that the interesting result is that: The the cheese location information is linearly separable from other information, in the middle of the network. I.e. it's not scrambled in a completely opaque way.
Alon’s book is the ideal counterargument to the idea that organisms are inherently human-opaque: it directly demonstrates the human-understandable structures which comprise real biological systems.
Both these posts are evidence for the hypothesis that we should expect evolved networks to be modular, in a way that is possible for us to decode.
By "evolved" I mean things in the same category as natural selection and gradient decent.
I agree that the reason EAs are usually not tracking favours is that we are (or assume we are) mission aligned. I picked the pay-it-forward framing, because it fitted better with other situation where I expected people not to try to balance social legers. But you're right that there are situations that are mission aligned that are not well described as paying it forward, but some other shared goal.
Another situation where there is no social ledger, is when someone is doing their job. (You talk about a company where people are passionate about the mission. But most employs are not passionate about the mission, and still don't end up owing each other favours for doing their job.)
I personally think that the main benefit of mental health professionals (e.g. psychologies, coaches, etc) is that you get to have a one sided relationship, where you get to talk about all your problem, and you don't owe them anything in return, because you're paying them instead. (Or sometimes the healthcare system, is paying, or your employer. The important thing is that they get paid, and helping you is their job.)
(I much rather talk to a friend about my problems, it just works better, since they know me. But when I do this I owe them. I need to track what cost I'm imposing them, and make sure it's not more than I have the time and opportunity to re-pay.)
The desire to display gigawatt devotion with zero responsibility is the standard maneuver of our times, note the trend of celebrity soundbite social justice, or children’s fascination with doing the extra credit more than the regular credit, and as a personal observation this is exactly what’s wrong with medical students and nurses. They’ll spend hours talking with a patient about their lives and feelings while fluffing their pillow to cause it to be true that they are devoted - they chose to act, chose to love - while acts solely out of ordinary duty are devalued if not completely avoided.
(I'm pretty sure the way to read Sady Porn (or Scott's review of it) is not to treat any of the text as evidence for anything, but as suggestions of things that may be interesting to pay attention to.)
In the real network, there are a lot more than two activations. Our results involve a 32,768-dimensional cheese vector, subtracted from about halfway through the network:
Did you try other locations in the network?
I would expect it to work pretty much anywhere, and I'm interested to know if my prediction is correct.
I'm pretty sure that what happens is (as you also suggest) that the agent stops seeing the cheese.
Imagine you did the cheese subtraction on the input layer (i.e. the pixel values of the maze). In this case this just trivially removed the cheese from the picture, resulting in behaviour that is identical to no cheese. So I expect something similar to happen to later layer, as long as what the network is mostly doing is just de-coding the image. So at what ever layer this trick stops working, this should mean that the agent has started planing it's moves.
“someone complimented me out of the blue, and it was a really good compliment, and it was terrible, because maybe I secretly fished for it in some way I can’t entirely figure out, and also now I feel like I owe them one, and I never asked for this, and I’m so angry!”
A blogpost I remember but can't find. The author talks about the benefits of favour economy. E.g. his friend could help him move at much lower total cost, than the market price for moving. This is because the market has much higher transaction cots than favours among friends. The blogpost also talks about how you only get to participate in the favour economy (and get it's benefits) if you understand that you're expected to return the favours, i.e. keep track of the social ledger, and make sure to pay your depts. Actually you should be overpaying when returning a favour, and now they owe you, and then they over pay you back, resulting in a virtual cycle of helping each other out. The author mentions being in a mutually beneficial "who can do the biggest favour" competition with his neighbour.
3)
And EA friend who told me that they experienced over and over in the EA community that they helped people (e.g. letting them stay at their house for free) but where denied similar help in return.
4)
A youtuber (who where discussing this topic) mentioned their grand mother who always gives everyone in the family lots of food to take home, and said that she would probably be offended if offered something in return for this help.
Todays thoughts:
I think there are pros and cons that comes with internally tracking a social ledger for favours. I also think there are things that makes this practice better or worse.
On receiving compliments
In the past, when ever I got a compliment, I felt obligated to give one back, so instead of taking it in, I started thinking as fast I as could what I could compliment back. At some point (not sure when) I decided this was dumb. Now if I get a compliment I instead take a moment to take it in, and thank the other person. I wish others would do this more. If I give someone a compliment, I don't want them to feel obligated to complement me back.
On paying forward instead of paying back
Some people experience the EA community as being super generous when it comes to helping each other out. But other have the opposite experience (see point 3 above). I've personally experienced both. Partly this is because EA is large and diverse, but partly I think it comes from thronging out the social ledger. I think the typical EA attitude is that you're not expected to pay back favours, you're expected to pay it forward, into what ever cause area you're working on. People are not even attempting to balancing the ledger of inter-personal favours, and therefore there will be imbalance, some people receiving more and some giving more.
I think there are lots of other pay-it-forward communities. For example any system where the older "generation" is helping the younger "generation". This can be parents raising their kids, or a marshal arts club where the more experienced are helping train the less experienced, etc.
I think a many of these are pretty good at paying the favour givers in status points. In comparison I expect EA to not be very good at giving status for helping out others in the community, because EA has lots of other things to allocate status points to too.
How to relate to the ledger
I think the social ledger defiantly have it's use, even among EA types. You need friends you can turn to for favours, which is only possible in the long term if you also return the favours, or you'll end up draining your friends.
In theory we could have grant supported community builders act as support for everyone else, with no need to pay it back. But I rather not. I do not want to trust the stability of my social network to the whims of the EA funding system.
To have a healthy positive relation with the social ledger,
You have to be ok with being in debt. It's ok to owe someone a favour. Being the sort of person that pays back favours, don't mean you have to do it instantly. You are allowed to wait for your friend to need help from you, and not worry about it in the mean time. You're also allowed to more or less actively look out for things to do for them, both is ok. Just don't stress over it.
Have some slack, at least some times. I have very uneven amount of slack in my life, so I try to be helpful to others when I have more, and less so when I have less. But someone who never have slack will never be able to help others back.
You can't be too nit-picky about it. The social ledger is not an exact thing. As a rule of thumb, if both you and your friend individually get more out of your friendship than you put in, then things are good. If you can't balance it such that you both get more out than you put in, then you're not playing a positive sum game, and you should either fix this, or just hang out less. It's supposed to be a positive sum game.
You can't keep legers with everyone. Or maybe you can? I can't. That's too many people to remember. My brain came with some pre-installed software to keep track of the social leger. This program also decides who to track or not, and it works well enough. I think it turns on when someone has done a favour that reaches some minimum threshold?
There are also obviously situations where keeping track of some leger don't make sense, and you should not do it, e.g. one-off encounters. And sometimes paying forward instead of paying back is the correct social norm.
Give your favours freely when it's cheap for you to do so, and when you don't mind if the favour is not paid back. Sometimes someone will return the favour.
The pre installed brain soft where
This probably varies a lot from person to person, and as all the mental traits it will be due to some mix of genetics and culture, and random other stuff.
For me, I'm not choosing whether to keep a social ledger or not, my brain just does that. I'm choosing how to interact with it and tweak it's function, by choosing what to pay attention to.
The way I experience this is that I feel gratitude when I'm in debt, and resentment when someone else have too much unacknowledged debt to me.
The gratitude makes me positively disposed to that person and more eager to help them.
The resentment flaggs to me that something is wrong. I don't need them to instantly re-pay me, but I need them to acknowledge the debt and that they intend to pay it back. If not, this is a sign that I should stop investing in that person.
If they disagree that I've helped them more than they helped me, then we're not playing a positive sum game, and we should stop.
If they agree that I helped them more, but also, they don't intend to return the favour... hm...? Does this ever happen? I could imagine this happening if the other person has zero slack. I think there are legit reasons to not be able to return favours, especially favours you did not ask for, but I also expect to not attempt to not want to build a friendship in this situation.
...thanks for reading my ramblings. I'll end here, since I'm out of time, and I think I've hit diminishing return on thinking about this...
Lightcone is banned from receiving any kind of OpenPhil funding
Why?
The reason for the ban is pretty crux-y. Are Lighitcone banned because OpenPhil dislikes you, or because you're too close so that would be a conflict of interests, or something else.
"As a AI safety community member, I predict I and others will be uncomfortable with 80k if this is where things end up settling, because of disagreeing. I could be convinced otherwise, but it would take extraordinary evidence at this point. If my opinions stay the same and 80k's also are unchanged, I expect this make me hesitant to link to and recommend 80k, and I would be unsurprised to find others behaving similarly."
But you did not say it (other than as a response to me). Why not?
I'd be happy for you to take the discussion with 80k and try to change their behaviour. This is not the first time I told them that if they list a job, a lot of people will both take it as an endorsement, and trut 80k that this is a good job to apply for.
As far as I can tell 80k is in complete denial on the large influence they have on many EAs, especially local EA community builders. They have a lot of trust, mainly for being around for so long. So when ever they screw up like this, it causes enormous harm. Also since EA have such a large growth rate (at any given time most EAs are new EAs), the community is bad at tracking when 80k does screw up, so they don't even loose that much trust.
On my side, I've pretty much given up on them caring at all about what I have to say. Which is why I'm putting so litle effort into how I word things. I agree my comment could have been worded better (with more effort), and I have tried harder in the past. But I also have to say that I find the level of extreme politeness, lot's of EA show towards high status orgs, to be very off-putting, so I never been able to imitate that style.
Again, if you can do better, please do so. I'm serious about this.
Someone (not me) had some success at getting 80k to listen, over at the EA forum version of this post. But more work is needed.
Actually, it's probably true if you don't confound for intelligence.
Autism is negatively correlated with intelligence, and if you're not very smart, everything gets harder. But I think it's wrong to see low intelligence as part of Autism. And even if you disagree, it's weird to classify a general intelligence problem as a specific social deficit problem.
But if you compare high functioning Autists with neurotypicals, in realistic enough settings, I'm convinced autists will be better at understanding autists than neurotypicals are at understanding autists. Although "realistic enough" might require the autists enough time to interact to spot each other as same type of person.
I don't put a lot of weight here on academic studies, over just all of my life experience. But in case you do: I did here of a study where autist worked better with other autists than neurotypicals with neurotypicals. I don't have the liks, sorry. Just my memory of someone I trust telling me about it.
The reason I don't trust academic studies on this topic, is that is really really hard to do them well, so most of them are not done well.
Empathy is a useful tool, I use it too to generate initial guesses about people. But I'm also aware that it's untrustworthy. In my experience it's common for neurotypicals to fail at this last step.
Neurotypicals are more accurate for other neurotypicals. Autists are more accurate for other autists.
Since there are more neurotypicals (are there? or just more people pretending to be?*) neurotypicals are statistically more often correct**. But I would still not claim that that is a higher level of social skill. Having higher skill level at a more common task, is not the same thing as having higher over all skill. This detail is very important for understanding autism.
* Not saying that autism is the majority. But there are more neurotypes out there, and probably lots that are better at masking than autists.
** In a statistically representative environment. When autists are free to self segregate, we no longer have problems. This is the main point actually. If it was just a one dimensional skill issue, concentrating lots of autists, would go terribly, since no one have social skills, but instead is's great. All the people I get along with the best are other autists (officially or self diagnosed).
that's, IMO, the mechanism by which empathy works
Empathy is a very unreliable source of information though. E.g. I feel empathy with my plushies.
However, we don’t conceptualize the board as endorsing organisations.
It don't matter how you conceptualize it. It matters how it looks, and it looks like an endorsement. This is not an optics concern. The problem is that people who trust you will see this and think OpenAI is a good place to work.
These still seem like potentially very strong roles with the opportunity to do very important work. We think it’s still good for the world if talented people work in roles like this!
How can you still think this after the whole safety team quit? They clearly did not think these roles where any good for doing safety work.
Edit: I was wrong about the whole team quitting. But given everything, I still stand by that these jobs should not be there with out at leas a warning sign.
As a AI safety community builder, I'm considering boycotting 80k (i.e. not link to you and reccomend people not to trust your advise) until you at least put warning labels on your job board. And I'll reccomend other community builders to do the same.
I do think 80k means well, but I just can't reccomend any org with this level of lack of judgment. Sorry.
I expect that this will be true for LLM token embeddings too. Have anyone checked this?
I also expect something similar to be true for internal LLM representations too, but that this might be harder to verify. However, maybe not, if you have interpretable SAE vectors?
As long as me and Remmelt are still alive and healthy a couple of months from now, then we're doing it.
Remmelt have organised 8 previous AISCs, I've been part of 3 of those. We know what we are doing. We know we can rely on each other, and we want to do this.
We just needed to make sure we have money to live and eat and such, before we could commit to running a next camp. But we have received the money now, so that's all good. Manifund have sent us the money, it's in our bank accounts.
I'll bet anyone who like, that there will be an AISC10 at 1:10 odds in your favour. I'm much more confident that that, but if you give me worse odds, then I don't think can be bothered about it.
I don't read much sensationalist tabloid, but my impression is that the things that get a lot of attention in the press, is things people can reasonable take either side of.
Scott Alexander writes about how everyone agrees that factory framing is terrible, but exactly because this overwhelming agreement, it get's no attention. Which is why PETA does outrageous things to get attention.
Soon, interacting with AI agents will be a part of daily life, presenting enormous regulatory and compliance challenges alongside incredible opportunities.
Norm Ai agents also work alongside other AI agents who have been entrusted to automate business processes. Here, the role of the Norm Ai agent is to automatically ensure that actions other AI agents take are in compliance with laws.
I'm not sure if this is worrying, because I don't think AI overseeing AI is a good solution. Or it's actually good, because, again, not a good solution, which might lead to some early warnings?
Sensationalist tabloid news stories and other outrage porn are not the opposite. These are actually more of the same. More edge cases. Anything that is divisive have the problem I'm talking about.
Fiction is a better choice.
Or even just completely ordinary every-day human behaviour. Most humans are mostly nice most of the time.
We might have to start with the very basic, the stuff we don't even notice, because it's too obvious. Things no-one would think of writing down.
The math in the post is super hand-wavey, so I don't expect the result to be exactly correct. However in your example, l up to 100 should be ok, since there is no super position. 2.7 is almost 2 orders of magnitude off, which is not great.
Looking into what is going on: I'm basing my results on the Johnson–Lindenstrauss lemma, which gives an upper bound on the interference. In the post I'm assuming that the actual interference is order of magnitude the same as the this upper bound. This assumption is clearly fails in your example since the interference between features is zero, and nothing is the same order of magnitude as zero.
I might try to do the math more carefully, unless someone else gets there first. No promises though.
I expect that my qualitative claims will still hold. This is based on more than the math, but math seemed easier to write down. I think it would be worth doing the math properly, both to confirm my claims, and it may be useful to have more more accurate quantitative formulas. I might do this if I got some spare time, but no promises.
my qualitative claims = my claims about what types of things the network is trading away when using super position
quantitative formulas = how much of these things are traded away for what amount of superposition.
Recently someone either suggested to me (or maybe told me they or someone where going to do this?) that we should train AI on legal texts, to teach it human values. Ignoring the technical problem of how to do this, I'm pretty sure legal text are not the right training data. But at the time, I could not clearly put into words why. Todays SMBC explains this for me:
Law is not a good representation or explanation of most of what we care about, because it's not trying to be. Law is mainly focused on the contentious edge cases.
Training an AI on trolly problems and other ethical dilemmas is even worse, for the same reason.
