Posts
Comments
1a -> Broadly agree. "Weaker" is an interesting word to pick here; I'm not sure whether an anarcho-primitivist society would be considered weaker or stronger than a communist one systemically. Maybe it depends on timescale. Of course, if this were the only size lever we had to move x-risk up and down, we'd be in a tough position - but I don't think anyone takes that view seriously.
1b -> Logically true, but I do see strong reason to think short term x-risk is mostly anthropogenic. That's why we're all here.
2 -> I do agree it would probably take a while.
3a -> Depends on how coarse or fine grained the distribution of resources is, a simple linear optimizer program would probably do the same job better for most coarser distribution schemes.
3b -> Kind of. I'm looking into them as a curiosity.
AI development is a tragedy of the commons
Per Wikipedia:
In economic science, the tragedy of the commons is a situation in which individual users, who have open access to a resource unhampered by shared social structures or formal rules that govern access and use, act independently according to their own self-interest and, contrary to the common good of all users, cause depletion of the resource through their uncoordinated action.
The usual example of a TotC is a fishing pond: Everyone wants to fish as much as possible, but fish are not infinite, and if you fish them faster than they can reproduce, you end up with less and less fish per catch.
AI development seems to have a similar dynamic: Everyone has an incentive to build more and more powerful AIs, because there is a lot of money to be made in doing so. But more and more powerful AIs being made increases the likelihood of an unstoppable AGI being made.
There are some differences, but I think this is the underlying dynamic driving AI development today. The biggest point of difference is that, whereas one person's overfishing eventually causes a noticeable negative effect on other fishers, and at the least does not improve their own catches, one firm building a more powerful AI probably does improve the economic situation of the other people who leverage it, up until a critical point.
Are there other tragedies of the commons that exhibit such non-monotonic behavior?
Would AGI still be an x-risk under communism?
1-bit verdict
Yes.
2-bit verdict
Absolutely, yes.
Explanation
An artificial general intelligence (AGI) is a computer program that can perform at least as good as an average human being can across a wide variety of tasks. The concept is closely linked to that of a general superintelligence, which can perform better than even the best human being can across a wide variety of tasks.
There are reasons to believe most, perhaps almost all, general superintelligences would end up causing human extinction. AI safety is a crossdisciplinary field of mathematics, economics, computer science, and philosophy which tackles the problem of how to stop such superintelligences.
AI alignment is a subfield of AI safety which studies theoretical conditions under which superintelligences aligned with human values can emerge. Another branch, which might be called AI deterrence, aims instead to make the production of unaligned superintelligences less likely in the first place.
One of the primary reasons why someone might want to create a superintelligence, even while understanding the risks involved, is because of the vast economic value such a program could generate. It makes sense then from a deterrence lens to look into the question of how this profit motive might be curtailed before catastrophe. Why not communism?
Unfortunately, this is almost certainly a bad move. Communism at almost every scale has to date never been able to escape the rampant black markets that appear due to the distortion of price signals. There is no reason to suspect such black markets wouldn't have just as strong a profit motive to create stronger and stronger AGIs. Indeed, because black markets are already illegal, this may worsen the problem: Well funded teams of people producing AGI outside of the eyes of the broader public is likely to generate less pushback and to be better equipped to avoid deterrence oriented legislation than a clear market team such as OpenAI is.
Towards a #1-flavored answer, a Hansonian fine insured bounty system seems like it might scale well for enforcing cooperation against AI research.
https://www.overcomingbias.com/2018/01/privately-enforced-punished-crime.html
OP here, talking from an older account because it was easier to log into on mobile.
Kill: I never said anything about killing them. Prisoners like this don't pose any immediate threat to anyone, and indeed are probably very skilled white collar workers who could earn a lot of money even behind bars. No reason you couldn't just throw them into a minimum security jail in Sweden or something and keep an eye on their Internet activity.
McCarthyism: Communism didn't take over in the US. That provides if anything weak evidence that these kinds of policies can work, even for suppressing much more controversial ideas than preventing the building of an unsafe AI.
q1: The hardcore answer would be "Sorry kid, nothing personal." If there was ever a domain where false positives were acceptable losses, stopping unaligned AI from being created in the first place would probably be it. People have waged wars for far less. The softcore answer, and the one I actually believe, is that you're probably a smart enough guy that if such a bounty were announced you would be able to drop those activities quickly and find new work or hobbies within a few months.
q2: I mean, you could. You can make a bounty to disincentivize any behavior. But who would have that kind of goal or support such a bounty, much less fund one? If you're worried about Goodhart's law here, just use a coarse enough metric like "gets paid to work on something AI-related" and accept there would be some false positives.
Metcalfe's (revised!) law states that the value of a communications network grows at about .
I frequently give my friends the advice that they should aim to become pretty good at 2 synergistic disciplines (CS and EE for me, for example), but I have wondered in the past why I don't give them the advice to become okay at 4 or 5 synergistic disciplines instead.
It just struck me these ideas might be connected in some way, but I am having trouble figuring out exactly how.
Try to think about this in terms of expected value. On your specific example, they do score more, but this is probabilistic thinking, so we want to think about it in terms of the long run trend.
Suppose we no longer know what the answer is, and you are genuinely 50/50 on it being either A or B. This is what you truly believe, you don't think there's a chance in hell it's C. If you sit there and ask yourself, "Maybe I should do a 50-25-25 split, just in case", you're going to immediately realize "Wait, that's moronic. I'm throwing away 25% of my points on something I am certain is wrong. This is like betting on a 3-legged horse."
Now let's say you do a hundred of these questions, and most of your 50-50s come up correct-ish as one or the other. Your opponent consistently does 50-25-25s, and so they end up more wrong than you overall, because half the time the answer lands on one of their two 25s, not their single 50.
It's not a game of being more correct, it's a game of being less wrong.
I disagree with your first point, I consider the 50:25:25:0 thing is the point. It's hard to swallow because admitting ignorance rather than appearing falsely confident always is, but that's why it makes for such a good value to train.
https://thingofthings.wordpress.com/2018/05/25/models-a-summary/
Agreed on the difference. Different subcultures, I think, all try to push different narratives about how they are significantly different from other subcultures; they are in competition with other subcultures for brain-space. On that observation, my priors that rationalist content is importantly different to other subcultures in that regard are low.
I suppose my real point in writing this is to advise against a sort of subcultural Fear Of Being Ordinary -- rationalism doesn't have to be qualitatively different from other subcultures to be valuable. For people under its umbrella, it can be very valuable, for reasons that have almost nothing to do with the quirks of the subculture itself.
This actually seems like a really, really good idea. Thanks!
Great post! Simple and useful. For spaced-repetition junkies in the crowd, I created a small Anki deck, created from this post to help me retain the basics.
You could normalize the scoring rule back to 1, so that should be fine.
Scattered thoughts on how the rationalist movement has helped me:
On the topic of rationalist self-improvement, I would like to raise the point that simply feeling as though there's a community of people who get me and that I can access when I want to has been hugely beneficial to my sense of happiness and belonging in the world.
That generates a lot of hedons for me, which then on occasion allow me to "afford" doing other things I wouldn't otherwise, like spend a little more time studying mathematics or running through Anki flashcards. There's a part of me that feels like I'm not just building up this knowledge for myself, but for the future possible good of "my people". I might tie together stuff in a way that other people find interesting, or insightful, or at least enjoy reading about, and that's honestly fricking awesome and blows standard delayed-gratification "self improvement" tactics outta the water 10/10 would recommend.
Also there's the whole thing that Ozy who is rat-almost-maybe-adjacent wrote the greatest summary of the greatest dating advice book I ever read, and I literally read that effortpost every day for like 8 months while I was learning how to be a half-decent romantic option, and holy SHIT is my life better for that. But again - nothing specific to the rationalist techniques themselves there; the value of the community was pointing me to someone who thinks and writes in a way my brain sees and says "mmm yes tasty good word soup i liek thanke" and then that person happened to write a post that played a big role in helping me with a problem that was causing me a ton of grief.
TLDR rationalists > rationalism
When I stop to think of people I support who I would peg as "extreme in words, moderate in actions", I think I feel a sense of overall safety that might be relevant here.
Let's say I'm in a fierce, conquering mood. I can put my weight behind their extremism, and feel powerful. I'm Making A Difference, going forth and reshaping the world a little closer to utopia.
When I'm in a defeatist mood, where nothing makes sense and I feel utterly hopeless, I can *also* get behind the extremism -- but it's in a different light, now. It's more, "I am so small, and the world is so big, but I can still live by what I feel is right".
Those are really emotionally powerful and salient times for me, and ones that have a profound effect on my sense of loyalty to certain causes. But most of the time, I'm puttering along and happy to be in the world of moderation. Intellectually, I understand that moderation is almost always going to be the best way forward; emotionally, it's another story entirely.
Upon first reading, I had the thought that a lot of people don't notice the extreme/moderate dichotomy of most of their leaders. I still think that's true. And then a lot of people do learn of that dichotomy, and they become disgusted by it, and turn away from anyone who falls in that camp. Which makes sense, honesty is a great virtue, why can't they just say what they mean? But then I look at myself, and while it doesn't feel *optimal* to me, it does feel like just another element of playing the game of power. There's this skill of reading between the lines that I think most people know is there, but they're a little reluctant to look straight at it.
Causality seems to be a property that we can infer in the Democritan atoms and how they interact with one another. But when you start reasoning with abstractions, rather than the interactions of the atoms directly, you lose information in the compression, which causes causality in the interactions of abstractions with another to be a harder thing to infer from watching them.
I don't yet have a stronger argument than that; this is a fairly new topic of interest to me.
I would picture them as rectangles and count. Like, 2x3 would look like
xxx
xxx
in my head, and for small numbers I could use the size of it to feel whether I was close. I remember doing really well with ratios and fractions and stuff for that reason.
For larger numbers, like 8x8, I would often subdivide into smaller squares (like 4x4 or 2x2), and count those. Then it would be easy to subdivide the larger one and repeat-add. I would often get a sour taste if the answer just "popped" into my head and I would actively fight against it, so I think there was a part of me that really just viscerally hated the idea of letting 'mere' memorization into my learning at all.
Incidentally, my past memories are saying that's why 6x7 and 7x7 gave me such trouble in particular; there was no "easy" way to decompose that in my head, it just looked like a square and another almost-maybe-a-square.
Agreed. I'm a big fan of spaced repetition systems now, even though I have a long way to go towards consistently using them.
For your specific situation, may I recommend curling up with Visual Complex Analysis for a few hours? 😊 http://pipad.org/tmp/Needham.visual-complex-analysis.pdf
On a more general note, I find that anyone who says they "learned it from first principles" is usually putting on airs. It's an odd intellectual purity norm that I think is unfortunately very common among the mathematically- and philosophically-minded.
As evolved chimpanzees, we are excellent at seeing a few examples of something and then understanding the more general abstractions that guide it on a gut level; we have an amazing ability to arrest form from thing, but our ability to go the other way around is a lot more limited.
I think most of your intellectual idols would agree that while eventually being able to build up "from first principles" is a great goal to shoot for, it's actually not the pedagogy you want. It's okay to start concrete and just practice and grind until the more abstract stuff becomes obvious!
Take it from a guy who leapt off the deep end this quarter into abstract algebra, real analysis, signal processing and probability theory at the same time -- there is no way I would be performing at the level I am in these classes if I didn't force my abstraction-loving ass down to ground level and actually just crank out problem sets until the abstractions finally started to make sense.
I always like seeing someone else on LessWrong who's as interested in the transformative potential of SRS as I am. 🙂
Sadly, I don't have any research to back up my claims. Just personal experiences, as an engineering student with a secondary love of computer science and fixing knowledge-gaps. Take this with several grains of salt -- it's not exactly wild, speculative theory, but it's not completely unfounded thinking either.
I'm going to focus on the specific case you mentioned because I'm not smart enough to generalize my thinking on this, yet.
First let's think about what design patterns are, and where they emerged from. As I understand it, design patterns emerged from decades of working computer programmers realizing that there existed understandable, replicable structures which prevented future problems in software engineering down the line. Most of this learning came out of previous projects in which they had been burned. In other words, they are artifacts of experience. They're the tricks of the trade that don't actually get taught in trade school, and are instead picked up during the apprenticeship (if you're lucky).
If I were designing an SRS deck for the purpose of being able to remember and recognize design patterns, then, I think I would build it on the following pillars:
1. " the name of the pattern on one side and the definition on the other", as you suggested. These cards aren't going to be terribly helpful right now, until I've gone through some of #2, but after a week or so of diligent review, their meanings will snap into place for me and I'll suddenly understand why they're so valuable.
2. " names and examples ", as you suggested. I am on the books as generally thinking there's a lot of value in generating concrete examples , to the point where I'd say the LessWrong community systemically underrates the value of ground-level knowledge. (We're a bunch of lazy nerds who want to solve the world-equation without leaving our bedrooms. I digress.)
3. motivations and reasons for those names and examples. Try taking them and setting up scenarios, and then asking "Why would design pattern X make sense/not make sense in this situation?" or "What design pattern do you think would work best in this situation?" You'll have to spend more than a couple seconds on these cards, but they will give you the appropriate training in critical thinking to be able to, later on, think through the problems in real life.
Hope some of this was food for thought. I might change this into a genuine post later on, since I'm on a writing kick the last couple days.
Aaaaaaaaaaaaaaaaaaaaand now I'm thinking I know what's wrong with me.
https://deponysum.com/2019/04/28/ocd-what-i-learned-fighting-mind-cancer/
Set up for Success: Insights from 'Naive Set Theory'
I very much doubt anyone else will care much about this post, so I will give my reasoning.
Please vote before you read my reasoning. :)
- This is the only post I've ever read that actually convinced me to do something with substantial effort, that is, actually read Naive Set Theory. I really, really wanted to practice kata on sets before I attempted a math minor and I still look back on that as the best 3 weeks of last summer.
- Reading NST the way I did taught me a lot about how not to read a math book. Don't try to memorize everything. Don't try to get every detail on the first pass. And definitely don't copy the book almost verbatim into a spaced repetition system ending up with over 8,000 cloze deletion cards which you then practice for 6 months. There is a really good reason why we learn math through proofs, problems, and puzzles.
- Also I like the tradition of having several smart people read the same classic book and give their slightly different spins on it. The information is mostly redundant, but my all-too-human memory is thankful for it.
To generalize this heuristic a bit, and to really highlight where its weaknesses lie: An ethical argument that you should make a significant change to your lifestyle should be backed up more strongly in proportion to that change.
For example, to most people, the GWWC 10% pledge is a mildly extraordinary claim. (My parents actually yelled at me when I donated my Christmas money at 17.) But I think it does meet our bar of evidence: 10% income is usually no great hardship if you plan for it, and the arguments that the various EAs put forward for it are often quite strong.
Where this heuristic breaks down is an exercise to the reader. :)
Thanks! :)
I think the approach is different for me, but maybe other person leverage gratitude as a way to fight their negative thoughts closer to the way you imply.
I'm very wary of this post for being so vague and not linking to an argument, but I'll throw my two cents in. :)
The future will not have a firm concept of individuals.
I see two ways to interpret this:
- You could see it as individuals being uploaded to some giant distributed AI - individual human minds coalescing into one big super-intelligence, or being replaced by one; or
- Having so many individuals that the entire idea of worrying about 1 person, when you have 100 billion people per planet per quadrant or whatever, becomes laughable.
The common thread is that "individuality" is slowly being supplanted by "information" - specifically that you, as an individual, only become so because of your unique inflows of information slowly carving out pathways in your mind, like how water randomly carves canyons over millions of years. In a giant AI, all the varying bits that make up one human from another would get crosslinked, in some immense database that would make Jorge Luis Borges blush; meanwhile, in a civilization of huge, huge populations, the value of those varying bits simply goes down, because it becomes increasingly unlikely that you'll actually be unique enough to matter on an individual level. So, the next bottleneck in the spread of civilization becomes resources.
This is probably my first comment on this site - feel free to browbeat me if I didn't get my point across well enough.
Hello from Boston. I've been reading LW since some point this summer. I like it a lot.
I'm an engineering student and willing to learn whatever it takes for me to tackle world problems like poverty, hunger and transmissible diseases. But for now I'm focusing my efforts on my degree.
I'm still a student, so I don't think I'd be able to take this sort of job. But consider my volunteer application sent. You guys are doing important work! -Andrew Quinn