Posts
Comments
I think it's great that you did this. It made the game more real, and hopefully the rest of us learned something.
I know that it is designed to guide decisions made in the real world. This does not force me to agree with the conclusions in all circumstances. Lots of models are not up to the task they are designed to deal with.
But I should have said "not in that game theory situation", becasue there is probably a way to construct some game theory game that applies here. That was my bad.
However, I stand by the claim that the full information game is too far from reality to be a good guide in this case. With stakes this high even small uncertainty becomes important.
What possible reason could Petrov or those in similar situations have had for not pushing the button? Maybe he believed that the US would retaliate and kill his family at home, and that deterred him. In other words, he believed his enemy would push the button.
Or maybe he just did not want to kill millions of people?
I should probably have said "we are not in that game theory situation".
(Though I do think that the real world is more complex that current game theory can handle. E.g. I don't think current game theory can fully handle unknown-unknown, but I could be wrong on this point)
The game of mutually assured destruction is very different even when just including known unknown.
But we are not in a game theory situation. We are in an imperfect world with imperfect information. There are malfunctioning warning systems and liars. And we are humans and not programs that get to read each others source code. There are no perfect commitments and if there where, there would be no way of verifying them.
So I think that the lesson is, that what ever your public stance, and whether or not you think that there are counterfactual situation where you should nuke. In practice, you should not nuke.
Do you see what I'm getting at?
From this we learn that you should not launch nukes, even if someone tells you to do it.
Applications are now closed. We'll be ~120 participants!
Me and the Helpers are finishing up the final preparations in Discord, and other places, so that we are ready to invite you in on Monday.
I've just sent out the last acceptance emails. If you have applied and not heard from me or anyone else on the organising team, then let me know asap, so we can find out what went wrong.
I may accept late applications in exchange for a bribe. (I'm actually serious about this. A *few* late applications is not a problem, and the bribe is so that you don't make a habit of it.)
Applications are now closed. We'll be ~120 participants!
Me and the Helpers are finishing up the final preparations in Discord, and other places, so that we are ready to invite you in on Monday.
I've just sent out the last acceptance emails. If you have applied and not heard from me or anyone else on the organising team, then let me know asap, so we can find out what went wrong.
I may accept late applications in exchange for a bribe. (I'm actually serious about this. A *few* late applications is not a problem, and the bribe is so that you don't make a habit of it.)
Good point. I also got this question elsewhere.
Sofia Gallego suggests:
If your country allows it, there are plenty of platforms (e.g. TransferWise) with which you can do international transfers easier and with lower fees than banks.
If that don't work, you can also send money to me on paypal, with a message what it is for, and I'll transfer it to LessWrong Deutschland for you. Bank transfer with in Europe is super easy.
he admits that she did not actually do any of the things she thinks she should have. But her brain persists in arguing that reality is wrong.
This is interesting. We use the word "should" both as to command ourselves and others. "You should eat vegetables", and to make predictions "This should work". Both types has a similar type of uncertainty, we do not know if the suggestion will be obeyed or if our prediction will be right.
I'm not sure how much one should read in to linguistic quirks like this.
That makes sense. Thanks for taking the time to answer.
Yes, that one.
Is this rule still in place?
Why do you have this rule? It seems to me like banning organizational announcement will make it much harder to get new initiatives of the ground.
This seems great. Would be ok to join one or two times to see if your group and method, and what ever topic you decide to zoom into, is a good fit for me?
I've started to collect various AI Safety initiative s here, so that we are aware of each other and hopefully can support each other. Let me know if you want to be listed there too.
Also people who are interested in joining elriggs group, might also be interested in the AI Safety discussion days, men and JJ are organising. Same topic, different format.
FLI have done a map of all AI Safety research (or all that they could find at the time). Would this be a useful recourse for you? I'm not linking it directly, becasue you might want to think for yourself first,before becoming to biased by others ideas. But it seems that at least it would be a useful tool at the literature review stage.
Good point. I don't think LessWrong Deutschland have paypal. But if you sent it to me, then I can forward the money.
My paypal is my normal email: linda.linsefors@gmail.com
A PhD is an opportunity to do focused, original research. People should only choose that path if that’s what they really want.
I completely agree. Doing a PhD for credentials is not a good strategy. Doing a PhD for money makes no sense what so ever.
The next discussion day is on July 18th.
More info here: https://docs.google.com/document/d/1Dlje81gtA5PynJKphGtYvKNOMkr5Hsyd59mh4fhqkL4/edit#
There is also the fact that there are much fewer academic post-doc jobs compared to PhD position. This is probably different in different fields, but my math friend says this is defiantly the case in math. Sure the more successful are more likely to get the next job, but it is more about relative success compared to your competition, than absolute success. I don't know if the bar to keep going happens to be reasonable in absolute terms.
The way I view a PhD is that it is an entry level research job. If you want to have a research career, you start with an entry level research job, more or less similar to other career path.
I wonder, if you want to do maths research, and don't do a PhD, what is the alternative? The best thing about a PhD is that you get paid to do research, which is very uncommon every where else, unless you do something very applied.
Do you know of any reasonable alternatives to working in academia for less applied research? Or maybe this is what you mean by gate-keeping, that academia has monopolised funding?
Some what related:
- This trailer for the documentary "Death (& Rebirth) Of A PhD" claims that getting a PhD used to be great, but is now crap.
- And here's an almost finished blogpost I'm working on: "Should you do a PhD?" Where I try to sort out some misconceptions I've seen, and give some very general advise.
However, neither of these exactly address the question of the post.
However again, I think it is probably more useful to ask the question: If I want to solve outstanding maths problems, is a PhD my best choice.
I've read this after I wrote my own reply. This seems like a reasonable hypothesis too. One thing a PhD supervisor is great for, is telling you what has already been done, and what papers you should read to learn more about some particular thing.
I don't think a PhD is necessary for ground breaking math. A more plausible explanation (or so I think) is that academia is a preferable work environment, compared to being by yourself. Even for an introvert, being part of academia will be more convenient. Therefore, everyone who want to do math research will try to find a job in academia, and everyone who is smart/competent enough to do groundbreaking research is also more than smart/competent enough to get a PhD.
I have to say that I also expected some of the work to be done by non-PhDs. But given the result I think that the correlation has at least as much to do with common cause, as with causality from PhD -> research.
On the other hand, it could be the other way around? Did you check if they got their PhD before or after that result. If you do a ground breaking research, you can just write it up as a thesis and get a PhD.
I'm leaving this comment so that I can find my way back here in the future.
So... apparently I underestimate the need to send out event reminders, but better late than never. Today is the 2:nd day (out of 4) of Web-TAISU, and it is not too late to join.
General information about the event:
https://docs.google.com/document/d/1AJ67N78A60njFmJPx3UN6ghwP3IRproPeh04DIvtgck/
Collaborative Schedule:
https://docs.google.com/document/d/1R07U0xwCohXZNwS9hc5xdV1Z_juy7WKbW572uxmC3u8/edit#
Let me know if you have any questions.
As mentioned, I did think of this of this model before, and I also disagree with Justin/Convergence on how to use it.
Lets say that the underlying space for the vector field is the state of the world. Should we really remove curl? I'd say no. It is completely valid to want to move along some particular path, even a circle, or more likely, a spiral.
Alternatively, lets say that the underlying space for the vector field is world histories. Now we should remove curl, becasue any circular preference in this space is inconsistent. But what even is the vector field in this picture?
***
My reason for considering values as a vector is becasue that is sort of how it feels to me on the inside. I have noticed that my own values are very different depending on my current mood and situation.
- When I'm sand/depressed, I become a selfish hedonist. All I care about is for me to be happy again.
- When I'm happy I have more complex and more altruistic values. I care about truth and the well-being of others.
It's like these wants are not tracking my global values at all, but just pointing out a direction in which I want to move. I doubt that I even have global values, because that would be very complicated, and also what would be the use of that? (Except when building a super intelligent AI, but that did not happen much in our ancestral environment.)
Ok, thanks. I have changed it ...
... I mean that is what what I wrote all along, can't you see? :P
Hm, I did not think about the tax part.
What country to you live in?
Maybe BERI would be willing to act as middle hand. They have non profit status in the US.
Would you be interested in just participating? I read your post about leaving AIS. Seems like you have enough experience to be able to contribute to the discussion.
Nice diagram.
I'm currently doing interviews with early career and aspiring AIS researchers to learn how to better support this group, since I know a lot of us are struggling. Even though you left, I think there are valuable information in your experience. You can answer here publicly or contact me via your preferred method.
(linda.linsefors@gmail.com, https://calendly.com/linda-linsefors/ais)
What could have been different about the world for you to succeed in getting a sustainable AI Safety research career?
What if you got more funding?
What if you got some sort of productivity coaching?
What if you had a collaboration partner?
-------------------------------------------------------------------------------------------------------------------------------
Random suggestion
Would you be interested in being a research sponsor. I'm guessing wildly here but maybe you can earn enough to live the fun life you want while also supporting a AI Safety researcher? Given that you been in the field, you have some capability to evaluate their work. You can give someone not just money but also a discussion partner and some amount of feedback.
If you can help someone else succeed, that creates as much good as doing the work yourself.
I just started doing these interviews with people, so I don't know for sure. But if my current model is more or less right, there will be lots of people who are in the situation you just left behind. And if I would make some wild guesses again, I would say that most of them will quit after a few year, like you, unless we can create better support.
This is just something that came to my mind late at night. I have not though long and hard about this idea. But maybe check if something like this feels right for you?
Let's do it!
If you pick a time and date and write up an abstract, then I will sort out the logistic. Worst case it's just you and me having a conversation, but most likely some more people will show up.
COVID-19 Update!
TAISU in it's planed form is cancelled. But there will be a Web-TAISU around the same theme, and around the same time. I will make an announcement and probably open up for more applications when this thing is a bit more planed out.
Hi Jarson.
Due to the current pandemic TAISU will take a very different form than originally planed. I will organize some sort of online event on the same theme around the same time, but I don't know much more yet. I don't want to take on board more participants until I know what I'm organising. But ass soon as I know a bit more, I will do a new announcement and open up applications again. I expect this will happen with in a week or two.
Regarding your project, I'd bee happy to take a look at your google dock. Pleas share it.
Official application deadline has now passed. Those of you who have applied to participate will soon get an email.
However, since TAISU is not full yet. I will now accept people on a first come first serve bases, for anyone who is qualified.
I'm basically ready to announce the next Technical AI Safety Unconference (TAISU). But I have hit a bit of decision paralysis as to what dates it should be.
If you are reasonably interested in attending, please help me by filling in this doodle
If you don't know what this is about, have a look at the information for the last one.
The venue will be EA Hotel in Blackpool UK again.
The way I understand your division of floors and sealing, the sealing is simply the highest level meta there is, and the agent has *typically* no way of questioning it. The ceiling is just "what the algorithm is programed to do". Alpha Go is had programed to update the network weights in a certain way in response to the training data.
What you call floor for Alpha Go, i.e. the move evaluations, are not even boundaries (in the sense nostalgebraist define it), that would just be the object level (no meta at all) policy.
I think this structure will be the same for any known agent algorithm, where by "known" I mean "we know how it works", rather than "we know that it exists". However Humans seems to be different? When I try to introspect it all seem to be mixed up, with object level heuristics influencing meta level updates. The ceiling and the floor are all mixed together. Or maybe not? Maybe we are just the same, i.e. having a definite top level, hard coded, highest level meta. Some evidence of this is that sometimes I just notice emotional shifts and/or decisions being made in my brain, and I just know that no normal reasoning I can do will have any effect on this shift/decision.
I agree that you can assign what ever belief you want (e.g. what ever is useful for the agents decision making proses) for for what happens in the counterfactual when omega is wrong, in decision problems where Omega is assumed to be a perfect predictor. However if you want to generalise to cases where Omega is an imperfect predictor (as you do mention), then I think you will (in general) have to put in the correct reward for Omega being wrong, becasue this is something that might actually be observed.
Weather this works or not is going to depend heavily on what looks like.
Given , i.e. , what does this say about ?
The answer depends on the amount of mutual information between , and . Unfortunately the the more generic is, (i.e. any function is possible) the less mutual information there will be. Therefore, unless we know some structure about , the restriction to is not going to do much. The agent will just find a very different policy that also actives very high in some very Goodharty way, but does not get penalized because low value for on is not correlated with low value on .
This could possibly be fixed by adding assumptions of the type for any that does too well on . That might yield something interesting, or it might just be a very complicated way of specifying as satisfiser, I don't know.
Mainly that we had two scheduling sessions, one on the morning of the first day an one on the morning of the third day. At each scheduling session, it was only possible to add activities for the upcoming two days.
At the start of unconference encouraged people to think of it as 2 day event and try to put in everything they really wanted to do the first two days. On the morning of day three, the schedule was cleared to let people add sessions about topic that where alive to them at that time.
The main reason for this design choice was to allow continued/deeper conversation. I if ideas where created during the first half, I wanted there to be space to keep talking about those ideas.
Also, some people only attended the last two days, and this set up guaranteed they would get a chance to add things to the schedule too. But that could also have been solved in other ways, so that was not a crux for my design choice.
I think UDT1.1 have two fundamentally wrong assumptions built in.
1) Complete prior: UDT1.1 follows the policy that is optimal according to it's prior. This is incommutable in general settings and will have to be approximated some how. But even an approximation of UDT1.1 assumes that UDT1.1 is at least well defined. However in some multi agent settings or when the agent is being fully simulated by the environment, or any other setting where the environment is necessary bigger than the agent, then UDT1.1 is ill defined.
2) Free will: In the problem Agent Simulates Predictor, the environment is smaller than the agent, so it is falls outside the above point. Here instead I think the problem is that the agent assumes that it has free will, when in fact it behaves in a deterministic manner.
The problem of free will in Decision Problems is even clearer in the smoking lesion problem:
You want to smoke and you don't want Cancer. You know that people who smoke are more likely get cancer, but you also know that smoking does not cause cancer. Instead, there is a common cause, some gene, that happens to both increase the risk of cancer and make it more likely that a person with this gene are more likely to choose to smoke. You can not test if you have the gene.
Say that you decide to smoke, becasue ether you have the gene or not so you might as well enjoy smoking. But what if everyone though like this? Then there would be no correlation between the cancer gene and smoking. So where did the statistics about smokers getting cancer come from (in this made up version of reality).
If you are the sort of person who smokes no mater what, then ether:
a) You are sufficiently different from most people such that the statistics does not apply to you.
or
b) The cancer gene is correlated with being the sort of person that has a decision possess that leads to smoking.
If b is correct, then maybe you should be the sort of algorithm that decides not to smoke, as to increase the chance of being implemented into a brain that lives in a body with less risk of cancer. But if you start thinking like that, then you are also giving up your hope at affecting the universe, and resign to just choosing where you might find yourself, and I don't think that is what we want from a decision theory.
But there also seems to be no good way of thinking about how to steer the universe with out pretending to have free will. But since that is actually a falls assumption, there will be weird edge cases where you're reasoning breaks down.
Do you agree with my clarification?
Because what you are trying to say makes very much sense to me, if and only if I replace "prediction" with "set point value" for cases when the so called prediction is fixed.
Set point (control system vocabulary) = Intention/goal (agent vocabulary)
It seems like people are talking in circles around each other in these comments, and I think the reason is that Gordon and other people who likes predictive processing theory are misusing the world "prediction"
By misuse I mean clearly deviating from common use. I don't really care about sticking to common use, but if you deviate from the expected meaning of a word it is good to let people know.
Lets say I have a model of the future in my head. If I try to adjust the model to fit reality this model is a prediction. If I try to fit reality to my model, this model is an intention.
If you have a control system that tries to minimise "prediction error" with respect to a "prediction" that it is not able to chance, so that the system resort to change reality instead, then that is not really a prediction anymore.
As I understand it predictive processing theory suggest that both updating predictions and executing intentions are optimising for the same thing, which is aligning reality with my internal model. However there is an important difference with is what is variables and what is constants in solving that problem. Gordon is mentioning at some places that sometimes "predictions" can't be updated.
This means that it won't always be the case that a control system is globally trying to minimize prediction error, but instead is locally trying to minimize prediction error, although it may not be able to become less wrong over time because it can't change the prediction to better predict the input.
There are probably some actual disagreement here (in this comment section) too, but we will not figure that out if we don't agree on what words mean first.
I have not read all the comments yet, so maybe this is redundant, but anyway...
I think it is plausible that humans and other life forms, are mostly made up of layers of control systems, stacked on each other. However it does not follow from this that humans are trying to minimise prediction error.
There are probably some part of the brain that is trying to minimise prediction error. Possibly organised as a control system that tries to keep expectations in line with reality. Because it is useful to be able to accurately predict the world.
But if we are a stack of control systems, then I would expect other parts of the brain to be control systems for other things. E.g. Having the correct level of blood sugar, having a good amount of social interaction, having a good amount of variety in our lives.
I can imagine someone figuring out more or less how the prediction control system works and what it is doing, then looking at everything else, noticing the similarity (becasue it is all types of control systems and evolution tend to reuse structures) and thinking "Hmm, maybe it is all about predictions". But I also think that would be wrong.
We are currently deciding between:
a) Running second Athena Workshop, similar to the first one, i.e. teaching a broad range of techniques for solving internal conflicts.
b) Running a workshop specifically focused on overcoming procrastination
c) Doing both
If you have any preferences, let me know.
Our current goal is to gather more information. Which method should we teach and how should we teach it? Is what we are teaching actual useful? To find this out we indent to:
1) Run various versions of the workshop
2) Experiment with various forms online teaching
3) Follow up with participants about what has been useful to them
We have also made a strategic decision to mostly learn from our own experiences, to hopefully find new local optimums for what and how to teach these types of things.
Because of the stage we are at, the quickest way for you to get more information about techniques would be to attend one of the workshop. We would like to eventually do something more scalable (e.g. realizing video lectures), but first we'll need to do a lot more testing.
For discussions between individuals about who is "more cognitively sophisticated", my current best guess is that you can actually have this conversation reasonably easily in private (where by "reasonably easily", I mean it maybe takes several hours of building trust and laying groundwork, but there's nothing mysterious about it)
I can confirm this (anecdotally).
The TAISU is now full. I might still accept exceptional applications. But don't expect to be accepted just becasue you meet the basic requirements.
I think that agency requires a membrane, something keeps particular people in and out, such that you have any deliberate culture, principles or decision making at all.
This TED talk Religion, evolution, and the ecstasy of self-transcendence, Jonathan Haidt talks about how having a membrane around a group is necessary for group selection to happen. This seems very related.
With out a membrane, the free rider problem can not be solved. If the free rider problem is not solved, then the group can not be fully aligned.
The Acceptance stuff was most useful for me. I don't remember any CFAR technique that focus on this.
There is still room for more participants at TAISU, but sleeping space is starting to fill up. The EA Hotel dorm rooms are almost fully booked. Fore those who don't fit in the dorm or want some more privet space, there are lots of near by hotel. However since TAISU happens to be on a UK bank holiday, these might fill up too.
This workshop is now full, but due to the enthusiasm I have received I am going to organize a second Learning-by-doing AI Safety workshop some time in October/November this year. If you want to influence when it will be you can fill in our doodle: https://doodle.com/poll/haxdy8iup4hes9xy
I am leaving the application form open. You can fill it in to show interest in the second Learning-by-doing AI Safety workshop and future similar events.
Fixed! Thank you for pointing this out.