Posts
Comments
Good point. I don't think LessWrong Deutschland have paypal. But if you sent it to me, then I can forward the money.
My paypal is my normal email: linda.linsefors@gmail.com
A PhD is an opportunity to do focused, original research. People should only choose that path if that’s what they really want.
I completely agree. Doing a PhD for credentials is not a good strategy. Doing a PhD for money makes no sense what so ever.
The next discussion day is on July 18th.
More info here: https://docs.google.com/document/d/1Dlje81gtA5PynJKphGtYvKNOMkr5Hsyd59mh4fhqkL4/edit#
There is also the fact that there are much fewer academic post-doc jobs compared to PhD position. This is probably different in different fields, but my math friend says this is defiantly the case in math. Sure the more successful are more likely to get the next job, but it is more about relative success compared to your competition, than absolute success. I don't know if the bar to keep going happens to be reasonable in absolute terms.
The way I view a PhD is that it is an entry level research job. If you want to have a research career, you start with an entry level research job, more or less similar to other career path.
I wonder, if you want to do maths research, and don't do a PhD, what is the alternative? The best thing about a PhD is that you get paid to do research, which is very uncommon every where else, unless you do something very applied.
Do you know of any reasonable alternatives to working in academia for less applied research? Or maybe this is what you mean by gate-keeping, that academia has monopolised funding?
Some what related:
- This trailer for the documentary "Death (& Rebirth) Of A PhD" claims that getting a PhD used to be great, but is now crap.
- And here's an almost finished blogpost I'm working on: "Should you do a PhD?" Where I try to sort out some misconceptions I've seen, and give some very general advise.
However, neither of these exactly address the question of the post.
However again, I think it is probably more useful to ask the question: If I want to solve outstanding maths problems, is a PhD my best choice.
I've read this after I wrote my own reply. This seems like a reasonable hypothesis too. One thing a PhD supervisor is great for, is telling you what has already been done, and what papers you should read to learn more about some particular thing.
I don't think a PhD is necessary for ground breaking math. A more plausible explanation (or so I think) is that academia is a preferable work environment, compared to being by yourself. Even for an introvert, being part of academia will be more convenient. Therefore, everyone who want to do math research will try to find a job in academia, and everyone who is smart/competent enough to do groundbreaking research is also more than smart/competent enough to get a PhD.
I have to say that I also expected some of the work to be done by non-PhDs. But given the result I think that the correlation has at least as much to do with common cause, as with causality from PhD -> research.
On the other hand, it could be the other way around? Did you check if they got their PhD before or after that result. If you do a ground breaking research, you can just write it up as a thesis and get a PhD.
I'm leaving this comment so that I can find my way back here in the future.
So... apparently I underestimate the need to send out event reminders, but better late than never. Today is the 2:nd day (out of 4) of Web-TAISU, and it is not too late to join.
General information about the event:
https://docs.google.com/document/d/1AJ67N78A60njFmJPx3UN6ghwP3IRproPeh04DIvtgck/
Collaborative Schedule:
https://docs.google.com/document/d/1R07U0xwCohXZNwS9hc5xdV1Z_juy7WKbW572uxmC3u8/edit#
Let me know if you have any questions.
As mentioned, I did think of this of this model before, and I also disagree with Justin/Convergence on how to use it.
Lets say that the underlying space for the vector field is the state of the world. Should we really remove curl? I'd say no. It is completely valid to want to move along some particular path, even a circle, or more likely, a spiral.
Alternatively, lets say that the underlying space for the vector field is world histories. Now we should remove curl, becasue any circular preference in this space is inconsistent. But what even is the vector field in this picture?
***
My reason for considering values as a vector is becasue that is sort of how it feels to me on the inside. I have noticed that my own values are very different depending on my current mood and situation.
- When I'm sand/depressed, I become a selfish hedonist. All I care about is for me to be happy again.
- When I'm happy I have more complex and more altruistic values. I care about truth and the well-being of others.
It's like these wants are not tracking my global values at all, but just pointing out a direction in which I want to move. I doubt that I even have global values, because that would be very complicated, and also what would be the use of that? (Except when building a super intelligent AI, but that did not happen much in our ancestral environment.)
Ok, thanks. I have changed it ...
... I mean that is what what I wrote all along, can't you see? :P
Hm, I did not think about the tax part.
What country to you live in?
Maybe BERI would be willing to act as middle hand. They have non profit status in the US.
Would you be interested in just participating? I read your post about leaving AIS. Seems like you have enough experience to be able to contribute to the discussion.
Nice diagram.
I'm currently doing interviews with early career and aspiring AIS researchers to learn how to better support this group, since I know a lot of us are struggling. Even though you left, I think there are valuable information in your experience. You can answer here publicly or contact me via your preferred method.
(linda.linsefors@gmail.com, https://calendly.com/linda-linsefors/ais)
What could have been different about the world for you to succeed in getting a sustainable AI Safety research career?
What if you got more funding?
What if you got some sort of productivity coaching?
What if you had a collaboration partner?
-------------------------------------------------------------------------------------------------------------------------------
Random suggestion
Would you be interested in being a research sponsor. I'm guessing wildly here but maybe you can earn enough to live the fun life you want while also supporting a AI Safety researcher? Given that you been in the field, you have some capability to evaluate their work. You can give someone not just money but also a discussion partner and some amount of feedback.
If you can help someone else succeed, that creates as much good as doing the work yourself.
I just started doing these interviews with people, so I don't know for sure. But if my current model is more or less right, there will be lots of people who are in the situation you just left behind. And if I would make some wild guesses again, I would say that most of them will quit after a few year, like you, unless we can create better support.
This is just something that came to my mind late at night. I have not though long and hard about this idea. But maybe check if something like this feels right for you?
Let's do it!
If you pick a time and date and write up an abstract, then I will sort out the logistic. Worst case it's just you and me having a conversation, but most likely some more people will show up.
COVID-19 Update!
TAISU in it's planed form is cancelled. But there will be a Web-TAISU around the same theme, and around the same time. I will make an announcement and probably open up for more applications when this thing is a bit more planed out.
Hi Jarson.
Due to the current pandemic TAISU will take a very different form than originally planed. I will organize some sort of online event on the same theme around the same time, but I don't know much more yet. I don't want to take on board more participants until I know what I'm organising. But ass soon as I know a bit more, I will do a new announcement and open up applications again. I expect this will happen with in a week or two.
Regarding your project, I'd bee happy to take a look at your google dock. Pleas share it.
Official application deadline has now passed. Those of you who have applied to participate will soon get an email.
However, since TAISU is not full yet. I will now accept people on a first come first serve bases, for anyone who is qualified.
I'm basically ready to announce the next Technical AI Safety Unconference (TAISU). But I have hit a bit of decision paralysis as to what dates it should be.
If you are reasonably interested in attending, please help me by filling in this doodle
If you don't know what this is about, have a look at the information for the last one.
The venue will be EA Hotel in Blackpool UK again.
The way I understand your division of floors and sealing, the sealing is simply the highest level meta there is, and the agent has *typically* no way of questioning it. The ceiling is just "what the algorithm is programed to do". Alpha Go is had programed to update the network weights in a certain way in response to the training data.
What you call floor for Alpha Go, i.e. the move evaluations, are not even boundaries (in the sense nostalgebraist define it), that would just be the object level (no meta at all) policy.
I think this structure will be the same for any known agent algorithm, where by "known" I mean "we know how it works", rather than "we know that it exists". However Humans seems to be different? When I try to introspect it all seem to be mixed up, with object level heuristics influencing meta level updates. The ceiling and the floor are all mixed together. Or maybe not? Maybe we are just the same, i.e. having a definite top level, hard coded, highest level meta. Some evidence of this is that sometimes I just notice emotional shifts and/or decisions being made in my brain, and I just know that no normal reasoning I can do will have any effect on this shift/decision.
I agree that you can assign what ever belief you want (e.g. what ever is useful for the agents decision making proses) for for what happens in the counterfactual when omega is wrong, in decision problems where Omega is assumed to be a perfect predictor. However if you want to generalise to cases where Omega is an imperfect predictor (as you do mention), then I think you will (in general) have to put in the correct reward for Omega being wrong, becasue this is something that might actually be observed.
Weather this works or not is going to depend heavily on what looks like.
Given , i.e. , what does this say about ?
The answer depends on the amount of mutual information between , and . Unfortunately the the more generic is, (i.e. any function is possible) the less mutual information there will be. Therefore, unless we know some structure about , the restriction to is not going to do much. The agent will just find a very different policy that also actives very high in some very Goodharty way, but does not get penalized because low value for on is not correlated with low value on .
This could possibly be fixed by adding assumptions of the type for any that does too well on . That might yield something interesting, or it might just be a very complicated way of specifying as satisfiser, I don't know.
Mainly that we had two scheduling sessions, one on the morning of the first day an one on the morning of the third day. At each scheduling session, it was only possible to add activities for the upcoming two days.
At the start of unconference encouraged people to think of it as 2 day event and try to put in everything they really wanted to do the first two days. On the morning of day three, the schedule was cleared to let people add sessions about topic that where alive to them at that time.
The main reason for this design choice was to allow continued/deeper conversation. I if ideas where created during the first half, I wanted there to be space to keep talking about those ideas.
Also, some people only attended the last two days, and this set up guaranteed they would get a chance to add things to the schedule too. But that could also have been solved in other ways, so that was not a crux for my design choice.
I think UDT1.1 have two fundamentally wrong assumptions built in.
1) Complete prior: UDT1.1 follows the policy that is optimal according to it's prior. This is incommutable in general settings and will have to be approximated some how. But even an approximation of UDT1.1 assumes that UDT1.1 is at least well defined. However in some multi agent settings or when the agent is being fully simulated by the environment, or any other setting where the environment is necessary bigger than the agent, then UDT1.1 is ill defined.
2) Free will: In the problem Agent Simulates Predictor, the environment is smaller than the agent, so it is falls outside the above point. Here instead I think the problem is that the agent assumes that it has free will, when in fact it behaves in a deterministic manner.
The problem of free will in Decision Problems is even clearer in the smoking lesion problem:
You want to smoke and you don't want Cancer. You know that people who smoke are more likely get cancer, but you also know that smoking does not cause cancer. Instead, there is a common cause, some gene, that happens to both increase the risk of cancer and make it more likely that a person with this gene are more likely to choose to smoke. You can not test if you have the gene.
Say that you decide to smoke, becasue ether you have the gene or not so you might as well enjoy smoking. But what if everyone though like this? Then there would be no correlation between the cancer gene and smoking. So where did the statistics about smokers getting cancer come from (in this made up version of reality).
If you are the sort of person who smokes no mater what, then ether:
a) You are sufficiently different from most people such that the statistics does not apply to you.
or
b) The cancer gene is correlated with being the sort of person that has a decision possess that leads to smoking.
If b is correct, then maybe you should be the sort of algorithm that decides not to smoke, as to increase the chance of being implemented into a brain that lives in a body with less risk of cancer. But if you start thinking like that, then you are also giving up your hope at affecting the universe, and resign to just choosing where you might find yourself, and I don't think that is what we want from a decision theory.
But there also seems to be no good way of thinking about how to steer the universe with out pretending to have free will. But since that is actually a falls assumption, there will be weird edge cases where you're reasoning breaks down.
Do you agree with my clarification?
Because what you are trying to say makes very much sense to me, if and only if I replace "prediction" with "set point value" for cases when the so called prediction is fixed.
Set point (control system vocabulary) = Intention/goal (agent vocabulary)
It seems like people are talking in circles around each other in these comments, and I think the reason is that Gordon and other people who likes predictive processing theory are misusing the world "prediction"
By misuse I mean clearly deviating from common use. I don't really care about sticking to common use, but if you deviate from the expected meaning of a word it is good to let people know.
Lets say I have a model of the future in my head. If I try to adjust the model to fit reality this model is a prediction. If I try to fit reality to my model, this model is an intention.
If you have a control system that tries to minimise "prediction error" with respect to a "prediction" that it is not able to chance, so that the system resort to change reality instead, then that is not really a prediction anymore.
As I understand it predictive processing theory suggest that both updating predictions and executing intentions are optimising for the same thing, which is aligning reality with my internal model. However there is an important difference with is what is variables and what is constants in solving that problem. Gordon is mentioning at some places that sometimes "predictions" can't be updated.
This means that it won't always be the case that a control system is globally trying to minimize prediction error, but instead is locally trying to minimize prediction error, although it may not be able to become less wrong over time because it can't change the prediction to better predict the input.
There are probably some actual disagreement here (in this comment section) too, but we will not figure that out if we don't agree on what words mean first.
I have not read all the comments yet, so maybe this is redundant, but anyway...
I think it is plausible that humans and other life forms, are mostly made up of layers of control systems, stacked on each other. However it does not follow from this that humans are trying to minimise prediction error.
There are probably some part of the brain that is trying to minimise prediction error. Possibly organised as a control system that tries to keep expectations in line with reality. Because it is useful to be able to accurately predict the world.
But if we are a stack of control systems, then I would expect other parts of the brain to be control systems for other things. E.g. Having the correct level of blood sugar, having a good amount of social interaction, having a good amount of variety in our lives.
I can imagine someone figuring out more or less how the prediction control system works and what it is doing, then looking at everything else, noticing the similarity (becasue it is all types of control systems and evolution tend to reuse structures) and thinking "Hmm, maybe it is all about predictions". But I also think that would be wrong.
We are currently deciding between:
a) Running second Athena Workshop, similar to the first one, i.e. teaching a broad range of techniques for solving internal conflicts.
b) Running a workshop specifically focused on overcoming procrastination
c) Doing both
If you have any preferences, let me know.
Our current goal is to gather more information. Which method should we teach and how should we teach it? Is what we are teaching actual useful? To find this out we indent to:
1) Run various versions of the workshop
2) Experiment with various forms online teaching
3) Follow up with participants about what has been useful to them
We have also made a strategic decision to mostly learn from our own experiences, to hopefully find new local optimums for what and how to teach these types of things.
Because of the stage we are at, the quickest way for you to get more information about techniques would be to attend one of the workshop. We would like to eventually do something more scalable (e.g. realizing video lectures), but first we'll need to do a lot more testing.
For discussions between individuals about who is "more cognitively sophisticated", my current best guess is that you can actually have this conversation reasonably easily in private (where by "reasonably easily", I mean it maybe takes several hours of building trust and laying groundwork, but there's nothing mysterious about it)
I can confirm this (anecdotally).
The TAISU is now full. I might still accept exceptional applications. But don't expect to be accepted just becasue you meet the basic requirements.
I think that agency requires a membrane, something keeps particular people in and out, such that you have any deliberate culture, principles or decision making at all.
This TED talk Religion, evolution, and the ecstasy of self-transcendence, Jonathan Haidt talks about how having a membrane around a group is necessary for group selection to happen. This seems very related.
With out a membrane, the free rider problem can not be solved. If the free rider problem is not solved, then the group can not be fully aligned.
The Acceptance stuff was most useful for me. I don't remember any CFAR technique that focus on this.
There is still room for more participants at TAISU, but sleeping space is starting to fill up. The EA Hotel dorm rooms are almost fully booked. Fore those who don't fit in the dorm or want some more privet space, there are lots of near by hotel. However since TAISU happens to be on a UK bank holiday, these might fill up too.
This workshop is now full, but due to the enthusiasm I have received I am going to organize a second Learning-by-doing AI Safety workshop some time in October/November this year. If you want to influence when it will be you can fill in our doodle: https://doodle.com/poll/haxdy8iup4hes9xy
I am leaving the application form open. You can fill it in to show interest in the second Learning-by-doing AI Safety workshop and future similar events.
Fixed! Thank you for pointing this out.
Accepted applicants so far (July 5)
Gavin Leech, University of Bristol (soon)
Michaël Trazzi, FHI
David Lindner, ETH Zürich
Gordon Worley, PAISRI
anonymous
Josh Jacobson, BERI
anonymous
Andrea Luppi, Harvard University / FHI
Dragan Mlakić
Noah Topper
Andrew Schreiber, Ought
Jan Brauner, University of Edinburgh - weekend only
Søren Elverlin, AISafety.com
Victoria Krakovna, DeepMind - weekend only
Janos Kramar, DeepMind - weekend only
Are you worried about the unconference not having enough participants (in total), or it not having enough senior participants?
commend removed by me
There i no specific deadline for signing up.
However, i might close the application at some point due to the unconference being full. We have more or less unlimited sleeping space since the EA Hotel is literally surrounded by other hotels. So the limitation is spaces for talks, discussions and workshops and such.
If all activities are in the EA Hotel, we should not be much more than 20 people. If it looks like I will get more applications than that I will see if it is possibly to rent some more commons spaces at other hotels I have not looked in to this yet, but I will soon.
We currently have 4 accepted applicants.
Good initiative. I will add a question to the application form, asking if the applicant allows me to share that they are coming. I then will share the participant list here (with the names of those how agreed) and update every few days.
For pledges, just write here as Ryan said.
I would decompose that in to a value trade + a blackmail.
The default for me would be to take the action that gives me 1 utility. But you can offer me a trade where you give me something better in return for me not taking that action. This would be a value trade.
Lets now take me agreeing to your proposition as the default. If I then choose to threaten to call the deal off, unless you pay me a even higher amount, than this is blackmail.
I don't think that these parts (the value trade and the blackmail) should be viewed as sequential. I wrote it that way for illustrative purposes. However, I do think that any value trade has a Game of Chicken component, where each player can threaten to call of the trade if they don't get the more favorable deal.
I did not mean to imply that the choices had to be made simultaneous, or in any other particular order, just that this is the type of payoff matrix. But I also think that "simultaneous choice" v.s. "sequential game" is a false dichotomy. If both players are UDT, every game is a game simultaneous choice game (where the choices are over complete policies).
I know that according to what I describe, the blackmailers threat is not credible in the game theory sense of the word. Sow what? It is still possible to make credible threats in the common-use meaning of the word, which is what matters.
Hi, approximately when will it be decided who gets funding this round?
I agree that "want" is not the correct word exactly. What I mean by prior is an agents actual a priori beliefs, so by definition there will be no mis-match there. I am not trying to say that you choose your prior exactly.
What I am gesturing at is that no prior is wrong, as long as it does not assign zero probability to the true outcome. And I think that much of the confusion in atrophic situation comes from trying to solve an under-constrained system.
I agree.
An even simpler example: If the agents are reward learners, both of them will optimize for their own reward signal, which are two different things in the physical world.
>it seems that in order to be worthwhile the person would most likely have to be co-located with the team
My conclusion was the opposite. For this to work well the bread winner should be in a high earning location (which typically high cost living) and the rest of the team should be in a low cost location (which typically have low earning potential).
Being the only one in the team that is i a separate lotion, is not optimal for inclusion. But many teams are spread out anyway. I am pretty sure RAISE is not all in one location. As an other example, the organizers of AI Safety Camp is spread out all over Europe.
>Also, if the organisation later receives funding, the amount of prestige/influence of those taking this role will seem to drop or they might even become completely obsolete.
This might actually be feature, not a bug. When the new organisation has grown up and are receiving all the grants they need, then it is time for the funder to move on, to the next project, brining with them knowledge and experience from the first project.
Basically, if I change the title, it can go on the front page?
Better?
Yes, that is correct.
I wrote the text and asked people to cosign if the agreed, for signaling value.
Do you have a good idea on how to make this clearer?