Posts
Comments
I think (not sure) Kelly Criterion applies to you only if you already are concave
I join the request. It's a good meme if and only if it is true.
can't we just look at weights?
As I understand, interpretability research doesn't exactly got stuck, but it's very-very-very far from something like this even for not-SotA models. And the gap is growing.
Which concept they might obtain by reading my book on Highly Advanced Epistemology 101 For Beginners, or maybe just my essay on Local Validity as a Key to Sanity and Civilization, I guess?"
Perhaps, there should be two links here?
Do you think that if someone filtered and steelmanned Quintin's criticism, it would be valuable? (No promises)
I think from Eliezer's point of view it goes kinda like this:
- People can't see why the arguments of other side are invalid.
- Eliezer tried to engage with them, but most listeners/readers can't tell who is right in this discussions.
- Eliezer thinks that if he provides people with strawmenned versions of other side's arguments and refutation of this strawmenned arguments, then the chance that this people will see why he's right in the real discussion will go up.
- Eliezer writes this discussion with strawmen as a fictional parable because otherwise it would be either dishonest and rude or a quite boring text with a lot of disclaimers. Or because it's just easier for him to write it this way.
After reading this text at least one person (you) thinks that the goal "avoid dishonesty and rudeness" were not achieved, so text is a failure.
After reading this text at least one person (me) thinks that 1. I got some useful ideas and models. 2. Of course, at least the smartest opponents of Eliezer have better arguments and I don't think Eliezer would disagree with that, so text is a success.
Ideally, Eliezer should update his strategy of writing texts based on both pieces of evidence.
I can be wrong, of course.
Are there any research about how can we change network structure or protocols to make it more difficult for rogue AI to create and run a distributed copies of itself?
What if just turn off the possibility to use the reaction by clicking it in the list of already used reactions? Yes, people would use them less, but more deliberately.
Lesswrong reactions system creates the same bias as normal reactions - it's much much easier to use the reaction someone already used. So the first person to use a reaction under a comment gets undue influence on what reactions there will be under that comment in the future.
I think I'm at least close to agreeing, but even if it's like this now, it doesn't mean that the complex-positive-value-optimizer can produce more value mass than simple-negative-value-optimizer.
On the definition question, in addition to what localdeity wrote:
- I assume that on the first axis we consider position "interested in two people" already pretty non-monogamous, while the position "my partner can have sexual or romantic relationships with anyone except one particular person" is still very poly. If that's the case, your position along "interested in one"/"interested in many" axis can be easily changed if the set of people you know changes, even slightly. This position isn't a fact about you, more like fact about you and your options in current environment. In contrast, your position along the "restricts partner/doesn't restrict partner" axis can't change much if your partner's environment slightly changes. So if we want the definition of something stable and identity-related using one axis, the second axis is better suited for this purpose.
- On practice (your experience may vary) the definition that uses axis "interested in one"/"interested in many" causes kinda missing-the-point arguments from monogamous people, like "I barely have time for one partner". I think, if the restriction based definition were generally accepted, the discourse would be better.
This totally makes sense! But "proteins are held together by van der Waals forces that are much weaker than covalent bonds" still is a bad communication.
As for 4 - even just remembering anything is a self modification of memory.
That's for humans, not abstract agents? Don't think it matters, we talk about other self-modifications anyway.
From your problem description
Not mine :)
utility on other branches
Maybe this interpretation is what repels you? Here's another 2:
- You choose to behave like EDT-agent or like FDT-agent in the situations where it matters in advance, before you got into (1) or (3). And you can't legibly for the predictors like one in this game decide to behave like FDT agent, and then, in the future, when you got into (1) because you're unlucky, just change your mind. It's just not an option. And between options "legibly choose to behave like EDT-agent" and "legibly choose to behave like FDT-agent" the second one is clearly better in expectation. You just not make another choice in (1) or (2), it's already decided.
- If you find yourself in (1) or (2) you can't differentiate between cases "I am real me" and "I am the model of myself inside predictor" (because if you could, you could behave differently in this two cases and it would be bad model and bad predictor). So you decide for both at once. (this interpretation doesn't work well for afents with explicitly self-indicated values (or how it is called? I hope it's clear what I mean.))
The earlier decision to precommit (whether actually made or later simulated/hallucinated) sacrifices utility of some future selves in exchange for greater utility to other future selves.
Yes. It's like choose to win on a 1-5 on a die roll rather then win on a 6. You sacrifice utility if some future selves (in the worlds, when die roll 6) in exchange for greater utility to other future selves, and it's perfectly rational.
We can also construct more specific variants of 5 where FDT loses - such as environments where the message at step B is from an anti-Omega which punishes FDT like agents.
Ok, yes. You can do it with all other types of agents too.
But naturally a powerful EDT agent will simply adopt that universal precommitment if when it believes it is in a universe distribution where doing so is optimal!
I think the ability to legibly adopt such precommitment and willingness to do so kinda turns EDT-agent into FDT-agent.
Well, yes, it loses in (1), but it's fine, because it wins in (4) and (5) and is on par with EDT-agent in (3). (1) is not the full situation in this game, it's always a consequence of (3), (4) or (5), depending on interpretation, the rules don't make sense otherwise.
PS. If FDT-agent is suddenly teleported into situation (1) in place of some other agent by some powerful entity who can deceive the predictor and the predictor predicted the behaviour of this other agent who was in the game before, and FDT-agent knows all this, it obviously takes $1, why not?
FDT-agent obviously never decide "I will never ever take $1 from the box". It decides "I will not take $1 in the box if the rules of the situation I'm in are like <rules of this game>".
Only it's more general, something like "When I realise that it would be better if I made some precommitment earlier, I act like I would act if I actually made it" (not sure that this phrasing is fully correct in all cases).
EDT-agent in (5) goes in (1) with 99% probability and in scenario (2) with 1% probability. It wins 99%*$1+1%*$100=$1.99 in expectation.
FDT-agent in (5) goes in (1) with 1% probability and in scenario (2) with 99% probability. It wins 1%*$0+99%*$100=$99 in expectation.
IMO, to say that FDT-agent loses in (1) and therefore it is inferior to EDT-agent is like say that it's better to choose to roll a die with win on 6 then to roll a die with win on 1-5 because this option is better in the case where a die rolls 6.
In what exact set of alternate rules EDT-agent wins more in expectation?
FDT outperforms EDT on
4. You are about to observe one of [$1, $100] in a transparent box, but your action set doesn't include any self-modifications or precomittments.
5. You are about to observe one of [$1, $100] in a transparent box, but you don't know about it and will know about the rules of this game only when you will already see the box.
If the best the EDT-agent can do is precommit to behave like FDT-agent or self-modify itself into FDT-agent, it's weird to say that EDT is better :)
Parfit's Hitchhiker? Smoking Lesion?
Do outcomes "Russia/France surrenders, France/Russia stays neutral or whitepeaces" count?
Not the full solution, but some random thoughts:
- Get in touch with some famous writers, tell them everything, let them write.
- Tell the leaders that when you bomb cities, the morale of enemy population goes up, not down. (IIRC there were hopes that if you bomb cities enough, people in them will demand that their state surrender to end the war)
- Does game theory count as "post-1914 technology"?
- Oh, and you need to stabilise and empower Russia somehow, so that the idea "attack Russia and just defend youself on all other fronts" does not look promising for Central Powers.
I'm confused, like I always confused with hardline libertarianism. Why would companies agree to this? Who would put capabilities researchers in jail if they, like, "I'd rather not purchase apocalypse insurance and create AI anyway"? Why this actor is not a state by other name? What should I read to became less confused?
It's hard to guess, but it happened when the only one known to us general intelligence was created by a hill-climbing process.
"Polyamory-as-a-default-option" would be a better social standard than "Monogamy-as-a-default-option".
"Open-source LLM-based agent with hacking abilities starts spreading itself over the Internet because some user asked it to do so or to do something like to conquer the world" is a quite probable point-of-no-return regarding AGI risk.
Pretty sure it's just false.
First found example: the last post by EY
And then I thought to myself, "This LK99 issue seems complicated enough that it'd be worth doing an actual Bayesian calculation on it"--a rare thought; I don't think I've done an actual explicit numerical Bayesian update in at least a year.
In the process of trying to set up an explicit calculation, I realized I felt very unsure about some critically important quantities, to the point where it no longer seemed worth trying to do the calculation with numbers. This is the System Working As Intended.
In our world my laptop doesn't fall because there is a table under it. In another world Flying Spaghetty Monster holds my laptop. And also FSM sends light in my (version of me from other world) eyes, so I think there is a table. And FSM copies all other causal effects which are caused by the table in our world. This other world is imaginable, therefore, the table is non-physical. What exactly makes this a bad analogy with your line of thought?
Epistemic status: I'm not sure if this assumtions are really here, am pretty sure what my current opinion about this assumptions is, but admit that this opinion can change.
Assumptions I don't buy:
- Have kids when we can have AGI in 10-25 years is good and not, actually, very evil OMG what are you doing.
- Right social incentives can't make A LOT of people poly pretty fast.
I think for all definitions of "curiosity" that make sense (that aren't like "we just use this word to refer to something completely unrelated to what people usually understand by it") maximally curious AI kills us, so it doesn't matter how curiosity is defined in RL literature.
I think last sentence kinda misses the point, but in general I agree. Why all this downvotes?
Humanity maybe is more "interesting" than nothing, but is it more "interesting" than anything that isn't humanity and can be assembled from the same matter? Definitely not!
It depends.
If croupier choose the players, then players learn that they were chosen, then croupier roll the dice, then players either get bajillion dollars each or die, then (if not snake eyes) croupier choose next players and so on - answer is 1/36.
If croupier choose the players, then roll the dice, then (if not snake eyes) croupier choose next players and so on, and only when dice come up snake eyes players learn that they were chosen, and then last players die and all other players get bajillion dollars each - answer is about 1/2.
Interesting!
A Couple of questions:
- You have a clause about China and India, but not about Russia. So, Russia is OK? (Among other things, in Russia, it is difficult to receive money from abroad: many banks are disconnected from SWIFT, some of the rest have stopped working with transactions from the US on their own initiative, and there is a chance that a Western bank will refuse to conduct a transaction to Russia. So the most reliable way is to have a trusted intermediary person with money in a bank account in Russia and a second bank account somewhere else.)
- I already applied to LTFF and now wait for an answer with not very high hopes for success. Does it make sense to apply to Lightspeed and, if I receive a positive response from one fund, just cancel the application to other one?
What does "I saw this" mean? "I already saw this in another place" or "I saw this comment, if it's important"? I think it needs clarification.
BTW, Done
I know this is the classic, but I just came up with a more elegant variation, without another world.
Toxoplasma infection makes you more likely to pet a cat. You like petting cats, but you are very afraid of getting toxoplasmosis. You don't know if you are infected, but you know this particular cat is healthy, so you can't become infected by petting it. Should you pet this cat?
I'm Russian and I think, when I will translate this, I will change "Russian" to "[other country's]". Will feel safer that way.
(about "hostile")
https://ui.stampy.ai?state=6982_
https://ui.stampy.ai?state=897I_
And suddenly it seems stampy has no answer for "Why inner misalignment is the default outcome". But EY said a lot about it, it's easy to find.
2. Depends of what you mean by "claims foom". As I understand, now EY thinks that foom isn't neccesary anyway, AGI can kill us before it.
4. "I doesn't like it" != "irrational and stupid and short sighted", you need arguments for why it isn't preferable in terms of the values of this systems
6, 7. "be ready to enforce a treaty" != "choose actions to kill billions of people living now".
Peregrin/Periklynian/Suvinian Dialog!
(Seriously, some explicit distinction between "dialogue as collaboration", "dialogue as debate" and "dialogue as explanation" would be nice. Not necessary at all, but nice.)
>A neural network is trained to optimize a loss function on input
No. Base optimizer optimize a loss function on inputs through changes in neural network. If neural network itself start to optimize something it can easily be something in the outside world.
Neural network : loss :: humans : human values
Neural network : loss :: humans : inclusive genetic fitness
(Am I using this notation correctly?)
It is not clear if this happened on its own, or if they deliberately trained the model not to make such mistakes.
Perhaps, in similar future studies, it is worth keeping half of the found tasks in secret in order to test future models with them.
Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.
Wow, that's good, right?
Also 100% me. Yes, it would be in demand!
I think the perfect balance of power is very unlikely, so in practice only the most powerful (most likely the first created) AGI will matter.
I don't think that measurements of the concept of "coherence" which implies that an ant is more coherent than AlphaGo is valuable in this context.
However, I think that pointing out the assumption about the relationship between intelligence and coherence is.
I always thought "shoggoth" and "pile of masks" are the same thing and "shoggoth with a mask" is just when one mask has become the default one and an inexperienced observer might think that the whole entity is this mask.
Maybe you are preaching to the chore here.