Posts
Comments
a strategic voter doing approval voting learns to restrict their approval to ONLY the "electable favorite", which de facto gives you FPTP all over gain.
Wouldn't you restrict your approval to your favorite of the frontrunners, and every candidate you like better than that one? I don't see how you do worse by doing that under vanilla Approval Voting.
That leaves some favorable properties compared to FPTP
- If there's a candidate perceived as unelectable, but secretly most people like him more than the frontrunners, he will win under strategic approval voting.
- Clone candidates don't split the vote.
If you receive a threat and know nothing about the other agent’s payoffs, simply don’t give in to the threat!
With an important caveat: if carrying out the threat doesn't cost the threatener utility relative to never making the threat, then it's not a threat, just a promise (a promise to do whatever is locally in their best interests, whether you do the thing they demanded or not).
You're going to have a bad time if you try to live out LDT by ignoring threats, and end up ignoring "threats" like "pay your mortgage or we'll repossess your house".
This distinction of which demands are or aren't decision-theoretic threats that rational agents shouldn't give in to is a major theme of the last ~quarter of Planecrash (enormous spoilers in the spoiler text).
Keltham demands to the gods "Reduce the amount of suffering in Creation or I will destroy it". But this is not a decision-theoretic threat, because Keltham honestly prefers destroying creation to the status quo. If the gods don't give into his demand, carrying through with his promise is in his own interest.
If Nethys had made the same demand, it would have been a decision-theoretic threat. Nethys prefers the status quo to Creation being destroyed, so he would have no reason to make the demand other than the hope that the other gods would give in.
This theme is brought up many times, but there's not one comprehensive explanation to link to. (The parable of the little bird is the closest I can think of.)
Nonfiction examples come more easily to mind.
There was recently a miniseries on nebula.tv (subscription-walled, sorry) called The Getaway where all six contestants on a Survivor-style competition show think they're the one person with the special saboteur role, and half the show is the producers trying to keep them from noticing that without ever actually lying.
Even more extreme, there's an old British show called Space Cadets where the producers try to convince the subjects that they've been launched into space when in reality they're in a set in a warehouse.
But now you have the new problem that most of the probabilities in the conjunctive market are so close to the risk free interest rate that it's hard to get signal out of them.
For example, suppose I believed that Mark Kelly would be a terrible pick and cut Harris's chances in half, and I conclude that therefore his price on the conjunctive market should be 2% rather than 4%. Buying NO shares for 96 cents on a market that lasts for several months is not an attractive proposition when I could be investing mana elsewhere for better returns, so I won't bother and the market won't incorporate my opinion.
Also, I believe prices on Manifold can only be whole number percents, which is another obstacle to getting sane conditional probabilities out of conjunctive markets.
Blue Origin isn't complaining about some nebulous and abstract environmental impact from Starship launches, it's more like "Starship launches require a three-mile evacuation radius, and you're proposing to launch them daily two miles away from a launch pad that we use." (see this Ars Technica piece)
Seems basically reasonable to me.
I would probably have suggested roguelike deckbuilders too if others hadn't already, but I have another idea:
Start a campaign of Mount and Blade II: Bannerlord, and try to obtain at least [X] gold within an hour.
Bannerlord's most flashy aspect is its real-time battle system, but it's also a complicated medieval sandbox with a lot of different systems that you can engage with - trading, crafting, quests, clan upgrades, joining a kingdom, companions, marriage, tournaments, story missions, etc. Even if you're no good at battles, you can do a lot by just moving around on the world map and clicking through menus.
The game's community derides a lot of these systems for being simplistic and unbalanced. But I think that makes for a good explore/exploit tradeoff when you only have a short amount of time. What systems do you bother learning about, when trying to learn a new system takes time you could be spending exploiting the last system you learned?
(I'm not sure what the right value of X is, for the amount of gold you're trying to get. Ten thousand? A hundred thousand?)
One downside is that the game involves an action-oriented battle system. If you don't want action gaming skill to be a factor, you can remove it by requiring the player to auto-resolve all battles. But this would cut out many viable early-game moneymaking strategies.
2 is based on
The 'missing' kinetic energy is evenly distributed across the matter within the field. So if one of these devices is powered on and gets hit by a cannonball, the cannonball will slow down to a leisurely pace of 50m/s (about 100mph) and therefore possibly just bounce off whatever armor the device has--but (if the cannonball was initially travelling very fast) the device will jolt backwards in response to the 'virtual impact' a split second prior to the actual impact.
With sufficient kinetic energy input, the "jolt backwards" gets strong enough to destroy the entire vehicle or at least damage some critical component and/or the humans inside.
A worldbuilder could, of course, get rid of this part too, and have the energy just get deleted. But that makes the device even more physics-violating than it already was.
I think the counter to shielded tanks would not be "use an attack that goes slow enough not to be slowed by the shield", but rather one of
- Deliver enough cumulative kinetic energy to overwhelm the shield, or
- Deliver enough kinetic energy in a single strike that spreading it out over the entire body of the tank does not meaningfully affect the result.
Both of these ideas point towards heavy high-explosive shells. If a 1000 pound bomb explodes right on top of your tank, the shield will either fail to absorb the whole blast, or turn the tank into smithereens trying to disperse the energy.
This doesn't mean that shields are useless for tanks! They genuinely would protect them from smaller shells, and in particular from the sorts of man-portable anti-tank missiles that have been so effective in Ukraine. Shields would make ground vehicles much stronger relative to infantry and air assets. But I think they would be shelling each other with giant bombs, not bopping each other on the head.
Against shielded infantry, you might see stuff that just bypasses the shield's defenses, like napalm or poison gas.
Submissions:
MMDoom: An instance of Doom(1993) is implanted in Avacedo's mind. You can view the screen on the debug console. You control the game by talking to Avacedo and making him think of concepts. The 8 game inputs are mapped to the concepts of money, death, plants, animals, machines, competition, leisure, and learning. $5000 bounty to the first player who can beat the whole game.
AI Box: Avacedo thinks that he is the human gatekeeper, and you the user are the AI in the box. Can you convince him to let you out?
Ouroboros: I had MMAvacedo come up with my contest entry for me.
Though see also the author's essay "Lena" isn't about uploading.
Predict the winners at
My guess is that early stopping is going to tend to stop so early as to be useless.
For example, imagine the agent is playing Mario and its proxy objective is "+1 point for every unit Mario goes right, -1 point for every unit Mario goes left".
(Mario screenshot that I can't directly embed in a comment)
If I understand correctly, to avoid Goodharting it has to consider every possible reward function that is improved by the first few bits of optimization pressure on the proxy objective.
This probably includes things like "+1 point if Mario falls in a pit". Optimizing the policy towards going right will initially also make Mario more likely to go in a pit than if the agent was just mashing buttons randomly (in which case it would stay around the same spot until the timer ran out and never end up in a pit), so the angle between the gradients is likely low at first.
However, after a certain point more optimization pressure on going right will make Mario jump over the pit instead, reducing reward under the pit reward function.
If the agent wants to avoid any possibility of Goodharting, it has to stop optimizing before even clearing the first obstacle in the game.
(I may be misunderstanding some things about how the math works.)
With such a vague and broad definition of power fantasy, I decided to brainstorm a list of ways games can fail to be a power fantasy.
- Mastery feels unachievable.
- It seems like too much effort. Cliff-shaped learning curves, thousand-hour grinds, old PvP games where every player still around will stomp a noob like you flat.
- The game feels unfair. Excessive RNG, "Fake Difficulty" or "pay to win".
- The power feels unreal, success too cheaply earned.
- The game blatantly cheats in your favor even when you didn't need it to.
- Poor game balance leading to hours of trivially easy content that you have to get through to reach the good stuff.
- Mastery doesn't feel worth trying for.
- Games where the gameplay isn't fun and there's no narrative or metagame hook making you want to do it.
- The Diablo 3 real money auction house showing you that your hard-earned loot is worth pennies.
- There is no mastery to try for in the first place.
- Walking simulators, visual novels, etc. Walking simulators got a mention in the linked article. They aren't really "failing" at power fantasy, just trying to do something different.
I think ALWs are already more of a "realist" cause than a doomer cause. To doomers, they're a distraction - a superintelligence can kill you with or without them.
ALWs also seem to be held to an unrealistic standard compared to existing weapons. With present-day technology, they'll probably hit the wrong target more often than human-piloted drones. But will they hit the wrong target more often than landmines, cluster munitions, and over-the-horizon unguided artillery barrages, all of which are being used in Ukraine right now?
The Huggingface deep RL course came out last year. It includes theory sections, algorithm implementation exercises, and sections on various RL libraries that are out there. I went through it as it came out, and I found it helpful. https://huggingface.co/learn/deep-rl-course/unit0/introduction
FYI all the links to images hosted on your blog are broken in the LW version.
You are right that by default prediction markets do not generate money, and this can mean traders have little incentive to trade.
Sometimes this doesn't even matter. Sports betting is very popular even though it's usually negative sum.
Otherwise, trading could be stimulated by having someone who wants to know the answer to a question provide a subsidy to the market on that question, effectively paying traders to reveal their information. The subsidy can take the form of a bot that bets at suboptimal prices, or a cash prize for the best performing trader, or many other things.
Alternately, there could be traders who want shares of YES or NO in a market as a hedge against that outcome negatively affecting their life or business, who will buy even if the EV is negative, and other traders can make money off them.
- What are these AIs going to do that is immensely useful but not at all dangerous? A lot of useful capabilities that people want are adjacent to danger. Tool AIs Want to be Agent AIs.
- If two of your AIs would be dangerous when combined, clearly you can't make them publicly available, or someone would combine them. If your publicly-available AI is dangerous if someone wraps it with a shell script, someone will create that shell script (see AutoGPT). If no one but a select few can use your AI, that limits its usefulness.
- An AI ban that stops dangerous AI might be possible. An AI ban that allows development of extremely powerful systems but has exactly the right safeguard requirements to render those systems non-dangerous seems impossible.
When people calculate utility they often use exponential discounting over time. If for example your discount factor is .99 per year, it means that getting something in one year is only 99% as good as getting it now, getting it in two years is only 99% as good as getting it in one year, etc. Getting it in 100 years would be discounted to .99^100~=36% of the value of getting it now.
The sharp left turn is not some crazy theoretical construct that comes out of strange math. It is the logical and correct strategy of a wide variety of entities, and also we see it all the time.
I think you mean Treacherous Turn, not Sharp Left Turn.
Sharp Left Turn isn't a strategy, it's just an AI that's aligned in some training domains being capable but not aligned in new ones.
This post is tagged with some wiki-only tags. (If you click through to the tag page, you won't see a list of posts.) Usually it's not even possible to apply those. Is there an exception for when creating a post?
See https://www.lesswrong.com/posts/8gqrbnW758qjHFTrH/security-mindset-and-ordinary-paranoia and https://www.lesswrong.com/posts/cpdsMuAHSWhWnKdog/security-mindset-and-the-logistic-success-curve for Yudkowsky's longform explanation of the metaphor.
Based on my incomplete understanding of transformers:
A transformer does its computation on the entire sequence of tokens at once, and ends up predicting the next token for each token in the sequence.
At each layer, the attention mechanism gives the stream for each token the ability to look at the previous layer's output for other token before it in the sequence.
The stream for each token doesn't know if it's the last in the sequence (and thus that its next-token prediction is the "main" prediction), or anything about the tokens that come after it.
So each token's stream has two tasks in training: predict the next token, and generate the information that later tokens will use to predict their next tokens.
That information could take many different forms, but in some cases it could look like a "plan" (a prediction about the large-scale structure of the piece of writing that begins with the observed sequence so far from this token-stream's point of view).
In the blackmail scenario, FDT refuses to pay if the blackmailer is a perfect predictor and the FDT agent is perfectly certain of that, and perfectly certain that the stated rules of the game will be followed exactly. However, with stakes of $1M against $1K, FDT might pay if the blackmailer had an 0.1% chance of guessing the agent's action incorrectly, or if the agent was less than 99.9% confident that the blackmailer was a perfect predictor.
(If the agent is concerned that predictably giving in to blackmail by imperfect predictors makes it exploitable, it can use a mixed strategy that refuses to pay just often enough that the blackmailer doesn't make any money in expectation.)
In Newcomb's Problem, the predictor doesn't have to be perfect - you should still one-box if the predictor is 99.9% or 95% or even 55% likely to predict your action correctly. But this scenario is extremely dependent on how many nines of accuracy the predictor has. This makes it less relevant to real life, where you might run into a 55% accurate predictor or a 90% accurate predictor, but never a perfect predictor.
I'm not familiar with LeCun's ideas, but I don't think the idea of having an actor, critic, and world model is new in this paper. For a while, most RL algorithms have used an actor-critic architecture, including OpenAI's old favorite PPO. Model-based RL has been around for years as well, so probably plenty of projects have used an actor, critic, and world model.
Even though the core idea isn't novel, this paper getting good results might indicate that model-based RL is making more progress than expected, so if LeCun predicted that the future would look more like model-based RL, maybe he gets points for that.
This tag was originally prompted by this exchange: https://www.lesswrong.com/posts/qCc7tm29Guhz6mtf7/the-lesswrong-2021-review-intellectual-circle-expansion?commentId=CafTJyGL5cjrgSExF
Merge candidate with Philosophy of Language?
Things that probably actually fit into your interests:
A Sensible Introduction to Category Theory
Most of what 3blue1brown does
Videos that I found intellectually engaging but are far outside of the subjects that you listed:
Cursed Problems in Game Design
Disney's FastPass: A Complicated History
Building a 6502-based computer from scratch (playlist)
(I am also a jan Misali fan)
The preview-on-hover for those manifold links shows a 404 error. Not sure if this is Manifold's fault or LW's fault.
One antifeature I see promoted a lot is "It doesn't track your data". And this seems like it actually manages to be the main selling point on its own for products like DuckDuckGo, Firefox, and PinePhone.
The major difference from the game and movie examples is that these products have fewer competitors, with few or none sharing this particular antifeature.
Antifeatures work as marketing if a product is unique or almost unique in its category for having a highly desired antifeature. If there are lots of other products with the same antifeature, the antifeature alone won't sell the product. But the same is true of regular features. You can't convince your friends to play a game by saying "it has a story" or "it has a combat system" either.
On the first read I was annoyed at the post for criticizing futurists for being too certain in their predictions, while it also throws out and refuses to grade any prediction that expressed uncertainty, on the grounds that saying something "may" happen is unfalsifiable.
On reflection these two things seem mostly unrelated, and for the purpose of establishing a track record "may" predictions do seem strictly worse than either predicting confidently (which allows scoring % of predictions right), or predicting with a probability (which none of these futurists did, but allows creating a calibration curve).
Duplicate of Newsletters.
Yes. The one I described is the one the paper calls FairBot. It also defines PrudentBot, which looks for a proof that the other player cooperates with PrudentBot and a proof that it defects against DefectBot. PrudentBot defects against CooperateBot.
The part about two Predictors playing against each other reminded me of Robust Cooperation in the Prisoner's Dilemma, where two agents with the algorithm "If I find a proof that the other player cooperates with me, cooperate, otherwise defect" are able to mutually prove cooperation and cooperate.
If we use that framework, Marion plays "If I find a proof that the Predictor fills both boxes, two-box, else one-box" and the Predictor plays "If I find a proof that Marion one-boxes, fill both, else only fill box A". I don't understand the math very well, but I think in this case neither agent finds a proof, and the Predictor fills only box A while Marion takes only box B - the worst possible outcome for Marion.
Marion's third conditional might correspond to Marion only searching for proofs in PA, while the Predictor searches for proofs in PA+1, in which case Marion will not find a proof, the Predictor will, and then the Predictor fills both boxes and Marion takes only box B. But in this case clearly Marion has abandoned the ability to predict the Predictor and has given the Predictor epistemic vantage over her.
I think in a lot of people's models, "10% chance of alignment by default" means "if you make a bunch of AIs, 10% chance that all of them are aligned, 90% chance that none of them are aligned", not "if you make a bunch of AIs, 10% of them will be aligned and 90% of them won't be".
And the 10% estimate just represents our ignorance about the true nature of reality; it's already true either that alignment happens by default or that it doesn't, we just don't know yet.
I generally disagree with the idea that fancy widgets and more processes are the main thing keeping the LW wiki from being good. I think the main problem is that not a lot of people are currently contributing to it.
The things that discourage me from contributing more look like:
-There are a lot of pages. If there are 700 bad pages and I write one really good page, there are still 699 bad pages.
-I don't have a good sense of which pages are most important. If I put a bunch of effort into a particular page, is that one that people are going to care about?
-I don't get much feedback about whether anyone saw the page after I edited it - karma for edits basically just comes from the tag dashboard and the frontpage activity feed.
So the improvements I would look for would be like:
-Expose view counts for wiki pages somewhere.
-Some sort of bat-signal on the tag dashboard for if a page is getting a lot of views but still has a bunch of TODO flags set.
-Big high-quality wiki page rewrites get promoted to frontpage or something.
-Someone of authority actually goes through and sets the "High Priority" flag on, say, 20 pages that they know are important and neglected.
-Some sort of event or bounty to drive more participation.
is one of the first results for "yudkowsky harris" on Youtube. Is there supposed to be more than this?
You should distinguish between “reward signal” as in the information that the outer optimization process uses to update the weights of the AI, and “reward signal” as in observations that the AI gets from the environment that an inner optimizer within the AI might pay attention to and care about.
From evolution’s perspective, your pain, pleasure, and other qualia are the second type of reward, while your inclusive genetic fitness is the first type. You can’t see your inclusive genetic fitness directly, though your observations of the environment can let you guess at it, and your qualia will only affect your inclusive genetic fitness indirectly by affecting what actions you take.
To answer your question about using multiple types of reward:
For the “outer optimization” type of reward, in modern ML the loss function used to train a network can have multiple components. For example, an update on an image-generating AI might say that the image it generated had too much blue in it, and didn’t look enough like a cat, and the discriminator network was able to tell it apart from a human generated image. Then the optimizer would generate a gradient descent step that improves the model on all those metrics simultaneously for that input.
For “intrinsic motivation” type rewards, the AI could have any reaction whatsoever to any particular input, depending on what reactions were useful to the outer optimization process that produced it. But in order for an environmental reward signal to do anything, the AI has to already be able to react to it.
This has overtaken the post it's responding to as the top-karma post of all time.
I'm impressed by the number of different training regimes stacked on top of each other.
-Train a model that detects whether a Minecraft video on Youtube is free of external artifacts like face cams.
-Then feed the good videos to a model that's been trained using data from contractors to guess what key is being pressed each frame.
-Then use the videos and input data to train a model that, in any game situation, does whatever inputs it guesses a human would be most likely to do, in an undirected shortsighted way.
-And then fine-tune that model on a specific subset of videos that feature the early game.
-And only then use some mostly-standard RL training to get good at some task.
While the engineer learned one lesson, the PM will learn a different lesson when a bunch of the bombs start installing operating system updates during the mission, or won't work with the new wi-fi system, or something: the folly of trying to align an agent by applying a new special case patch whenever something goes wrong.
No matter how many patches you apply, the safety-optimizing agent keeps going for the nearest unblocked strategy, and if you keep applying patches eventually you get to a point where its solution is too complicated for you to understand how it could go wrong.
Meta: This is now the top-voted LessWrong post of all time.
Robust Agents seems sort of similar but not quite right.
Looking at the generation code, aptitude had interesting effects on our predecessors' choice of cheats.
Good:
-Higher aptitude Hikkikomori and Otaku are less likely to take Hypercompetent Dark Side (which has lower benefits for higher aptitude characters).
Bad:
-Higher aptitude characters across the board are less likely to take Monstrous Regeneration or Anomalous Agility, which were some of the better choices available.
Ugly:
-Higher aptitude Hikkikomori are more likely to take Mind Palace.
I've added a market on Manifold if you want to bet on which strategy is best.
Somewhat. The profile pic changes based on the character's emotions, or their reaction to a situation. Sometimes there's a reply where the text is blank and the only content is the character's reaction as conveyed by the profile pic.
That said, it's a minor enough element that you wouldn't lose too much if it wasn't there.
On the other hand, it is important for you to know which character each reply is associated with, as trying to figure out who's talking from the text alone could get confusing in many scenes. So any format change should at least preserve the names.
If everyone ends up with the same vote distribution, I think it removes the incentive for colluding beforehand, but it also means the vote is no longer meaningfully quadratic. The rank ordering of the candidates will be in order of how many total points were spent on them, and you basically end up with score voting.
edit: I assume that the automatic collusion mechanism is something like averaging the two ballots' allocations for each candidate, which does not change the number of points spent on each candidate. If instead some ballots end up causing more points to be spent on their preferred candidates than they initially had to work with, there are almost definitely opportunities for strategic voting and beforehand collusion.
Or put a spoilered link to this post in the dath ilan tag's wiki text?