Posts
Comments
Oh that makes sense!
If the predictors can influence the world in addition to making a prediction, they would also have an incentive to change the world in ways that make their predictions more accurate than their opponents right? For example, if everyone else thinks Bob is going to win the presidency, one of the predictors can bribe Bob to drop out and then bet on Alice winning the presidency.
Is there work on this? To be fair, it seems like every AI safety proposal has to deal with something like this.
This is super cool stuff, thank you for posting!
I may have missed this, but do these scoring rules prevent agents from trying to make the environment more un-predictable? In other words, if you're competing against other predictors, it may make sense to influence the world to be more random and harder to understand.
I think this prediction market type issue has been discussed elsewhere but I can't find a name for it.
Thanks for this! I misinterpreted Lucius as saying "use the single highest and single lowest eigenvalues to estimate the rank of a matrix" which I didn't think was possible.
Counting the number of non-zero eigenvalues makes a lot more sense!
You can absolutely harvest potential energy from the solar system to spin up tethers. ToughSF has some good posts on this:
https://toughsf.blogspot.com/2018/06/inter-orbital-kinetic-energy-exchanges.html https://toughsf.blogspot.com/2020/07/tethers-all-way.html
Ideally your tether is going to constantly adjust its orbit so it says far away from the atmosphere, but for fun I did a calculation of what would happen if a 10K tonne tether (suitable for boosting 100 tonne payloads) fell to the Earth. Apparently it just breaks up in the atmosphere and produces very little damage. More discussion here:
The launch cadence is an interesting topic that I haven't had a chance to tackle. The rotational frequency limits how often you can boost stuff.
Since time is money you would want a shorter and faster tether, but a shorter time of rotation means that your time window to dock with the tether is smaller, so there's an optimization problem there as well.
It's a little easier when you've got catapults on the moon's surface. You can have two running side by side and transfer energy between them electrically. So load up catapult #1, spin it up, launch the payload, and then transfer the remaining energy to catapult #2. You can get much higher launch cadence that way.
Oops yes, that should read "Getting oxygen from the moon to LEO requires less delta V than going from the Earth to LEO!". I edited the original comment.
Lunar tethers actually look like they will be feasible sooner than Earth tethers! The lack of atmosphere, micrometeorites, and lower gravity (g) makes them scale better.
In fact, you can even put a small tether system on the lunar surface to catapult payloads to orbit: https://splittinginfinity.substack.com/p/should-we-get-material-from-the-moon
Whether tethers are useful on the moon depends on the mission you want to do. Like you point out, low delta-V missions probably don't need a tether when rockets work just fine. But if you want to take lunar material to low earth orbit or send it to Mars, a lunar tether is a great option.
The near-term application I'm most excited about is liquid oxygen. Getting oxygen from the moon to LEO requires less delta V than going from the Earth to LEO! Regolith is ~45% oxygen by mass and a fully-fueled Starship is 80% LOX by mass. So refueling ships in LEO with lunar O2 could be viable.
Even better, the falling lunar oxygen can spin up a tether in LEO which can use that momentum to boost a Starship to other parts of the solar system.
Thanks for the comments! Going point-by-point:
-
I think both fiberglass and carbon fiber use organic epoxy that's prone to UV (and atomic oxygen) degradation? One solution is to avoid epoxy entirely using parallel strands or something like a Hoytether. The other option is to remove old epoxy and reapply over time, if its economical vs just letting the tether degrade.
-
I worry that low-thrust options like ion engines and sails could be too expensive vs catching falling mass, but I could be convinced either way!
-
Yeah, some form of vibration damping will be important, I glossed over this. Bending modes are particularly a problem for glass. Though I would guess that vibrations wouldn't make the force along the tether any higher?
-
Catching the projectile is a key engineering challenge here! One that I probably can't solve from my armchair. As for missing the catch, I guess I don't see this as a huge issue? If the rocket can re-land, missing the catch means that the only loss is fuel. Though colliding with the tether would be a big problem.
-
Yeah I think low orbits are too challenging for tethers, so they're definitely going to be at risk of micrometeorite impacts. I see this as a key role of the "safety factor". Tether should be robust to ~10-50% of fibers being damaged, and there should be a way to replace/repair them as well.
-
Right, though tethers can't help satellites get to LEO, they can help them get to higher orbits which seems useful. But the real value-add comes when you want to get to the Moon and beyond.
-
Good to know! I would love to see more experiments on glass fibers pulled in space, small-scale catches, and data on what kinds of defects form on these materials in orbit.
Yeah, my overall sense is that using falling mass to spin the tether back up is the most practical. But solar sails and ion drives might contribute too, these are just much slower which hurts launch cadence and costs.
The fact that you need a regular supply of falling mass from e.g. the moon is yet another reason why tethers need a mature space industry to become viable!
That makes sense, I guess it just comes down to an empirical question of which is easier.
Question about what you said earlier: How can you use the top/bottom eigenvalues to estimate the rank of the Hessian? I'm not as familiar with this so any pointers would be appreciated!
Isn't calculating the Hessian for large statistical models kind of hard? And aren't second derivatives prone to numerical errors?
Agree that this is only valuable if sampling on the loss landscape is easier or more robust than calculating the Hessian.
You may find this interesting "On the Covariance-Hessian Relation in Evolution Strategies":
https://arxiv.org/pdf/1806.03674
It makes a lot of assumptions, but as I understand it if you: a. Sample points near the minima [1]. b. Select only the lowest loss point from that sample and save it. c. Repeat that process many times d. Create a covariance matrix of the selected points
The covariance matrix will converge to the inverse of the Hessian, assuming the loss landscape is quadratic. Since the inverse of a matrix has the same rank, you could probably just use this covariance matrix to bound the local learning coefficient.
Though since a covariance matrix has rank less than n-1 (where n is the number of sample points) you would need to sample and evaluate roughly d/2 points. The process seems pretty parallelize-able though.
[1] Specifically using an an isotropic, unit variance normal distribution centered at the minima.
Exciting to see this up and running!
If I'm understanding correctly, the system looks for modifications to certain viruses. So if someone modified a virus that NAO wasn't explicitly monitoring for modifications, then that would go undetected?
I like the simple and clear model and I think discussions about AI risk are vastly improved by people proposing models like this.
I would like to see this model extended by including the productive capacity of the other agents in the AI's utility function. In other words, the other agents have a comparative advantage over the AI in producing some stuff and the AI may be able to get a higher-utility bundle overall by not killing everyone (or even increasing the productivity of the other agents so they can produce more stuff for the AI to consume).
Super useful post, thank you!
The condensed vaporized rock is particularly interesting to me. I think it could be an asset instead of a hindrance. Mining expends a ton of energy just crushing rock into small pieces for processing, turning ores into dust you can pump with air could be pretty valuable.
I was always skeptical of enhanced geothermal beating solar on cost, though I do think the supercritical water Quaise could generate has interesting chemical applications: https://splittinginfinity.substack.com/p/recycling-atoms-with-supercritical
This post has some useful info:
https://milkyeggs.com/biology/lifespan-extension-separating-fact-from-fiction/
It basically says that sunscreen, ceramide moisturizers, and retinols are the main evidence-based skincare products. I would guess that more expensive versions of these don't add much value.
Some amount of experimentation is required to find products that don't irritate your skin.
Good framing! Two forms of social credit that I think are worth paying attention to:
- Play money prediction markets and forecasting. I think it's fruitful to think about these communities as using prediction accuracy as a form of status/credit.
- Cryptocurrencies, which are essentially financial credit but with its own rules and community. The currency doesn't have to have a dollar value to induce coordination it can still function as a reputation system and medium of exchange.
It's somewhat tangential, but Sarah Constantin discussing attestation has some insight on this I think (I put some comments here).
Note that these sorts of situations are perfectly foreseeable from the perspective of owners. They know precisely what they will pay each year in taxes based on their bid. It's prudent to re-value the home every once in a while if taxes drift too much, but the owner can keep the same schedule if they want. They can also use the public listing of local bids, so they know what to bid and can feel pretty safe that they will keep their home. They truly have the highest valuation of all the bidders in most cases.
The thing is, every system of land ownership faces a tradeoff between investment efficiency and allocative efficiency. This is a topic in the next post, where I'll discuss why the best growth rate of taxes closely follows the true growth rate of land values. Essentially, you want people to pay their fair share. Unfortunately, any system that has taxes move along with land values will risk "taxing people out of their homes", there are legitimate ways to do land policy on either end of the spectrum.
The neat thing about this system is that you can choose where on the spectrum you want to be! If you want high investment efficiency (i.e. people can securely hold their homes and don't have to worry about re-auctioning) then just set the tax growth rate to zero; that way the owner pays a fixed amount each year indefinitely. In net present value terms, the indefinite taxes will be finite and the tax rate can be set to adjust this amount up or down.
If for some reason you want allocative efficiency, you can crank the growth rate high enough to trigger annual auctions. This is bad for physical land, but this could be valuable for other types of economic land like broadband spectrum.
Land value taxation is designed to make land ownership more affordable by lowering the cost to buy land. Would it change the value of property as an investment for current owners? I'm not sure, one one hand, land values would go down, but on the other, land would get used more efficiently and deadweight loss of taxation would go down, boosting the local economy.
As for the public choice hurdles, reform doesn't seem intractable. Detroit is considering a split-rate property tax, and it's not infeasible that other places switch. Owners hate property taxes and land values are less than property values. Why not slowly switch to using land values and lower everyone's property tax bill? That seems like it could be popular with voters, economists, and politicians.
This proposal doesn't involve any forced moves, owners only auction when they want to sell their land.
So yes, taxing property values is undesirable, but it also happens with imperfect land value assessments: https://www.jstor.org/stable/27759702
It looks like you have different numbers for the cost of land, sale value of a house, and cost of construction. I'm not an expert, so I welcome other estimates. A couple comments:
- Land value assessors typically say that the land value is larger than the improvement value. In urban centers, land can be over 70% of the overall property value. I would guess this is where the discrepancy comes from with our numbers. AEI has a nice graphic of this here:
https://www.aei.org/housing/land-price-indicators/
- Overhead costs of construction would act to reduce the overall distortion, since those are included in C_b in the formula for distortion. The construction costs look larger in that article than what I used, but I guess what we really need to know is the markup from construction.
Let's just keep all the construction and demolition costs the same and use your land value ($100K) and improvement value ($400K):
P = 400K + 0.5*(76K -(400K + 10K)) = 233K B = 100K + ((400-233) - 0.05*(400-233)*10)*0.31 = 126K
Total = 359K
So the buyer gets 500K of property for $359K, a 28% price reduction. The land tax is ~25% improvement value. It's easy to adjust land taxes down by 25% so that you tax the correct amount, but the implicit tax on property is a big problem in this case.
The thing is, I don't think land value being only 20% of property values is realistic, especially in urban areas. Median land share in the US is more like 50% so I'm not really sure where the discrepancy comes from.
As for skyscrapers, the interesting thing about this proposal is that hard-to-remove amendments essentially become land. For example, if you made a plot of land fertile, that improvement is difficult/undesirable to remove, so when you go to sell it, the owner pays for it as if it were land. I'll tackle this more in the second post.
Thanks for the clarification! Do you know if either condition is associated with abnormal levels of IGF-1 or other growth hormones?
Are there examples of ineffective drugs leading to increased FDA stringency? I'm not as familiar with the history. For example, people agree that Aducanumab is ineffective, has that cause people to call for greater scrutiny? (genuinely asking, I haven't followed this story much).
There are definitely examples of a drug being harmful that caused increased scrutiny. But unless we get new information that this drug is unsafe, that doesn't seem to be the case here.
I agree that the difference between disease-treating interventions (that happen to extend life) versus longevity interventions is murky.
For example, would young people taking statins to prevent heart disease be a longevity intervention?
https://johnmandrola.substack.com/p/why-i-changed-my-mind-about-preventing
See this post arguing that rapamycin is not a longevity drug:
https://nintil.com/rapamycin-not-aging
Broadly, I'm not too concerned with what we classify a drug as as long as its safe, effective, well-understood, and gets approved by regulatory authorities.
I personally don't expect very high efficacy, and I do expect that Loyal will sell the drug for the next 4.5 years. However, as long as Loyal is clear about the nature of the approval of the drug, I think this is basically fine. People should be allowed to, at their own expense, give their pets experimental treatments that won't hurt them and might help them. They should also be able to do the same for themselves, but that's a fight for another day.
Agreed! Beyond potentially developing a drug, think Loyal's strategy has the potential to change regulations around longevity drugs, raise profits for new trials, and bring attention/capital to the longevity space. I don't see many downside risks here unless the drug turns out to be unsafe.
Note: I'm not affiliated with Loyal or any other longevity organization, I'm going off the same outside information as the author.
I think there's a substantial chance that this criticism is misguided. A couple points:
The term "efficacy nod" is a little confusing, the FDA term is "reasonable expectation of effectiveness", which makes more sense to me, it sounds like the drug has enough promise that the FDA thinks its worth continuing testing. They may not have actual effectiveness data yet, just evidence that it's safe and a reasonable explanation for why it might work.
It's surprising, to say the least, to see a company go from zero information to efficacy nod, because, well, what are you basing your efficacy on?
I don't know what the standard practices are for releasing trial data, especially for an initial trial like this. Are we sure this isn't standard practice? Even if it isn't, I don't think this is sufficient to assume that Loyal is being disingenuous.
They then looked at healthy pet dogs, and found that big dogs had higher levels of IGF-1, which is one of the reasons they're big. Small dogs had lower levels of IGF-1. Small dogs, as we all know, live longer than big dogs. Therefore, Loyal said, our IGF-1 inhibitor will extend the life of dogs. Needless to say, this is bad science. Really bad science.
Take the outside view here, both Loyal and the FDA have veterinarians who seem to think that the drug is promising.
I also think there's a reasonable argument to be made for an IGF-1 inhibitor in large-breed dogs. Large breed dogs often die of heart disease which is often due to dilated cardiomyopathy (heart becomes enlarged and can't pump blood effectively). This enlargement can come from hypertrophic cardiomyopathy (overgrowth of the heart muscle). I don't know if it's known why large breed dogs have hypertrophic cardiomyopathy, but maybe IGF-1 makes the heart muscle grow over a dog's lifetime which would suggest that an IGF-1 inhibitor is worth trying. It's also suggestive that diabetes is a risk-factor for cardiomegaly (enlarged heart).
With this in mind, we can answer the next point:
And, even if they did, there's no reason to believe that lowering levels of IGF-1 would reverse any of the "damage" caused by high levels of IGF-1! The big dogs will still be big!
So my theory says that high IGF-1 over a lifetime progressively increases the size of the heart muscle until you get dilated cardiomyopathy. Stopping IGF-1 even in middle age might help. We can falsify this theory by checking if large breed dogs show heart enlargement over their lifetime (instead of growth stopping after puberty like it should). Why would heart muscle keep growing while nothing else does? I'm not sure.
Now we can turn to the question: do large breed dogs actually have elevated IGF-1?
Looking at your first figure, the answer seems to be yes! There's a straightforward correlation between bodyweight and IGF-1 concentration, the slope would likely be higher without the 3 outliers on the right. Notice also that the sample doesn't have many large breed dogs (great danes weigh 110-180 lbs). I would guess that those 3 dogs are large breed dogs, and they do in fact have IGF-1 levels higher than most of the dogs in the sample.
Now lets turn to the second plot, we see that IGF-1 concentration decreases with age. Remember that there is survivorship bias at higher ages, large breed dogs with higher IGF-1 will die at around 72 months while chihuahuas will live over 150 months. Declining IGF-1 with age is exactly what we should see if we expected IGF-1 to correlate with longevity! The plot supports the theory that IGF-1 is important for aging, you can't cherry-pick outliers and ignore the overall relationship in the plot.
oh, did I forget to mention that IGF-1 inhibitors have existed for humans for decades, and they have zero evidence of being longevity drugs?
I'm no expert, but I think there's interest in IGF-1 inhibitors for longevity. To quote Sarah Constantin:
There’s a _lot _of evidence that the IGF/insulin signaling/growth hormone metabolic pathway is associated with aging and short lifespan, and that inhibiting genes on that pathway results in longer lifespan. IGF-receptor-inhibiting or growth-hormone-inhibiting drugs could be studied for longevity, but haven’t yet.
I would guess this is one of the reasons Loyal had an interest in IGF-1 inhibitors from the outset.
IGF-1 inhibitors can cause low levels of blood platelets, elevated liver enzymes, and hyperglycemia
The dose makes the poison! Every drug has negative effects at a high enough dose, the trials will determine if these actually arise at the dose they are using.
I'm no expert, but this evidence doesn't seem sufficient to stop research on this drug. Will it prove safe or effective? Will it also benefit human health? I have no idea, but unless we discover that the drug is hurting patients, I think its fine for Loyal to carry on.
Thanks for writing this!
In addition to regulatory approaches to slowing down AI development, I think there is room for "cultural" interventions within academic and professional communities that discourage risky AI research:
https://www.lesswrong.com/posts/ZqWzFDmvMZnHQZYqz/massive-scaling-should-be-frowned-upon
Could someone help me collect the relevant literature here?
I think the complete class theorems are relevant: https://www.lesswrong.com/posts/sZuw6SGfmZHvcAAEP/complete-class-consequentialist-foundations
The Non-Existence of Representative Agents: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3302656
Representative Agents: https://en.wikipedia.org/wiki/Representative_agent
John Wentworth on Subagents: https://www.lesswrong.com/posts/3xF66BNSC5caZuKyC/why-subagents
Two arguments I would add:
- Conflict has direct costs/risks, a fight between AI and humanity would make both materially worse off
- Because of comparative advantage, cooperation between AI and humanity can produce gains for both groups. Cooperation can be a Pareto improvement.
Alignment applies to everyone, and we should be willing to make a symmetric commitment to a superintelligence. We should grant them rights, commit to their preservation, respect it's preferences, be generally cooperative and avoid using threats, among other things.
It may make sense to commit to a counterfactual contract that we expect an AI to agree to (conditional on being created) and then intentionally (carefully) create the AI.
Standardization/interoperability seems promising, but I want to suggest a stranger option: subsidies!
In general, monopolies maximize profit by setting an inefficiently high price, meaning that they under-supply the good. Essentially, monopolies don't make enough money.
A potential solution is to subsidize the sale of monopolized goods so the monopolist increases supply to the efficient level.
For social media monopolies, they charge too high a "price" by using too many ads, taking too much data, etc. Because of the network effect, it would be socially beneficial to have more users, but the social media company drives them away with their high "prices". The socially efficient network size could be achieved by paying the social media company per active user!
I was planning to write this up in more detail at some point (see also). There are of course practical difficulties with identifying monopolies, determining the correct subsidy in an adversarial environment, Sybil attacks, etc.
Nice post, thanks!
Is there a formulation of UDASSA that uses the self-indication assumption instead? What would be the implications of this?
Frowning upon groups which create new, large scale models will do little if one does not address the wider economic pressures that cause those models to be created.
I agree that "frowning" can't counteract economic pressures entirely, but it can certainly slow things down! If 10% of researchers refused to work on extremely large LM's, companies would have fewer workers to build them. These companies may find a workaround, but it's still an improvement on the situation where all researchers are unscrupulous.
The part I'm uncertain about is: what percent of researchers need to refuse this kind of work to extend timelines by (say) 5 years? If it requires literally 100% of researchers to coordinate, then it's probably not practical, if we only have to convince the single most productive AI researcher, then it looks very doable. I think the number could be smallish, maybe 20% of researchers at major AI companies, but that's a wild guess.
That being said, work on changing the economic pressures is very important. I'm particularly interested in open-source projects that make training and deploying small models more profitable than using massive models.
On outside incentives and culture: I'm more optimistic that a tight-knit coalition can resist external pressures (at least for a short time). This is the essence of a coordination problem; it's not easy, but Ostrom and others have identified examples of communities that coordinate in the face of internal and external pressures.
I like this intuition and it would be interesting to formalize the optimal charitable portfolio in a more general sense.
I talked about a toy model of hits-based giving which has a similar property (the funder spends on projects proportional to their expected value rather than on the best projects):
https://ea.greaterwrong.com/posts/eGhhcH6FB2Zw77dTG/a-model-of-hits-based-giving
Updated version here: https://harsimony.wordpress.com/2022/03/24/a-model-of-hits-based-giving/
Great post!!
I think the section "Perhaps we don’t want AGI" is the best argument against these extrapolations holding in the near-future. I think data limitations, practical benefits of small models, and profit-following will lead to small/specialized models in the near future.
https://www.lesswrong.com/posts/8e3676AovRbGHLi27/why-i-m-optimistic-about-near-term-ai-risk
Yeah I think a lot of it will have to be resolved at a more "local" level.
For example, for people in a star system, it might make more sense to define all land with respect to individual planets ("Bob owns 1 acre on Mars' north pole", "Alice owns all of L4" etc.) and forbid people from owning stationary pieces of space. I don't have the details of this fleshed out, but it seems like within a star system, its possible to come up with a sensible set of rules and have the edge cases hashed out by local courts.
For the specific problem of predicting planetary orbits, if we can predict 1000 years in the future, it seems like the time-path of land ownership could be updated automatically every 100 years or so, so I don't expect there to be huge surprises there.
For taxation across star systems, I'm having trouble thinking of a case where there might be ownership ambiguity given how far apart they are. For example, even when the Milky Way and Andromeda galaxies collide, its unlikely that any stars will collide. Once again, this seems like something that can be solved by local agreements where owners of conflicting claims renegotiate them as needed.
I feel like something important got lost here. The colonists are paying a land value tax in exchange for (protected) possession of the planet. Forfeiting the planet to avoid taxes makes no sense in this context. If they really don’t want to pay taxes and are fine with leaving, they could just leave and stop being taxed; no need to attack anyone.
The “its impossible to tax someone who can do more damage than their value” argument proves too much; it suggests that taxation is impossible in general. It’s always been the case that individuals can do more damage than could be recouped in taxation, and yet, people still pay taxes.
Where are the individuals successfully avoiding taxation by threatening acts of terrorism? How are states able to collect taxes today? Why doesn’t the U.S. bend to the will of weaker states since it has more to lose? It’s because these kinds of threats don’t really work. If the U.S. caved to one unruly individual then nobody would pay taxes, so the U.S. has to punish the individual enough to deter future threats.
... this would provide for enough time for a small low value colony, on a marginally habitable planet, to evacuate nearly all their wealth.
But the planet is precisely what's being taxed! Why stage a tax rebellion only to forfeit your taxable assets?
If the lands are marginal, they would be taxed very little, or not at all.
Even if they left the planet, couldn’t the counter strike follow them? It doesn’t matter if you can do more economic damage if you also go extinct. It’s like refusing to pay a $100 fine by doing $1000 of damage and then ending up in prison. The taxing authority can precommit to massive retaliation in order to deter such behavior. The colony cannot symmetrically threaten the tax authority with extinction because of the size difference.
All of this ignores the practical issues with these weapons, the fact that earth’s value is minuscule compared to the sun, the costs of forfeiting property rights, the relocation costs, and the fact that citizens of marginal lands would receive net payments from the citizens dividend.
There are two possibilities here:
-
Nations have the technology to destroy another civilization
-
Nations don't have the technology to destroy another civilization
In either case, taxes are still possible!
In case 1, any nation that attempts to destroy another nation will also be destroyed since their victim has the same technology. Seems better to pay the tax.
In case 2, the Nation doesn't have a way to threaten the authorities, so they pay the tax in exchange for property rights and protection.
Thus threatening the destruction of value several orders of magnitude greater than the value to be collected is a viable deterrent.
Agreed, but to destroy so much value, one would have to destroy at least as much land as they currently control. Difficulties of tax administration mean that only the largest owners will be taxed, likely possessing entire solar systems. So the tax dodge would need to destroy a star. That doesn't seem easy.
It’s impossible, without some as yet uninvented sensing technology, to reliably surveil even the few hundred closest star systems.
I'm more optimistic about sensing tech. Gravitational lensing, superlenses, and simple scaling can provide dramatic improvements in resolution.
It's probably unnecessary to surveil many star systems. Allied neighbors on Alpha Centauri can warn earth about an incoming projectile as it passes by (providing years of advanced notice), so nations might only need to surveil locally and exchange information.
... once it’s past the Oort Cloud and relatively easy to detect again, there will be almost no time left at 0.5 c
It would take 2-3 years for a 0.5c projectile to reach earth from the outer edge of the Oort cloud, this seems like enough time to adapt. At the very least, its enough time to launch a counterstrike.
Fast projectiles may be impractical given that a single collision with the interstellar medium would destroy them. Perhaps thickening the Oort cloud could be an effective defense system.
A second-strike is only a credible counter if the opponent has roughly equal amounts to lose.
Agents pre-commit to a second strike as a deterrent, regardless of how wealthy the aggressor is. If the rebelling nation has the technology to destroy another nation and uses it, they're virtually guaranteed to be destroyed by the same technology.
Given the certainty of destruction, why not just pay the (intentionally low, redistributive, efficiency increasing, public-good funding) taxes?
I imagine that these policies will be enforced by a large coalition of members interested in maintaining strong property rights (more on that later).
Its not clear that space war will be dominated by kinetic energy weapons or MAD:
-
These weapons seem most useful when entire civilizations are living on a single planet, but its possible that people will live in disconnected space habitats. These would be much harder to wipe out.
-
Any weapon will take a long time to move over interstellar distances. A rebelling civilization would have to wait thousands of years for the weapon to reach its target. It seems like their opponent could detect the weapon and respond in this time.
-
Even if effective civilization-destroying weapons are developed, mutual defense treaties and AI could be used to launch a second strike, making civilization-scale attacks unlikely.
In general, any weapon a single civilization might use to avoid taxation can also be used by a group of civilizations to enforce taxation.
On the other hand, it does seem like protecting territory/resources from theft will be an issue. This is where property rights come in. Governments, corporations, and AI's will want to have their property protected over very long timescales and don't want to spend most of their resources on security.
Over long timescales, a central authority can help them defend their territory (over short timescales, these groups will still have to protect their own property since it will take a while for allies to reach them). But order to receive this protection, these groups must register what they own and pay taxes towards mutual defense. I think a land value tax is the "right" way to raise taxes in this case.
This approach makes more sense for smaller systems where defense is easier but it may also be useful across much larger distances if agents are very patient.
(The "central authority" also doesn't have to be an actual central government. States could pay neighbors to come to their aid using a portion of the collected taxes, kind of like "defense insurance")
I haven't given this a thoroughly read yet, but I think this has some similarities to retroactive public goods funding:
https://harsimony.wordpress.com/2021/07/02/retroactive-public-goods-funding/
https://medium.com/ethereum-optimism/retroactive-public-goods-funding-33c9b7d00f0c
The impact markets team is working on implementing these:
Going by figure 5, I think the way to format climate contingent finance like an impact certificate would be:
- 'A' announces that they will award $X in prizes to different project based on how much climate change damage a project averts
- 'B' funds different projects (based on how much damage they think each project will avert in the future) in exchange for a percentage of the prize money
... robust broadly credible values for this would be incredibly valuable, and I would happily accept them over billions of dollars for risk reduction ...
This is surprising to me! If I understand correctly, you would prefer to know for certain that P(doom) was (say) 10% than spend billions on reducing x-risks? (perhaps this comes down to a difference in our definitions of P(doom))
Like Dagon pointed out, it seems more useful to know how much you can change P(doom). For example, if we treat AI risk as a single hard step, going from 10% -> 1% or 99% -> 90% both increase the expected value of the future by 10X, it doesn't matter much whether it started at 10% or 99%.
For prioritization within AI safety, are there projects in AI safety that you would stop funding as P(doom) goes from 1% to 10% to 99%? I personally would want to fund all the projects I could, regardless of P(doom) (with resources roughly proportional to how promising those projects are).
For prioritization across different risks, I think P(doom) is less important because I think AI is the only risk with greater than 1% chance of existential catastrophe. Maybe you have higher estimates for other risks and this is the crux?
In terms of institutional decision making, it seems like P(doom) > 1% is sufficient to determine the signs of different interventions. In a perfect world, a 1% chance of extinction would make researchers, companies, and governments very cautious, there would be no need to narrow down the range further.
Like Holden and Nathan point out, P(doom) does serve a promotional role by convincing people to focus more on AI risk, but getting more precise estimates of P(doom) isn't necessarily the best way to convince people.
Yes, "precision beyond order-of-magnitude" is probably a better way to say what I was trying to.
I would go further and say that establishing P(doom) > 1% is sufficient to make AI the most important x-risk, because (like you point out), I don't think there are other x-risks that have over a 1% chance of causing extinction (or permanent collapse). I don't have this argument written up, but my reasoning mostly comes from the pieces I linked in addition to John Halstead's research on the risks from climate change.
You need to multiply by the amount of change you think you can have.
Agreed. I don't know of any work that addresses this question directly by trying to estimate how much different projects can reduce P(doom) but would be very interested to read something like that. I also think P(doom) sort of contains this information but people seem to use different definitions.
Setting aside how important timelines are for strategy, the fact that P(doom) combines several questions together is a good point. Another way to decompose P(doom) is:
-
How likely are we to survive if we do nothing about the risk? Or perhaps: How likely are we to survive if we do alignment research at the current pace?
-
How much can we really reduce the risk with sustained effort? How immutable is the overall risk?
Though people probably mean different things by P(doom) and seems worthwhile to disentangle them.
Talking about our reasoning for our personal estimates of p(doom) is useful if and only if it helps sway some potential safety researchers into working on safety ...
Good point, P(doom) also serves a promotional role, in that it illustrates the size of the problem to others and potentially gets more people to work on alignment.
You may also want to check out John Wentworth's natural abstraction hypothesis work:
Oh I didn't realize! Thanks for clarifying. Uncertainty about location probably doesn't contribute much to the loss then.
Is it known how well performance scales with the size of the prompt and size of the fine-tuning dataset? i.e. something like the Chinchilla paper but for prompt and dataset size.
Interesting!
So if I am understanding correctly, SIA puts more weight on universes with many civilizations, which lowers our estimate of survival probability q. This is true regardless of how many expanding civs. we actually observe.
The latter point was surprising to me, but on reflection, perhaps each observation of an expanding civ also increases the estimated number of civilizations. That would mean that there are two effects of observing an expanding civ: 1) Increased the feasibility of passing a late filter 2) increasing the expected number of civilizations that didn't pass the filter. These effects might cancel out leaving one with no net update.
So I was wrong in saying that Grabby Aliens should reduce our x-risk estimates. This is interesting because in a simple discounting model, higher baseline risk lowers the value of trying to mitigate existential risks:
This implies that a person hoping to encourage work on existential risks may want to convince others that it's feasible to expand into space.
I wonder about different approaches to SIA. For example, could a different version of SIA be used to argue that we are likely the ancestors of a large civilization? Would this up-weight the chances of cosmic expansion?
Some other order-of-magnitude estimates on available data, assuming words roughly equal tokens:
Wikipedia: 4B English words, according to this page.
Library of Congress: from this footnote a assume there are at most 100 million books worth of text in the LoC and from this page assume that books are 100k words, giving 10T words at most.
Constant writing: I estimate that a typical person writes at most 1000 words per day, with maybe 100 million people writing this amount of English on the internet. Over the last 10 years, these writers would have produced 370T words.
Research papers: this page estimates ~4m papers are published each year, at 10k words per paper with 100 years of research this amounts to 4T words total.
So it looks like 10T words is an optimistic order-of-magnitude estimate of the total amount of data available.
I assume the importance of a large quantity of clean text data will lead to the construction of a text database of ~1T tokens and that this database (or models trained on it) will eventually be open-sourced.
From there, it seems like really digging in to the sources of irreducible error will be necessary for further scaling. I would guess that a small part of this is "method error" (training details, context window, etc.) but that a significant fraction comes from intrinsic text entropy. Some entropy has to be present, or else text would have no information value.
I would guess that this irreducible error can probably be broken down into:
-
Uncertainty about the specific type of text the model is trying to predict (e.g. it needs some data to figure out that it's supposed to write in modern English, then more data to learn that the writing is flamboyant/emotional, then more to learn that there is a narrative structure, then more to determine that it is a work of fiction etc.). The model will always need some data to specify which text-generating sub-model to use. This error can be reduced with better prompts (though not completely eliminated)
-
Uncertainty about location within the text. For example, even if the model had memorized a specific play by Shakespeare, if you asked it to do next-word prediction on a random paragraph from the text, it would have trouble predicting the first few words simply because it hasn't determined which paragraph it has been given. This error should go away when looking at next-word prediction after the model has been fed enough data. Better prompts and a larger context window should help.
-
Uncertainty inherent to the text. This related to the actual information content of the text, and should be irreducible. I'm not sure about the relative size of this uncertainty compared to the other ones, but this paper suggests an entropy of ~10 bits/word in English (which seems high?). I don't know how entropy translates into training loss for these models. Memorization of key facts (or database access) can reduce the average information content of a text.
EDIT: also note that going from 10T to 100T tokens would only reduce the loss by 0.045, so it may not be worthwhile to increase dataset size beyond the 10T order-of-magnitude.
Right. Similar to a property tax, this would discourage land improvements somewhat (though unlike a property tax, it would not discourage non-land improvements like houses).
All land value taxes do something like this. In practice, the effect is small because individual changes to land values are dwarfed by changes caused by external factors like local economic growth.
Great idea, thanks for posting this!
I wrote a post on how to have productive disagreements with loved ones:
https://harsimony.wordpress.com/2022/06/21/winning-arguments-with-loved-ones/
Here is the subsection on analyzing disagreements:
Because arguments are emotional, it can be helpful to try to dispassionately assess the situation with your partner and get to the root of the problem.
The first step is to break the disagreement down into isolated chunks. Identify the handful of differences you are having, and deal with them as independently as possible. If you notice discussion of one problem bleeding into another, try to refocus on the subject at hand.
For each problem, try to set aside your ego and emotions; what’s the real issue? Ask fair questions of your partner and hear them out. At the end of your discussion, you should be able to articulate your partner’s point in a way they would find satisfactory. A surprising number of arguments can be resolved simply by giving both partners a fair hearing.
It can be helpful to try to come up with neutral examples related to your problem. Flip the roles that you and your partner play in the example and be honest about how it would change things. It also can be helpful to consider things from a third-person perspective [4].
At the end of all of this, you should come to an agreement about the things you will do differently in the future. Once again, it doesn’t have to be perfect, it just has to be something you can build upon.
Suppose I could buy some very cheap land, clear trees and scrub, remove rocks, construct a dam, fertilize the soil and so on, such that on the open market I could now lease it to a farmer at $50,000/year instead of nearly nothing.
Since the taxes are based on the sale price of the empty land you bought, your taxes in this case would remain the same despite the improvements (not 50k/year). But once you sold the land, the next owner would pay 50k/year, since they paid the true price at auction.
There is an incentive to improve land, but unfortunately this plan also encourages people to hold onto land that they bought for cheap in order to avoid taxes (which lowers efficiency).
I think LVT's have to tradeoff between:
- Raising taxes to accurately reflect land values at the cost of "taxing people out of their home" (i.e. for some people, nearby economic growth will raise their land value, increase taxes, and outpace their ability to pay, forcing them to move out).
- Keeping taxes relatively constant at the cost of lower economic efficiency (forcing people to move out is a key part of ensuring that land goes to the person that most values using it).
I think there is a spectrum of possible approaches that take different positions on this tradeoff, but I haven't found one that fully satisfies me.