People generally expected math AI to progress pretty fast already. I was angry about machine-assisted math being a neglected area before this and my anger levels aren't substantially increased by the news.

I'm not sure that cleaving a distinction between the meanings of "possession" and "ownership" is a winnable battle.

but more because of measurement errors

I sometimes think of growing private wealth as the fee that a democracy must pay for the services of the techne that it cannot expropriate.

Actually, in my father's day, there was this one infamous landlord who people used to point at. Who was it, Claude? "Peter Rachman" yes that was it. People used to talk about "Rachmanism" which apparently just meant "being a cruel and unscrupulous landlord", but which could probably be expanded to encompass home owner associations organizing to lobby against zoning reform if efforts were made, because when people hear ism they expect to see a civic ideology.

Unfortunately one of his most prominent sins was excessive subdivision, so he wasn't exactly a nimby.

And also, yeah, in reality — for instance in New Zealand which has it especially bad, where substantially more than half of the families in the country own land — land investment is just a popular way of keeping savings. The last Labour government felt a need to promise, despite confessing the will, not to touch it.

Maybe "abstract" was the wrong word. Communism and minarchy both have very simple visceral moral impulses supporting them. Fairness/equality vs liberty/choice. It's possible to get a person into a state where they feel one pole so intensely that they will be willing to fight against someone fighting earnestly for the other pole (right? But I'm not sure there's actually been a civil war between communists and minarchists, it's usually been communists vs monarchists/nationalists)

For grey civics, I don't know what the unifying principle is. Commitment to growth? Progress? Hey, maybe that's it. I've been considering the slogan "defending zoning isn't compatible with calling yourself a progressive. If you believe in urban zoning you don't believe in progress."

Progress seems to require meritocracy, rewarding work in proportion to its subjective EV or capricious outcomes, distributing rewards unevenly, and progress comes with a duty to future generations that minarchists might not like very much, but at least in tech, people seem alright with that.

On the left, the tension mostly comes out of earnest disbelief, it's not intuitive that progress is real. For most of our evolutionary history it wasn't real, and today it happens only on the scale of years, and its every step is unprecedented.

But how would we resolve the tension with humanism. I guess e/acc is the faction within the grey tribe who don't try to resolve that tension, they lean into it, they either explicitly reject the duty to defend the preferences of present humanity against those aspects of progress that threaten it, or they find reasons to downplay the immanence of those threats. The other faction has to sit and listen while Hanson warns them about the inevitability of absolute cultural drift, and I don't think we know what to say to that.

A focus question I keep returning to, partly out of a dark sense of humor, is what kind of slogans I would have to write before people would be willing to shed blood, to fight, in the name of George, or in the name of Approval Voting, or Minor Parties, or any other system whose merits are abstract and which is neither minarchist nor soviet.

I sometimes feel like part of the problem is that there is no villain to get mad at. Our enemy is mostly just ambient incompetence, myopia, legacy systems and institutional sclerosis. Perhaps we can call it Moloch.

Afaik rent in japan is low on a global stage, and is only high relative to japanese wages?

OTOH, japanese rent shouldn't be high relative to japanese wages (unless there are a whole lot of foreigners doing remote work or something) so I guess you could still say that the rent is too high.

I wonder who, if anyone, is in charge of deciding what Perplexity's list of reliable sources is.

Defecting becomes unlikely if everyone can track the compute supply chain and if compute is generally supposed to be handled exclusively by the shared project.

Afaik there were not Generals saying "Covid could kill every one of us if we don't control the situation" and controlling the situation would have required doing politically unpopular things rather than politically popular things.

Change either of those factors and it's a completely different kind of situation.

I tend to dismiss scenarios where it's obvious, because I expect the demonstration of strong misaligned systems to inspire a strong multi-government response. Why do you expect this not to happen?

It occurs to me that the earliest demonstrations will be ambiguous to external parties, it will be one research org saying that something that doesn't quite look strong enough to take over would do something if it were put in a position it's not obvious it could get to, and then the message will spread incompletely, some will believe it, others wont, a moratorium wont be instated, and a condition of continuing to race in sin could take root.

But I doubt that ambiguous incidents like this would be reported to government at all? Private research orgs generally have a really good idea of what they can or can't communicate to the outside world. Why cry wolf when you still can't show them the wolf? People in positions of leadership in any sector are generally very good at knowing when to speak or not.

On my homeworld, with specialist consultants (doctors, lawyers etc), we subsidize "open consultation", which is when a client meets with more than one fully independent consultant at a time.
If one consultant misses something, the others will usually catch it, healthy debate will take place, a client will decide who did a better job and contract them or recommend them more often in the future. You do have the concept of "getting a second opinion" here, but I think our version worked a lot better for some subtle reasons.

It produced a whole different atmosphere, like, the water was cleaner. It totally changes the dynamic of the consultations. People tried harder, and they listened more, they had to, and they were happy to, they found it energizing.

As a result of this, we didn't really have any need for occupational licensing. If someone was making bad recommendations, the other people in the room would just tell the client that lmao.

As part of this increased mobility and independence of specialists, practices end up being a very different kind of business, they weren't brands or instutitions who owned their consultants, they were just facilities which specialists and clients used. Consultants worked in different offices on different days depending on who they were meeting with.

Drawbacks? Not really, but like, to facilitate the inclusion of younger specialists, we had to make sure an aspirant with good investor assessments could bid to attend open consultations and learn from them and to advertise themselves to clients and get discovered.
Occasionally this would result in the presence of a corrupt aspirant who has no real career prospects and was just there to cash in and sell the client some addiction or other on behalf of an advertiser. Usually, though, experienced specialists are able to convince the client to just ask those aspirants to leave pretty quickly. It didn't happen often and those few clients who've experienced this issue came away mostly just amused.
Usually, attending aspirants are suitably humble and only speak up when they have something to contribute.

I guess this was coming from our general all-pervading paranoia about cult dynamics, which got pretty extreme sometimes (ie, most kids movies were legally required to end with a section where two or three independent education theorists reflect on the themes, and offer alternative views. The kids always want to skip them! xD), but I definitely preferred our way of doing specialist consultation over the way you do it here :S here I feel like you're expected to just trust your doctor? And also the fact that you have to use occupational licensing because you don't have this is part of the reason I think there's so little innovation, you haven't created conditions where the better specs can step up and compete and shine and get discovered!

Prediction in draft: Linkposts from blogs are going to be the most influential form of writing over the next few years, as they're the richest data source for training LLM-based search engines, which will soon replace traditional keyword-based search engines.

I wrote about this, and I agree that it's very important to retain archival copies of misaligned AIs, I go further and claim it's important even for purely selfish diplomacy reasons https://www.lesswrong.com/posts/audRDmEEeLAdvz9iq/do-not-delete-your-misaligned-agi

IIRC my main sysops suggestion was to not give the archival center the ability to transmit data out over the network.

The ‘answer engine’ idea is great, and occasionally I find it the right tool for the right job although mostly I end up at either the Google Search or Claude Sonnet ends of the spectrum.

It's surprisingly rare that Perplexity can answer a question that google can't for me.

What I really need from AI search is usually far better serviced by the little known and unspectacular exa.ai.

Humorously, in retrospect, it turns out we actually did need a much larger government project to realize the potential of the internet. Licklider's visions of a new infrastructures for trade and collective intelligence were all totally realistic, those things could have been built, the internet would have been very different and much more useful if the standards had been designed, the distributed database design problems had been solved, but no one did that work! There was no way to turn it into a business! So now the internet is mostly just two or three entertainment/advertising tubes.

I think I agree that this is possible, and it's closely connected to the reasons I think making alignable ASI open source definitely wouldn't lead to egalitarian outcomes:

Artificial Superintelligence, lacking the equalizing human frailties of internal opacity, corruption (principle-agent problems), and senescence, is going to be even more prone to rich-get-richer monopoly effects than corporate agency was.
You can give your opponent last-gen ASI, but if they have only a fraction of the hardware to run it on, and only a fraction of the advanced manufacturing armatures. Being behind on the superexponential curve of recursively accelerating progress leaves them with roughly zero part of the pie.

(Remember that the megacorporations who advocate for open source AI in the name of democratization while holding enduring capital advantages in reserve all realize this.)

Well let's fix this then?

I find that it is better than not racing. Advocating for an international project to build TAI instead of racing turns out to be good if the probability of such advocacy succeeding is ≥20%.

Both of these sentences are false if you accept that my position is an option (racing is in fact worse than international cooperation which is encompassed within the 'not racing' outcomes, and advocating for an international project is in fact not in tension with racing whenever some major party is declining to sign on.)

There are actually a lot of people out there who don't think they're allowed to advocate for a collective action without cargo culting it if the motion fails, so this isn't a straw-reading.

As an advocate for an international project, I am not advocating for individual actors to self-sacraficially pause while their opponent continues. Even if it were the right thing to do, it seems politically non-viable, and it isn't remotely necessary as a step towards building a treaty, it may actually make us less likely to succeed by seeming to present our adversaries with an opportunity to catch up and emboldening them to compete.

Slight variation: If China knew that you weren't willing to punish competition with competition, that eliminates their incentive to work toward cooperation!

destroying current western robotics industry in the same way that the West's small kitchen appliance industry was utterly crushed. (70%)

I've heard that the US is already ahead on advanced — IE, fully automated — manufacturing. China's manufacturing economy depends on cheap human labor, which is an advantage they seem to be losing for some reason. I don't see much of a reason to think there's going to be a continuity between Chinese dominance in the pre-robotics manufacturing era and the next manufacturing era.

Funnily enough, Nvidia's recent 340B parameter chat assistant release did boast about being number one on the reward model leaderboard, however, the reward model only claims to capture helpfulness and a bunch of other metrics of usefulness to the individual user. But that's still pretty good.

Theory: the existence of the GreaterWrong lesswrong mirror is actually protecting everyone from the evil eye by generating google search results that sound like they're going to give you The Dirt on something (the name "Greater Wrong" vibes like it's going to be a hate site/controversy wiki) when really they just give you the earnest writings, meaning that the many searchers who're looking for controversy about a person or topic will instead receive (and probably boost the rankings of) evenhanded discussion.

And so I kind of wonder if the reason Elon is being so dismissive of that is that he's decided that's a moral standard he can't compete with and so he has to downplay it, claim, absurdly, that they're not really doing it. X.AI and Tesla's business model is all about training on user data without negotiating for it. I guess a less cynical take would be that he just can't believe the commitment is going to be stuck to, because access to massive training sets may be seen as critical to the near term success of all of these products. Right now, none them work all that well, if the users data-unionize (or if the EU makes Apple-style hardware privacy mandatory) and make it much more expensive to do ever larger training runs, growth may reverse. If he's right, it might not be possible for Apple to convince OpenAI to move their inference onto PCC.

It's really heartening to see Apple taking the initiative and making themselves accountable to third party audits of the hardware security properties of their inference machines. That's always seemed like an obvious step, consumers care about this stuff, win on trust, win consumers.
They're going to try to negotiate OpenAI into running their inference on Apple's PCC. And if they manage that, they're going to get consumers to recognize and celebrate it, and that really could raise expectations, it could even garner political interest.

where $nlo{g}_2$ denotes logarithm base $\frac{1}{2}$, aka the negative of the (base 2) logarithm

I've never heard of $lo{g}_{1/2}$ before and I'm having difficulty looking it up. Is it distinct from $2^n$, or $-lo{g}_2\left(n\right)$?

Edit: No, it's equivalent to $-log$, but this is only confirmed later in the post. Okay, where does this obscuritan "$nlog$" notation come from?

I think yall will be okay if you make sure your twitter account isn't your primary social existence, and you don't have to play twitter the usual way. Write longform stuff. Retweet old stuff. Be reasonable and conciliatory while your opponents are being unreasonable and nasty, that's how you actually win.

Remember that the people who've fallen in deep and contracted twitter narcissism are actually insane, It's not an adaptive behavior, they're there to lose. Every day they're embarrassing themselves and alienating people and all you have to do is hang around, occasionally point it out, and be the reasonable alternative.

Remember that you were only proposing discreet auditing systems to mollify the elves. They think of this as a privacy-preserving technology, because it is one, and that's largely what we're using it for.

Though it's also going to cause tremendous decreases in transaction costs by allowing ledger state to be validated without requiring the validator to store a lot of data or replay ledger history. If most crypto investors could foresee how it's going to make it harder to take rent on ledger systems, they might not be so happy about it.

Oh! ""10x" faster than RISC Zero"! We're down to a 1000x slowdown then! Yay!

Previous coverage btw.

However, one big challenge of building this is how two adversarial parties could ever gain enough confidence to allow such a hardware/software package into a secure facility, especially if it's whole point is to have a communication channel to their adversary.

Isn't it enough to constrain the output format to so that no steganographic leaks would be possible? Wont the counterparty usually be satisfied just with an hourly signal saying either "Something is wrong" (encompassing "Auditor saw a violation" / "no signal, the host has censored the auditor's report" / "invalid signal, the host has tampered with the auditor system" / "auditor has been blinded to the host's operations, or has ascertained that there are operations which the auditor cannot see") or "Auditor confirms that all systems are nominal and without violation."?

The host can remain in control of their facilities, as long as the auditor is running on tamperproof hardware. It's difficult to prove that a physical device can't be tampered with, it may be possible to take some components of the auditor even further and run them in a zero knowledge virtual machine, which provides a cryptographic guarantee that the program wasn't tampered with, so long as you can make it lithe enough to fit (zero knowledge virtual machines currently run at a 10,000x slowdown, though I don't think specialized hardware for them is available yet, crypto may drive that work), though a ZKVM wont provide a guarantee that the inputs to the system aren't being controlled, the auditor is monitoring inputs of such complexity — either footage of the real world or logs of a large training run — that it may be able to prove algorithmically to itself that the sensory inputs weren't tampered with either and the algorithm does have a view into the real world (I'm contending that even large state actors could not create Descarte's evil demon).

AWS probably wouldn't do anything, assuming a human has signed their name to the account, there might be policy against doing anything. But if you called the NSA/cybercom instead, they probably would do something given that they've been dealing with botnets and foreign hackers for decades.

I guess in this case I'm arguing that it's accidentally, accidentally, productive.

Agreed but initially downvoted due to being obviously unproductive, but then upvoted for being an exquisite proof by absurdity about what's productive: This is the first time I have seen clearly how good communication must forbid some amount of nuance.

The insight: You have a limited amount of time to communicate arguments and models; methods for reproducing some of your beliefs. With most people, you will never have enough time to transmit our entire technoeschatology or xenoeconomics stuff. It is useless to make claims about it, as the recipient has no way of checking them for errors or deceptions. You can only communicate approximations and submodules. No one will ever see the whole truth. (You do not see the whole truth. Your organization, even just within itself, will never agree about the whole truth.)

We are not investing in grass-roots advocacy, protests, demonstrations, and so on.

I like this, I'd be really interested to ask you, given that you're taking a first principles no bullshit approach to outreach, what do you think of protest in general?

Every protest I've witnessed seemed to be designed to annoy and alienate its witnesses, making it as clear as possible that there was no way to talk to these people, that their minds were on rails. I think most people recognize that as cult shit and are alienated by that.

A leftist friend once argued that protest is not really a means, but a reward, a sort of party for those who contributed to local movementbuilding. I liked that view. Perhaps we should frame our public gatherings to be closer to being that. If there is to be chanting of slogans, it must be an organic ebullition of the spirit of a group that was formed around some more productive purpose than that, maybe the purpose of building inclusive networks for shared moral purpose? (EA but broader?)

Hmm but I think it'll be solved like 5 years from now so I'd be eager to start working on VR boardgames/social role playing games today. I believe jon blow, when he says it usually takes that long to figure out what a really fresh kind of game wants to be.

And the first good VR RPGs are going to be really culturally impactful.

but there's also more design language built around 2D UIs. I still think there's a ton of unexplored design space around "tabletop games" that make use of modern web flows

Oh? I guess asynchrony is one of the things in that design language. And a web based game could leave players in a groupchat/forum after the game, they could make friends there. I used to play Neptune's Pride, which I guess is an example of that kind of game. It was a... good game... I think. It was emotionally brutal.

Actually, that experience with Neptune's Pride is probably a large part of the reason I want to make cohabitive games today. It was a game that forced you to forge friendships that were all absolutely destined to collapse. I forget whether there was any benefit to coming second or third, but if there was it wouldn't have resonated with the narrative, it was a war of domination, the mechanics of the gameworld were such that anyone with an advantage would be able to grow their advantage until there was nothing left for anyone else (and there was no flourishing along the way, just war) so second or third wouldn't have really meant anything within the narrative of the game.
Honest negotiation wasn't possible, every message we sent was laced with deception, and often the opponent would pick up on that and not admit it and that would be another deception of their own. This is the norm in diplomacy games. And I guess I became aware of how ruinously that misrepresents the diplomatic games we're playing in the real world (at least, post WWII, it is a misrepresentation. Maybe when nationalism was more of a thing our game was really like this. But today global culture is getting everywhere.).

I guess there's a subcultural aspect in how comfortable people feel with declining to answer questions: In subcultures where people can (or know to) just say "don't worry about it" and people don't get offended by that, questions are free.

But for the most part I think whether people are happy or annoyed with being asked a question comes down to how many times they have been asked. Asked zero times → very happy, like summer rain. Asked 30 times → not so much, but idk maybe you can mitigate the annoyance by copy and pasting something/publishing a statement.

I had fun by ridiculing the entire situation, waiting it out while asking the doctors questions like:

• Can identical twin brothers transmit testicle cancer through sodomy?
• Can I keep my surgically removed ball? (For storing in a jar of formaldehyde)
• Does hair loss from chemotherapy proceed in the same stages as male pattern baldness?

If you were hoping we weren't going to want to know the answers to these questions, I gotta disappoint ya

I think your response mostly didn't make sense. "What's that like" imposes a greater burden on you by requiring you to explain something again and again, and bluntly, I'm only going to be interested in that if me knowing helps you in some way, and I don't see why it would, but maybe it would, I wouldn't know, so it makes sense for me to just express willingness to help if needed, and that starts with "If you need to talk to someone, I’m here for you".

I get frustrated with imaginary social scripts too, but that part of the script is fine.

I might be interested in doing that, but to clarify; the reason you need VR for board games is that VR facilitates the same kind of sense of shared presence and quality of audio conversation that playing in person does. I'm fairly sure VR boardgaming is going to be better than physical boardgaming (larger tables, no upkeep, more immersion, venues that can be teleported to from home so more repeat interactions so more legacy games are possible). That is the sense in which you need VR. But for prototyping, sure, developers have pre-existing buyin and can put up with the limitations of 2d.

That's interesting thanks, but I hope you can understand how keeping all of the individual goals secret would make it much harder to practice negotiation. It's okay (great, even) if there's some way of exposing the secret goals. In most games with secret goals that doesn't happen during the game, but since, iirc, it's a legacy game, maybe players tend to figure out each others' secret goals as the campaign goes on. Is that the case? If so, I'd be very interested in seeing that stuff, and the late-game.

(only including downstream effects of your policy)

I'm not sure I know what you mean by this, but if you mean causal effects, no, it considers all pasts, and all timelines.

(A reader might balk, "but that's computationally infeasible", but we're talking about mathematic idealizations, the mathematical idealization of CDT is also computationally infeasible. Once we're talking about serious engineering projects to make implementable approximations of these things, you don't know what's going to be feasible.)

I don't know how to convince anyone of this but just having low network latency to California may be worth a lot after VR gets good. Physical access will matter much less than the quality of your network connection, the delay between speaking and seeing and hearing others' reactions, and timezone overlap.

I'm not sure how to put a price on these things. People seem to be able to adjust to the delay in conversation, but adjusting requires becoming comfortable with talking over people sometimes, it can get noisy. The timezone overlap issue also seems important, if you don't get off work/have your mid day break at the same time as others, you get left out of things, but people can adjust their sleep/wake time (I certainly can) and synch with a remote timezone so idk.

I note that in the cooperative bargaining domain, a CDT agent will engage in commitment races, using the commitment mechanism to turn itself into a berzerker, a threatmaker. If they're sharing a world with other CDT agents, that is all they will do. Whoever's able to constitutionalize first will make a pre-commitment like "I'll initiate a nuclear apocalypse if you don't surrender all of your land to us."

If they're sharing the world with UDT agents, they will be able to ascertain that those sorts of threats will be ignored (reflected in the US's principle of "refusing to negotiate with terrorists"), and recognize that it would just lead to MAD with no chance of a surrender deal. I think commitment mechanisms only lead to good bargaining outcomes if UDT agents already hold a lot of power.

A general pre-commitment mechanism is just self-modification. CDT with self-modification has been named "Son of CDT", and seems to have been discussed most recently on this arbital article.

It behaves like a UDT agent about everything after the modifications are made, but not about anything that was determined before then.
I'm not aware of any exploits for that. I suspect that there will be some.

It seems like the anki for notetaking is https://www.remnote.com/ ? Suggested here https://www.lesswrong.com/posts/CoqFpaorNHsWxRzvz/what-comes-after-roam-s-renaissance?commentId=9LY2cTTAKbaJsBCNf

I think the whole roam fad may have been an example of a phenomenon I'm noticing where these fundamentally social apes that we are cannot conceive of an activity failing to become social, the idea that what I see isn't what others see, or the idea that there are activities that can't be shared, is unnatural and unintuitive to us. Roam would have made a lot of sense as a new kind of social network oriented around collaboratively, iteratively building evergreen knowledge together, and it vibed like it was that, and I think that's why there was so much excitement, but due to the way Roam works and the shoddy way it was built, it actually couldn't grow into that. The human sense for social fun isn't smart enough to notice that it wasn't going to happen and so it took a long time to wind down.

Specifically, multiplayer roam has no notification system, and it's siloed into groups. They don't have per-block/per-page read/write permissions, meaning that you kind of have to be sharing all of the notes in a graph or none of them, and as far as I'm aware there are no communities that makes sense for. Most peoples' notes end up being mostly private, rough or intimate.

But massively multiplayer roam as a concept is totally possible, it just has to be built in a different way, with different systems.
It's one of the things I want to build.

From what it sounds like, the Roam community would never stand for that

Not sure exactly what's meant by roam community but Subconscious are thinking about that stuff. I forget what exactly they're doing with it, I haven't been paying attention to it (maybe I'm the roam community), but I think it was something like... it goes and surfaces something from the past to remind you of it to make sure things you might have forgotten about get linked together. (Gosh that would be bad for my productivity.)

At least for me—and most of the people I know—we got a garbage dump full of crufty links and pieces of text we hardly ever revisit.

For me, some stuff fell into disrepair when I realized no one else was going to read it or add to it, which I think is mostly due to a bad model for shared use, but a lot of other stuff kept growing and turned out to be really transformative. I've continued using it as a very good, very long todo list, a pensieve to relinquish and defer burdensome thoughts into, to sort every possible idea I could be pursuing and to prioritize the one currently most important thing.

Some other pages that've remained active over the years:

• recipes
• tasteweb engineering notes, pitch concepts
• UFO stuff (some interesting evidence, but mostly debunkings). I guess this could add up to a post eventually.
• "people", a list of various things to talk about with people next time we're in a room together, if we ever are. The uncertainty, that we may not meet, is the reason I think it's important to externalize these thoughts.

while Subconscious has taken the blockchain/protocol approach.

Subconscious isn't blockchain. The noosphere protocol has the foundational features of a smart contract system, but it doesn't seem quite secure enough against inconsistencies/double-spends to run finance (and that's not an aspiration they seem to have), and the sacrifices it makes will make it cheap and easy to scale horizontally/federate. The protocol it's most similar to is bluesky's ATProtocol. I actually wanted to complain to them that they're too similar and they should merge, but noosphere started developing before ATProtocol existed, so who can blame them really.

Although a part of me does wonder, if a protocol like atproto took off, whether people would start doing finance on it, security be damned, and then legal enforcement and auditing would come along and it would be effectively secured through international social technologies and trusted computing instead of cryptographic technologies and game theory and we might just end up in the same place. It would be a weird future.

That isn't anyone's first/preferred plan. I assure you everyone born in a liberal democracy has considered another plan before arriving at that one.

I'm not sure why people would think LLMs understand their own output, we know they're not up to spotting sometimes human-obvious inconsistencies in it (as soon as they are, things will start moving very quickly).

:( that isn't what cooperation would look like. The gazelles can reject a deal that would lead to their extinction (they have better alternatives) and impose a deal that would benefit both species.

Cooperation isn't purely submissive compliance.