LessWrong 2.0 Reader

Collapse of mega-project to create AI based on linguistics

ABBYY spent 100 million USD for 30 years to create a model of language using hundreds of linguists. It fails to compete with transformers. This month the project was closed. More in Russian here: https://sysblok.ru/blog/gorkij-urok-abbyy-kak-lingvisty-proigrali-poslednjuju-bitvu-za-nlp/

Our strategy is for variants to preserve well-defined behavior in the application but introduce diversity in the effect of undefined behavior (such as out-of-bounds accesses).

This Galois work is a lot narrower and targeted at low-level details irrelevant to most code, which thankfully is now written in non-C languages- where out-of-bounds accesses don't pwn your machine and undefined behavior does not summon nasal demons and stuff like ASLR is largely irrelevant.

So AI is wholly necessary for most of the value of such an idea.

And yeah, I think it's a pretty decent idea: with cheap enough LLMs, you can harden applications by sampling possible implementations which pass all unit-tests, and whose final combination pass all end-to-end or integration tests. You can already do this a bit to check things with LLMs being so cheap. (Last night, Achmiz asked a Markov chain question and I was too lazy to try to figure it out myself, so I had ChatGPT solve it 3 ways in R: Monte Carlo, solving the matrix, and proving an exact closed-form probability. The answer could be wrong but that seems unlikely when they all seem to agree. If I wanted to write it up, I'd also have Claude solve it independently in Python so I could cross-check all 6 versions...)

This would help avoid a decent number of logic bugs and oversights, and it would also have some benefits in terms of software engineering: you are getting a lot of automated 'chaos engineering' and unit-test generation and performance benchmarking for free, by distributing a combinatorial number of implementations. It's almost like a mass fuzzing exercise, where the users provide the fuzz.

You might think this would run into issues with tracking the combinatorial number of binaries, which could take up petabytes if you are distributing, say, a 1GB package to 1 million users, but this has plenty of possible fixes: if you are using reproducible builds, as you ought to, then you only need to track a list of the variants for each function and store that per user, and then you can rebuild the exact binary for a given user on-demand. I think a bigger issue is that forcing diversity out of tuned LLMs is quite hard, and so you would run into the systematic error problem at a higher level: all the tuned LLMs, feeding on each others' outputs & mode-collapsed, will turn in code with the same implicit assumptions & algorithms & bugs, which would mostly defeat the point.

Basic Q: has anyone written much down about what sorts of endgame strategies you'd see just-before-ASI from the perspective of "it's about to go well, and we want to maximize the benefits of it" ?

For example: if we saw OpenPhil suddenly make a massive push to just mitigate mortality at the cost of literally every other development goal they have, I might suspect that they suspect that we're about to all be immortal under ASI, and they're trying to get as many people possible to that future... 

yup, as @sanxiyn [LW · GW] says, this already exists.  Their example is, AIUI, a high-end research one; an actually-on-your-laptop-right-now, but admittedly more narrow example is address space layout randomization.   

Before power grid dissolves, it has also to hit factories (and business in general). I don't think resulting price increments are predictable - they might as well start some crisis in economy. (And there might be money-unrelated outcomes, like if restaurants start ignoring some safety standards trying to save on electricity or like, which could be disastrous with habit of eating out.)

We can create a vacuum airship if we make its hull based on rigid ribs with fabric stretched between them. I heard that so-called black triangle UFOs could be military blimps based on such technology. 

The whole article is about Amazon employees being on the clock while they are using the bathroom. Spending more time in the bathroom reduces the productivity/per hour on their KPIs and thus they are incentivized against spending time in the bathroom.

Getting down-voted to -27 is an achievement. Most things judged 'bad AI takes' only go to -11 or so, even that recent  P=NP proof only got to -25. Of course, if the author is right, then downvoting further is providing helpful incentives to him. 

I think that bullying is quite distinct from status hierarchies. The latter are unavoidable. There will always be some clique of cool kids in the class who will not invite the non-cool kids to their parties. This is ok. Sometimes, status is correlated with behaviors which are pro-social (kids not smoking; donating to EA), sometimes it is correlated with behaviors which are net-negative (kids smoking; serving in the SS). I was not part of the cool kids circle, and I was fine with that. Live and let live and all that. 

'Bullying' has a distinct negative connotation. The central example is someone who is targeted for sport for being different from the others. The bullies don't want the victims to change their ways, they just like to make their life miserable for thrills. I am sure that sometimes it unintentionally helps their victim, if you push enough people, at some point you are bound to push someone out of the path of a car or bullet. In the grand scheme of things, the bullies are however net negative for their victims and society overall. 

I see this as less of an endorsement of linear models and more of a scathing review of expert performance. 

• When an arithmetic model is calibrated, it is specifically by including feedback from the real-world effects of its predictions. Experts do not, as a rule, seek out any feedback on their calibration.

This. Basically, if your job is to do predictions, and the accuracy of your predictions is not measured, then (at least the prediction part of) your job is bullshit. 

I think that if you compare simple linear models in domains where people actually care about their predictions, the outcome would be different. For example, if simple models predicted stock performance better than experts at investment banks, anyone with a spreadsheet could quickly become rich. There are few if any cases of 'I started with Excel and 1000$, and now I am a billionaire'. Likewise, I would be highly surprised to see a simple linear model outperform Nate Silver or the weather forecast. 

Even predicting chess outcomes from mid-game board configurations is something where I would expect human experts to outperform simple statistical models working on easily quantifiable data (e.g. number of pieces remaining, number of possible moves, being in check, etc).

Neural networks contained in animal brains (which includes human brains) are quite capable of implementing linear models, and such should at least perform equally well when they are properly trained. A wolf pack deciding to chase or not chase some prey has direct evolutionary skin in the game of making their prediction of success as accurate as possible which the average school counselor predicting academic success simply does not have.

--

You touch this a bit in 'In defense of explainatory modeling', but I want to emphasize that uncovering causal relationships and pathways is central to world modelling. Often, we don't want just predictions, we want predictions conditional on interventions. If you don't have that, you will end up trying to cure chickenpox with makeup, as 'visible blisters' is negatively correlated with outcomes. 

Likewise, if we know the causal pathway, we have a much better basis to judge if some finding can be applied to out-of-distribution data. No matter how many anvils you have seen falling, without a causal understanding (e.g. Newtonian mechanics), you will not be able to reliably apply your findings to falling apples or pianos. 
 

But the Summoned Heroine doesn't know that until the end, and it's stated that she specifically set up the market to "help them anticipate and counter the Demon King’s next moves".