LessWrong 2.0 Reader

Commenting on the basis of lessons from some experience doing UBI analysis for Switzerland/Europe:

The current systems has various costs (time and money, but maybe more importantly, opportunities wasted by perverse incentives) associated with proving that you are eligible for some benefit.

On the one hand, yes, and its a key reason why NIT/UBI systems are often popular on the right; even Milton Friedman already advocated for a NIT. That said, there are also discussions that suggest the poverty trap - i.e. overwhelmingly strong labor disincentives for poor, from outrageously high effective marginal tax rates from benefits fade-out/tax kicking-in - may be partly overrated, so smoothing the earned-to-net income function may not help as much as some may hope. And, what tends to be forgotten, is that people with special needs may not be able to live purely from a UBI, so not all current social security benefit mechanisms can usually be replaced by a standard UBI.

On the other hand, once you have a conditional welfare system that does not have crazily strong/large poverty traps, labor incentives might overall still be mostly stronger than under a UBI (assumed sufficiently generous to allow a reasonable life from it), once you also take into account the high marginal tax rates required to finance that UBI. This seems to hold in even relatively rich countries (we used to calculate it for Switzerland).

Of course, with AI Joblessness all this might change anyway, in line with the underlying topic of the post here.

Plus you need to pay the people who verify all this evidence.

This tends to be overrated; when you look at the stats, this staff cost is really small compared to the total traditional social security or the UBI costs (we looked at #s in Switzerland but I can only imagine it's exactly similar orders of magnitudes in other developed countries).

Might be mainly driven by an improved tokenizer.

I would be shocked if this is the main driver, they claim that English only has 1.1x fewer tokens, but seem to claim much bigger speed-ups

I think this is a useful model. If I understand correctly what you're saying, then it is that for any particular thing we can think about whether that thing is optimal to do, and whether I could get this thing to work seperately.

I think what I was saying is different. I was advocating confidence not at the object level of some concrete things you might do. Rather I think being confident in the overall process that you engage in to make process is a thing that you can have confidence in.

Imagine there is a really good researcher, but now this person forgets everything that they ever researched, except for their methodology. It some sense they still know how to do research. If they fill in some basic factual knowledge in their brain, which I expect wouldn't take that long, I expect they would be able to continue being an effective researcher.

Wildlife Welfare Will Win

The long arc of history bend towards gentleness and compassion. Future generations will look with horror on factory farming. And already young people are following this moral thread to its logical conclusion; turning their eyes in disgust to mother nature, red in tooth and claw. Wildlife Welfare Done Right, compassion towards our pets followed to its forceful conclusion would entail the forced uploading of all higher animals, and judging by the memetic virulences of shrimp welfare to lower animals as well. 

Morality-upon-reflexion may very well converge on a simple form of pain-pleasure utilitarianism. 

There are few caveats: future society is not dominated, controlled and designed by a singleton AI-supervised state, technology inevitable stalls and that invisible hand performs its inexorable logic for the eons and an Malthuso-Hansonian world will emerge once again - the industrial revolution but a short blip of cornucopia. 

Perhaps a theory of consciousness is discovered and proves once and for all homo sapiens and only homo sapiens are conscious ( to a significant degree). Perhaps society will wirehead itself into blissful oblivion.  Or perhaps a superior machine intelligence arises, one whose final telos is the whole of and nothing but office supplies. Or perhaps stranger things still happen and the astronomo-cosmic compute of our cosmic endowment is engaged for mysterious purposes. Arise, self-made god of pancosmos. Thy name is UDASSA. 

In Magna Alta Doctrina [LW · GW] Jacob Cannell talks about exponential gradient descent as a way of approximating solomonoff induction using ANNs

While that approach is potentially interesting by itself, it's probably better to stay within the real algebra. The Solmonoff style partial continuous update for real-valued weights would then correspond to a multiplicative weight update rather than an additive weight update as in standard SGD.

Has this been tried/evaluated? Why actually yes - it's called exponentiated gradient descent, as exponentiating the result of additive updates is equivalent to multiplicative updates. And intriguingly, for certain 'sparse' input distributions the convergence or total error of EGD/MGD is logarithmic rather than the typical inverse polynomial of AGD (additive gradient descent): O(logN) vs O(1/N) or O(1/N2), and fits naturally with 'throw away half the theories per observation'.

The situations where EGD outperforms AGD, or vice versa, depend on the input distribution: if it's more normal then AGD wins, if it's more sparse log-normal then EGD wins. The morale of the story is there isn't one single simple update rule that always maximizes convergence/performance; it all depends on the data distribution (a key insight from bayesian analysis).

The exponential/multiplicative update is correct in Solomonoff's use case because the different sub-models are strictly competing rather than cooperating: we assume a single correct theory can explain the data, and predict through an ensemble of sub-models. But we should expect that learned cooperation is also important - and more specifically if you look more deeply down the layers of a deeply factored net at where nodes representing sub-computations are more heavily shared, it perhaps argues for cooperative components.

My read of this is we get a criterion for when one should be a hedgehog versus a fox in forecasting: One should be a fox when the distributions you need to operate in are normal, or rather when it does not have long tails, and you should be a hedgehog when your input distribution is more log-normal, or rather when there may be long-tails.

This makes some sense. If you don't have many outliers, most theories should agree with each other, its hard to test & distinguish between the theories, and if one of your theories does make striking predictions far different from your other theories, its probably wrong, just because striking things don't really happen.

In contrast, if you need to regularly deal with extreme scenarios, you need theories capable of generalizing to those extreme scenarios, which means not throwing out theories for making striking or weird predictions. Striking events end up being common, so its less an indictment.

But there are also reasons to think this is wrong. Hits based entrepreneurship approaches for example seem to be more foxy than standard quantitative or investment finance, and hits based entrepreneurship works precisely because the distribution of outcomes for companies is long-tailed.

In some sense the difference between the two is a "sin of omission" vs a "sin of commission" disagreement between the two approaches, where the hits-based approach needs to see how something could go right, while the standard finance approach needs to see how something could go wrong. So its not so much a predictive disagreement between the two approaches, but more a decision theory or comparative advantage difference.

This exact thought, from my diary in ~June 2022: "I advocate keeping a clear separation between how confident you are that your plan will work with how confident you are that pursuing the plan is optimal."

I think perhaps I don't fully advocate alieving that your plan is more likely to work than you actually believe it is. Or, at least, I advocate some of that on the margin, but mostly I just advocate keeping a clear separation between how confident you are that your plan will work with how confident you are that pursuing the plan is optimal. As long as you tune yourself to be inspired by working on the optimal, you can be more ambitious and less risk-averse.

Unfortunately, if you look like you're confidently pursuing a plan (because you think it's optimal, but your reasons are not immediately observable), other people will often mistake that for confidence-in-results and perhaps conclude that you're epistemically crazy. So it's nearly always socially safer to tune yourself to confidence-in-results lest you risk being misunderstood and laughed at.

You also don't want to be so confident that what you're doing is optimal that you're unable to change your path when new evidence comes in. Your first plan is unlikely to be the best you can do, and you can only find the best you can do by trying many different things and iterating. Confidence can be an obstacle to change.

On the other hand, lack of confidence can also be an obstacle to change. If you're not confident that you can do better than you're currently doing, then you'll have a hard time motivating yourself to find alternatives. Underconfidence is probably underappreciated as a source of bias due to social humility being such a virtue.

To use myself as an example (although I didn't intend for this to be about me rather than about my general take on mindset): I feel pretty good about the ideas I've come up with so far. So now I have a choice to make: 1) I could think that the ideas are so good that I should just focus on building and clarifying them, or 2) I could use the ideas as evidence that I'm able to produce even better ideas if I keep searching. I'm aiming for the latter, and I hold my current best ideas in contempt because I'm still stuck with them. In some sense, confidence makes it easier to Actually Change My Mind.

I guess the recipe I might be advocating is:
1. separate between confidence-in-results and confidence-in-optimality
2. try to hold accurate and precise beliefs about the results of your plans/ideas, mistakes here are costly
3. try to alieve that you're able to produce more optimal ideas/plans than the ones you already have, mistakes here are less costly, and gains in positive alief are much higher

I'm going to call this the Way of Aisi because it reminds me of an old friend who just did everything better than everyone else (including himself) because he had faith in himself. :p

This has been a disagreement people have had for many years. Why expect it to come to a head this year?

Safety-wise, they claim to have run it through their Preparedness framework and the red-team of external experts.

I'm disappointed and I think they shouldn't get much credit PF-wise: they haven't published their evals, published a report on results, or even published a high-level "scorecard." They are not yet meeting the commitments in their beta Preparedness Framework — some stuff is unclear but at the least publishing the scorecard is a commitment.

@nostalgebraist [LW · GW] bites that bullet here:

“Reinforcement learning” (RL) is not a technique.  It’s a problem statement, i.e. a way of framing a task as an optimization problem, so you can hand it over to a mechanical optimizer.

What’s more, even calling it a problem statement is misleading,  because it’s (almost) the most general problem statement possible for any arbitrary task.  If you try to formalize a concept like “doing a task well,“ or even “being an entity that acts freely and wants things,” in the most generic terms with no constraints whatsoever, you end up writing down “reinforcement learning.”

and so does Russell & Norvig 3rd edition

Reinforcement learning might be considered to encompass all of AI.

…That’s not my perspective though.

For my part, there’s a stereotypical core of things-I-call-RL which entails:

• (A) There’s a notion of the AI’s outputs being better or worse
• (B) But we don’t have ground truth (even after-the-fact) about what any particular output should ideally have been
• (C) Therefore the system needs to do some kind of explore-exploit.

By this (loose) definition, both model-based and model-free RL are central examples of “reinforcement learning”, whereas LLM self-supervised base models are not reinforcement learning (cf. (B), (C)), nor are ConvNet classifiers trained on ImageNet (ditto), nor are clustering algorithms (cf. (A), (C)), nor is A* or any other exhaustive search within a simple deterministic domain (cf. (C)), nor are VAEs (cf. (C)), etc.

(A,B,C) is generally the situation you face if you want an AI to win at videogames or board games, control bodies while adapting to unpredictable injuries or terrain, write down math proofs, design chips, found companies, and so on.

This (loose) definition of RL connects to AGI safety because (B-C) makes it harder to predict the outputs of an RL system. E.g. we can plausibly guess that an LLM base model, given internet-text-like prompts, will continue in an internet-text-typical way. Granted, given OOD prompts, it’s harder to say things a priori about the output. But that’s nothing compared to e.g. AlphaZero or AlphaStar, where we’re almost completely in the dark about what the trained model will do in any nontrivial game-state whatsoever. (…Then extrapolate the latter to human-level AGIs acting in the real world!)

(That’s not an argument that “we’re doomed if AGI is based on RL”, but I do think that a very RL-centric AGI would need tailored approaches to thinking about safety and alignment that wouldn’t apply to LLMs; and I likewise think that likewise a massive increase in the scope and centrality of LLM-related RL (beyond the RLHF status quo) would raise new (and concerning) alignment issues, different from the ones we’re used to with LLMs today.)

highly convergent

Huh? A hyperphone is a two-player tool. Loom is a one-player tool.