Posts
Comments
Interesting...
Wouldn't I expect the evidence to come out in a few big chunks, e.g. OpenAI releasing a new product?
I agree with you.
Epsilon machine (and MSP) construction is most likely computationally intractable [I don't know an exact statement of such a result in the literature but I suspect it is true] for realistic scenarios.
Scaling an approximate version of epsilon reconstruction seems therefore of prime importance. Real world architectures and data has highly specific structure & symmetry that makes it different from completely generic HMMs. This must most likely be exploited.
The calculi of emergence paper has inspired many people but has not been developed much. Many of the details are somewhat obscure, vague. I also believe that most likely completely different methods are needed to push the program further. Computational Mechanics' is primarily a theory of hidden markov models - it doesn't have the tools to easily describe behaviour higher up the Chomsky hierarchy. I suspect more powerful and sophisticated algebraic, logical and categorical thinking will be needed here. I caveat this by saying that Paul Riechers has pointed out that actually one can understand all these gadgets up the Chomsky hierarchy as infinite HMMs which may be analyzed usefully just as finite HMMs.
The still-underdeveloped theory of epsilon transducers I regard as the most promising lens on agent foundations. This is uncharcted territory; I suspect the largest impact of computational mechanics will come from this direction.
Your point on True Names is well-taken. More basic examples than gauge information, synchronization order are the triple of quantites entropy rate , excess entropy and Crutchfield's statistical/forecasting complexity . These are the most important quantities to understand for any stochastic process (such as the structure of language and LLMs!)
Non exhaustive list of reasons one could be interested in computational mechanics: https://www.lesswrong.com/posts/GG2NFdgtxxjEssyiE/dalcy-s-shortform?commentId=DdnaLZmJwusPkGn96
I agree with you that the new/surprising thing is the linearity of the probe. Also I agree that not entirely clear how surprising & new linearity of the probe is.
If you understand how the causal states construction & the MSP works in computational mechanics the experimental results isn't surprising. Indeed, it can't be any other way! That's exactly the magic of the definition of causal states.
What one person might find surprising or new another thinks trivial. The subtle magic of the right theoretical framework is that it makes the complex simple, surprising phenomena apparent.
Before learning about causal states I would have not even considered that there is a unique (!) optimal minimal predictor canonical constructible from the data. Nor that the geometry of synchronizing belief states is generically a fractal. Of course, once one has properly internalized the definitions this is almost immediate. Pretty pictures can be helpful in building that intuition !
Adam and I (and many others) have been preaching the gospel of computational mechanics for a while now. Most of it has fallen on deaf ears before. Like you I have been (positively!) surprised and amused by the sudden outpouring of interest. No doubt it's in part a the testimony to the Power of the Visual! Never look a gift horse in the mouth ! _
I would say the parts of computational mechanics I am really excited are a little deeper - downstream of causal states & the MSP. This is just a taster.
I'm confused & intrigued by your insistence that this is follows from the good regulator theorem. Like Adam I don't understand it. It is my understanding is that the original 'theorem' was wordcelled nonsense but that John has been able to formulate a nontrivial version of the theorem. My experience is that it the theorem is often invoked in a handwavey way that leaves me no less confused than before. No doubt due to my own ignorance !
I would be curious to hear a *precise * statement why the result here follows from the Good Regular Theorem.
Military nerds correct me if I'm wrong but I think the answer might be the following. I'm not a pilot etc etc.
Stealth can be a bit of a misleading term. F35 aren't actually 'stealth aircraft' - they are low-observable aircraft. You can detect F35s with longwave radar.
The problem isn't knowing that there is a F35 but to get a weapon -grade lock on it. This is much harder and your grainy gpt-interpreted photo isn't close to enough for a missile I think. You mentioned this already as a possibility.
The Ukrainians pioneered something similar for audio which is used to detect missiles & drones entering Ukrainian airspace.
It also suggests that there might some sort of conservation law for pain for agents.
Conservation of Pain if you will
ingular Sure! I'll try and say some relevant things below. In general, I suggest looking at Liam Carroll's distillation over Watanabe's book (which is quite heavy going, but good as a reference text). There are also some links below that may prove helpful.
The empirical loss and its second derivative are statistical estimator of the population loss and its second derivative. Ultimately the latter controls the properties of the former (though the relation between the second derivative of the empirical loss and the second derivative of the population loss is a little subtle).
The [matrix of] second derivatives of the population loss at the minima is called the Fischer information metric. It's always degenerate [i.e. singular] for any statistical model with hidden states or hierarchichal structure. Analyses that don't take this into account are inherently flawed.
SLT tells us that the local geometry around the minimum nevertheless controls the learning and generalization behaviour of any Bayesian learner for large N. N doesn't have to be that large though, empirically the asymptotic behaviour that SLT predicts is already hit for N=200.
In some sense, SLT says that the broad basin intuition is broadly correct but this needs to be heavily caveated. Our low-dimensional intuition for broad basin is misleading. For singular statistical models (again everything used in ML is highly singular) the local geometry around the minima in high dimensions is very weird.
Maybe you've heard of the behaviour of the volume of a sphere in high dimensions: most of it is contained on the shell. I like to think of the local geometry as some sort of fractal sea urchin. Maybe you like that picture, maybe you don't but it doesn't matter. SLT gives actual math that is provably the right thing for a Bayesian learner.
[real ML practice isn't Bayesian learning though? Yes, this is true. Nevertheless, there is both empirical and mathematical evidence that the Bayesian quantitites are still highly relevant for actual learning]
SLT says that the Bayesian posterior is controlled by the local geometry of the minimum. The dominant factor for N~>= 200 is the fractal dimension of the minimum. This is the RLCT and it is the most important quantity of SLT.
There are some misconception about the RLCT floating around. One way to think about is as an 'effective fractal dimension' but one has to be careful about this. There is a notion of effective dimension in the standard ML literature where one takes the parameter count and mods out parameters that don't do anything (because of symmetries). The RLCT picks up on symmetries but it is not just that. It picks up on how degenerate directions in the fischer information metric are ~= how broad is the basin in that direction.
Let's consider a maximally simple example to get some intuition. Let the population loss function be . The number of parameters and the minimum is at .
For the minimum is nondegenerate (the second derivative is nonzero). In this case the RLCT is half the dimension. In our case the dimension is just so
For the minimum is degenerate (the second derivative is zero). Analyses based on studying the second derivatives will not see the difference between but in fact the local geometry is vastly different. The higher is the broader the basin around the minimum. The RLCT for is . This means, the is lower the 'broader' the basin is.
Okay so far this only recapitulates the broad basin story. But there are some important points
- this is an actual quantity that can be estimated at scale for real networks that provably dominates the learning behaviour for moderately large .
- SLT says that the minima with low rlct will be preferred. It evens says how much they will be preferred. There is tradeoff between lower rlct minima with moderate loss ('simpler solutions') and minima with higher rlct but lower loss. As This means that the RLCT is actually 'the right notion of model complexity/ simplicty' in the parameterized Bayesian setting. This is too much to recap in this comment but I refer you to Hoogland & van Wingerden's post here. This is the also the start of the phase transition story which I regard as the principal insight of SLT.
- The RLCT doesn't just pick up on basin broadness. It also picks up on more elaborate singular structure. E.g. a crossing valley type minimum like . I won't tell you the answer but you can calculate it yourself using Shaowei Lin's cheat sheet. This is key - actual neural networks have highly highly singular structure that determines the RLCT.
- The RLCT is the most important quantity in SLT but SLT is not just about the RLCT. For instance, the second most important quantity the 'singular fluctuation' is also quite important. It has a strong influence on generaliztion behaviour and is the largest factor in the variance of trained models. It controls approximation to Bayesian learning like the way neural networks are trained.
- We've seen that the directions defined by the matrix of second derivatives is fundamentally flawed because neural networks are highly singular. Still, there is something noncrazy about studying these directions. There is upcoming work which I can't discuss in detail yet that explains to large degree how to correct this naive picture both mathematically and empirically.
This is all answered very elegantly by singular learning theory.
You seem to have a strong math background! I really encourage you take the time and really study the details of SLT. :-)
I would not say that the central insight of SLT is about priors. Under weak conditions the prior is almost irrelevant. Indeed, the RLCT is independent of the prior under very weak nonvanishing conditions.
The story that symmetries mean that the parameter-to-function map is not injective is true but already well-understood outside of SLT. It is a common misconception that this is what SLT amounts to.
To be sure - generic symmetries are seen by the RLCT. But these are, in some sense, the uninteresting ones. The interesting thing is the local singular structure and its unfolding in phase transitions during training.
The issue of the true distribution not being contained in the model is called 'unrealizability' in Bayesian statistics. It is dealt with in Watanabe's second 'green' book. Nonrealizability is key to the most important insight of SLT contained in the last sections of the second to last chapter of the green book: algorithmic development during training through phase transitions in the free energy.
I don't have the time to recap this story here.
All proofs are contained in the Watanabe's standard text, see here
Did I just say SLT is the Newtonian gravity of deep learning? Hubris of the highest order!
But also yes... I think I am saying that
- Singular Learning Theory is the first highly accurate model of breath of optima.
- SLT tells us to look at a quantity Watanabe calls , which has the highly-technical name 'real log canonical threshold (RLCT). He proves several equivalent ways to describe it one of which is as the (fractal) volume scaling dimension around the optima.
- By computing simple examples (see Shaowei's guide in the links below) you can check for yourself how the RLCT picks up on basin broadness.
- The RLCT = first-order term for in-distribution generalization error and also Bayesian learning (technically the 'Bayesian free energy'). This justifies the name of 'learning coefficient' for lambda. I emphasize that these are mathematically precise statements that have complete proofs, not conjectures or intuitions.
- Knowing a little SLT will inoculate you against many wrong theories of deep learning that abound in the literature. I won't be going in to it but suffice to say that any paper assuming that the Fischer information metric is regular for deep neural networks or any kind of hierarchichal structure is fundamentally flawed. And you can be sure this assumption is sneaked in all over the place. For instance, this is almost always the case when people talk about Laplace approximation.
- It's one of the most computationally applicable ones we have? Yes. SLT quantities like the RLCT can be analytically computed for many statistical models of interest, correctly predicts phase transitions in toy neural networks and it can be estimated at scale.
EDIT: no hype about future work. Wait and see ! :)
- Scott Garrabrant's discovery of Logical Inductors.
I remembered hearing about the paper from a friend and thinking it couldn't possibly be true in a non-trivial sense. To someone with even a modicum of experience in logic - a computable procedure assigning probabilities to arbitrary logical statements in a natural way is surely to hit a no-go diagonalization barrier.
Logical Inductors get around the diagonalization barrier in a very clever way. I won't spoil how it does here. I recommend the interested reader to watch Andrew's Critch talk on Logical Induction.
It was the main reason convincing that MIRI != clowns but were doing substantial research.
The Logical Induction paper has a fairly thorough discussion of previous work. Relevant previous work to mention is de Finetti's on betting and probability, previous work by MIRI & associates (Herreshof, Taylor, Christiano, Yudkowsky...), the work of Shafer-Vovk on financial interpretations of probability & Shafer's work on aggregation of experts. There is also a field which doesn't have a clear name that studies various forms of expert aggregation. Overall, my best judgement is that nobody else was close before Garrabrant.
- The Antikythera artifact: a Hellenistic Computer.
- You probably learned heliocentrism= good, geocentrism=bad, Copernicus-Kepler-Newton=good epicycles=bad. But geocentric models and heliocentric models are equivalent, it's just that Kepler & Newton's laws are best expressed in a heliocentric frame. However, the raw data of observations is actually made in a geocentric frame. Geocentric models stay closer to the data in some sense.
- Epicyclic theory is now considered bad, an example of people refusing to see the light of scientific revolution. But actually, it was an enormous innovation. Using high-precision gearing epicycles could be actually implemented on a (Hellenistic) computer implicitly doing Fourier analysis to predict the motion of the planets. Astounding.
- A Roman author (Pliny the Elder?) describes a similar device in posession of Archimedes of Rhodes. It seems likely that Archimedes or a close contemporary (s) designed the artifact and that several were made in Rhodes.
Actually, since we're on the subject of scientific discoveries
- Discovery & description of the complete Antikythera mechanism. The actual artifact that was found is just a rusty piece of bronze. Nobody knew how it worked. There were several sequential discoveries over multiple decades that eventually led to the complete solution of the mechanism.The final pieces were found just a few years ago. An astounding scientific achievement. Here is an amazing documentary on the subject:
Singular Learning Theory is another way of "talking about the breadth of optima" in the same sense that Newton's Universal Law of Gravitation is another way of "talking about Things Falling Down".
Don't forget Wallace !
Yes, beautiful example ! Van Leeuwenhoek was the one-man ASML of the 17th century. In this case, we actually have evidence to the counterfactual impact as other lensmakers trailed van Leeuwenhoek by many decades.
It's plausible that high-precision measurement and fabrication is the key bottleneck in most technological and scientific progress- it's difficult to oversell the importance of van Leeuwenhoek.
Antonie van Leeuwenhoek made more than 500 optical lenses. He also created at least 25 single-lens microscopes, of differing types, of which only nine have survived. These microscopes were made of silver or copper frames, holding hand-made lenses. Those that have survived are capable of magnification up to 275 times. It is suspected that Van Leeuwenhoek possessed some microscopes that could magnify up to 500 times. Although he has been widely regarded as a dilettante or amateur, his scientific research was of remarkably high quality.[39]
The single-lens microscopes of Van Leeuwenhoek were relatively small devices, the largest being about 5 cm long.[40][41] They are used by placing the lens very close in front of the eye. The other side of the microscope had a pin, where the sample was attached in order to stay close to the lens. There were also three screws to move the pin and the sample along three axes: one axis to change the focus, and the two other axes to navigate through the sample.
Van Leeuwenhoek maintained throughout his life that there are aspects of microscope construction "which I only keep for myself", in particular his most critical secret of how he made the lenses.[42] For many years no one was able to reconstruct Van Leeuwenhoek's design techniques, but in 1957, C. L. Stong used thin glass thread fusing instead of polishing, and successfully created some working samples of a Van Leeuwenhoek design microscope.[43] Such a method was also discovered independently by A. Mosolov and A. Belkin at the Russian Novosibirsk State Medical Institute.[44] In May 2021 researchers in the Netherlands published a non-destructive neutron tomography study of a Leeuwenhoek microscope.[22] One image in particular shows a Stong/Mosolov-type spherical lens with a single short glass stem attached (Fig. 4). Such lenses are created by pulling an extremely thin glass filament, breaking the filament, and briefly fusing the filament end. The nuclear tomography article notes this lens creation method was first devised by Robert Hooke rather than Leeuwenhoek, which is ironic given Hooke's subsequent surprise at Leeuwenhoek's findings.
Here are some reflections I wrote on the work of Grothendieck and relations with his contemporaries & predecessors.
Take it with a grain of salt - it is probably too deflationary of Grothendieck's work, pushing back on mythical narratives common in certain mathematical circles where Grothendieck is held to be an Christ-like figure. I pushed back on that a little. Nevertheless, it would probably not be an exaggeration to say that Grothendieck's purely scientific contributions [as opposed to real-life consequences] were comparable to those of Einstein.
Here's a document called "Upper and lower bounds for Alien Civilizations and Expansion Rate" I wrote in 2016. Hanson et al. Grabby Aliens paper was submitted in 2021.
The draft is very rough. Claude summarizes it thusly:
The document presents a probabilistic model to estimate upper and lower bounds for the number of alien civilizations and their expansion rates in the universe. It shares some similarities with Robin Hanson's "Grabby Aliens" model, as both attempt to estimate the prevalence and expansion of alien civilizations, considering the idea of expansive civilizations that colonize resources in their vicinity.
However, there are notable differences. Hanson's model focuses on civilizations expanding at the highest possible speed and the implications of not observing their visible "bubbles," while this document's model allows for varying expansion rates and provides estimates without making strong claims about their observable absence. Hanson's model also considers the idea of a "Great Filter," which this document does not explicitly discuss.
Despite these differences, the document implicitly contains the central insight of Hanson's model – that the expansive nature of spacefaring civilizations and the lack of observable evidence for their existence imply that intelligent life is sparse and far away. The document's conclusions suggest relatively low numbers of spacefaring civilizations in the Milky Way (fewer than 20) and the Local Group (up to one million), consistent with the idea that intelligent life is rare and distant.
The document's model assumes that alien civilizations will become spacefaring and expansive, occupying increasing volumes of space over time and preventing new civilizations from forming in those regions. This aligns with the "grabby" nature of aliens in Hanson's model. Although the document does not explicitly discuss the implications of not observing "grabby" aliens, its low estimates for the number of civilizations implicitly support the idea that intelligent life is sparse and far away.
The draft was never finished as I felt the result wasn't significant enough. To be clear, the Hanson-Martin-McCarter-Paulson paper contains more detailed models and much more refined statistical analysis. I didn't pursue these ideas further.
I wasn't part of the rationality/EA/LW community. Nobody I talked to was interested in these questions.
Let this be a lesson for young people: Don't assume. Publish! Publish in journals. Publish on LessWrong. Make something public even if it's not in a journal!
Idk the Nobel prize committee thought it wasn't significant enough to give out a separate prize 🤷
I am not familiar enough with the particulars to have an informed opinion. My best guess is that in general statements to the effect of "yes X also made scientific contribution A but Y phrased it better' overestimate the actual scientific counterfactual impact of Y. It generically weighs how well outsiders can understand the work too much vis a vis specialists/insiders who have enough hands-on experience that the value-add of a simpler/neater formalism is not that high (or even a distraction).
The reason Dick Feynmann is so much more well-known than Schwinger and Tomonaga surely must not be entirely unrelated with the magnetic charisma of Dick Feynmann.
Depending on what one means by 'learn' this is provably impossible. The reason has nothing to do with the transformer architecture (which one shouldn't think of as a canonical architecture in the grand scheme of things anyway).
There is a 2-state generative HMM such that the optimal predictor of the output of said generative model provably requires an infinite number of states. This is for any model of computation, any architecture.
Of course, that's maybe not what you intend by 'learn'. If you mean by 'learn' express the underlying function of an HMM then the answer is yes by the Universal Approximation Theorem (a very fancy name for a trivial application of the Stone-Weierstrass theorem).
Hope this helped. 😄
Not inconceivable, I would even say plausible, that surreal numbers & combinatorial game theories impact is still in the future.
An example that's probably * not* a highly counterfactual discovery is the discovery of DNA as the inheritance particle by Watson & Crick [? Wilkins, Franklin, Gosling, Pauling...].
I had great fun reading Watson's scientific-literary fiction the Double Helix. Watson and Crick are very clear that competitors were hot on their heels, a matter of months, a year perhaps.
EDIT: thank you nitpickers. I should have said structure of DNA, not its role as the carrier of inheritance.
Feymann's path integral formulation can't be that counterfactually large. It's mathematically equivalent to Schwingers formulation and done several years earlier by Tomonaga.
Not at all cringe! This is the age of AI. We either channel its immense power or ignore it at our own peril.
There is no human alive today that is utilizing even last-generation's LLMs at their full potential. We should all be copying, delegating and cyborging much more from, to and with LLM - not less.
Could you say more about the motivation here ?
I'd be worried about evaporative cooling. It seems that the net result of this would be that labs would be almost completely devoid of people earnest about safety.
I agree with you government pathways to impact are most plausible and until recently undervalued. I also agree with you there are weird competitive pressures at labs.
There used to be a lot more 'conversation starter' LW posts. Nowadays posts are generally longer but I feel those short ones often were highly valuable.
eg some of Wei Dai's single pagers from a decade ago
I really like this. I'd wish this would become a top-level post.
If you would post this comment with minimal editing I think it would be worthwhile. Top level LWposts are too nowadays
Thank you practicing the rationalist virtue of scholarship Christian. I was not aware of this paper.
You will have to excuse me for practicing rationalist vice and not believing nor investigating further this paper. I have been so traumatized by the repeated failures of non-hard science, I reject most social science papers as causally confounded p-hacked noise unless it already confirms my priors or is branded correct by somebody I trust.
You are taking subagents too literally here. If you prefer take another word like shard, fragment, component, context-dependent action impulse generator etc
Does internal bargaining and geometric rationality explain ADHD & OCD?
Self- Rituals as Schelling loci for Self-control and OCD
Why do people engage in non-social Rituals 'self-rituals'? These are very common and can even become pathological (OCD).
High-self control people seem to more often have OCD-like symptoms.
One way to think about self-control is as a form of internal bargaining between internal subagents. From this perspective, Self-control, time-discounting can be seen as a resource. In the absence of self-control the superagent
Do humans engage in self-rituals to create Schelling points for internally bargaining agents?
Exploration, self-control, internal bargaining, ADHD
Why are exploration behaviour and lack of selfcontrol linked ? As an example ADHD-people often lack self-control, conscientiousness. At the same time, they explore more. These behaviours are often linked but it's not clear why.
It's perfectly possible to explore, deliberately. Yet, it seems that the best explorers are highly correlated with lacking self-control. How could that be?
There is a boring social reason: doing a lot of exploration often means shirking social obligations. Self-deceiving about your true desires might be the only way to avoid social repercussions. This probably explains a lot of ADHD - but not necessarily all.
If self-control = internal bargaining then it would follow that a lack of self-control is a failure of internal bargaining. Note that with subagents I mean both subagents in space *and* time . From this perspective an agent through time could alternatively be seen as a series of subagents of a 4d worm superagent.
This explains many of the salient features of ADHD:
[Claude, list salient features and explain how these are explained by the above]
- Impulsivity: A failure of internal subagents to reach an agreement intertemporaly, leading to actions driven by immediate desires.
- Difficulty with task initiation and completion: The inability of internal subagents to negotiate and commit to a course of action.
- Distractibility: A failure to prioritize the allocation of self-control resources to the task at hand.
- Hyperfocus: A temporary alignment of internal subagents' interests, leading to intense focus on engaging activities.
- Disorganization: A failure to establish and adhere to a coherent set of priorities across different subagents.
- Emotional dysregulation: A failure of internal bargaining to modulate emotional reactions.
Arithmetic vs Geometric Exploration. Entropic drift towards geometric rationality
[this section obviously owes a large intellectual debt to Garrabrant's geometric rationality sequence]
Sometimes people like to say that geometric exploration = kelly betting =maximizing geometric mean is considered to be 'better' than arithmetic mean.
The problem is that actually just maximizing expected value rather than geometric expected value does in fact maximize the total expected value, even for repeated games (duh!). So it's not really clear in what sense geometric maximization is better in a naive sense.
Instead, Garrabrant suggests that it is better to think of geometric maximizing as a part of a broader framework of geometric rationality wherein Kelly betting, Nash bargaining, geometric expectation are all forms of cooperation between various kinds of subagents.
If self-control is a form of sucessful internal bargaining then it is best to think of it as a resource. It is better to maximize arithmetic mean but it means that subagents need to cooperate & trust each other much more. Arithmetic maximization means that the variance of outcomes between future copies of the agent is much larger than geometric maximization. That means that subagents should be more willing to take a loss in one world to make up for it in another.
It is hard to be coherent
It is hard to be a coherent agent. Coherence and self-control are resources. Note that having low time-discounting is also a form of coherence: it means the subagents of the 4d-worm superagent are cooperating.
Having subagents that are more similar to one another means it will be easier for them to cooperate. Conversely, the less they are alike the harder it is to cooperate and to be coherent.
Over time, this means there is a selective force against an arithmetic mean maximizing superagent.
Moreover, if the environment is highly varied (for instance when the agent select the environment to be more variable because it is exploring) the outcomes for subagents is more varied so there is more entropic pressure on the superagent.
This means that in particular we would expect superagents that explore more (ADHDers) are less coherent over time (higher time-discounting) and space (more internal conflict etc).
If therapist quality would actually matter why don't we see this reflected in RCTs?
I internalized the Dodo verdict and concluded that the specific therapist or therapist style didn't matter anyway. A therapist is just a human mirror. The answer was inside of you all along Miles
Why (talk-)Therapy
Therapy is a curious practice. Therapy sounds like a scam, quackery, pseudo-science but it seems RCT consistently show therapy has benefits above and beyond medication & placebo.
Therapy has a long history. The Dodo verdict states that it doesn't matter which form of therapy you do - they all work equally well. It follows that priests and shamans served the functions of a therapist. In the past, one would confessed ones sins to a priest, or spoken with the local shaman.
There is also the thing that therapy is strongly gendered (although this is changing), both therapists and their clientele lean female.
Self-Deception
Many forecasters will have noticed that their calibration score tanks the moment they try to predict salient facts about themselves. We are not-well calibrated about our own beliefs and desires.
Self-Deception is very common, arguably inherent to the human condition. There are of course many Hansonian reasons for this. I refer the reader to the Elephant and the Brain. Another good source would be Robert Trivers. These are social reasons for self-deception.
It is also not implausible that there are non-social reasons for self-deception. Predicting one-self perfectly can in theory lead one to get stuck in Procrastination Paradoxes. Whether this matters in practice is unclear to me but possible. Exuberant overconfidence is another case that seems like a case of self-deception.
Self-deception can be very useful, but one still pays the price for being inaccurate. The main function of talk-therapy seems to be to have a safe, private space in which humans can temporarily step out of their self-deception and reasses more soberly where they are at.
It explains many salient features of talk- therapy: the importance of talking extensively to another person that is (professionally) sworn to secrecy and therefore unable to do anything with your information.
Four levels of information theory
There are four levels of information theory.
Level 1: Number Entropy
Information is measured by Shannon entropy
Level 2: Random variable
look at the underlying random variable ('surprisal') of which entropy is the expectation.
Level 3: Coding functions
Shannon's source coding theorem says entropy of a source is the expected number of bits for an optimal encoding of samples of .
Related quantity like mutual information, relative entropy, cross entropy, etc can also be given coding interpretations.
Level 4: Epsilon machine (transducer)
On level 3 we saw that entropy/information actually reflects various forms of (constrained) optimal coding. It talks about the codes but it does not talk about how these codes are implemented.
This is the level of Epsilon machines, more precisely epsilon transducers. It says not just what the coding function is but how it is (optimally) implemented mechanically.
That is my understanding, yes.
Encrypted Batteries
(I thank Dmitry Vaintrob for the idea of encrypted batteries. Thanks to Adam Scholl for the alignment angle. Thanks to the Computational Mechanics at the receent compMech conference. )
There are no Atoms in the Void just Bits in the Description. Given the right string a Maxwell Demon transducer can extract energy from a heatbath.
Imagine a pseudorandom heatbath + nano-Demon. It looks like a heatbath from the outside but secretly there is a private key string that when fed to the nano-Demon allows it to extra lots of energy from the heatbath.
P.S. Beyond the current ken of humanity lies a generalized concept of free energy that describes the generic potential ability or power of an agent to achieve goals. Money, the golden calf of Baal is one of its many avatars. Could there be ways to encrypt generalized free energy batteries to constraint the user to only see this power for good? It would be like money that could be only spent on good things.
Clem's Synthetic- Physicalist Hypothesis
The mathematico-physicalist hypothesis states that our physical universe is actually a piece of math. It was famously popularized by Max Tegmark.
It's one of those big-brain ideas that sound profound when you first hear about it, then you think about it some more and you realize it's vacuous.
Recently, in a conversation with Clem von Stengel they suggested a version of the mathematico-physicalist hypothesis that I find provoking.
Synthetic mathematics
'Synthetic' mathematics is a bit of weird name. Synthetic here is opposed to 'analytic' mathematics, which isn't very meaningful either. It has nothing to do with the mathematical field of analysis. I think it's supposed to a reference to Kant's synthetic/ apriori/ a posteriori. The name is probably due to Lawvere.
"In “synthetic” approaches to the formulation of theories in mathematics the emphasis is on axioms that directly capture the core aspects of the intended structures, in contrast to more traditional “analytic” approaches where axioms are used to encode some basic substrate out of which everything else is then built analytically."
If you read synthetic read 'Euclidean'. As in - Euclidean geometry is a bit of an oddball field of mathematics, despite being the oldest - it defines points and lines operationally instead of out of smaller pieces (sets).
In synthetic mathematics you do the same but for all the other fields of mathematics. We have synthetic homotopy theory (aka homotopy type theory), synthetic algebraic geometry, synthetic differential geometry, synthetic topology etc.
A type in homotopy type theory is solely defined by its introduction rules and elimination rules (+ univalence axiom). It means a concept it defined solely by how it is used - i.e. operationally.
Agent-first ontology & Embedded Agency
Received opinion is that Science! says there is nothing but Atoms in the Void. Thinking in terms of agents, first-person view concepts like I and You, actions & observations, possibilities & interventions is at best an misleading approximation at worst a degerenerate devolution to cavemen thought. The surest sign of a kook is their insistence that quantum mechanics proves the universe is conscious.
But perhaps the way forward is to channel our inner kook. What we directly observe is qualia, phenomena, actions not atoms in the void. The fundamental concept is not atoms in the void, but agents embedded in environments
(see also Cartesian Frames, Infra-Bayesian Physicalism & bridge rules, UDASSA)
Physicalism
What would it look like for our physical universe to be a piece of math?
Well internally to synthetic mathematical type theory there would be something real - the universe is a certain type. A type such that it 'behaves' like a 4-dimensional manifold (or something more exotic like 1+1+3+6 rolled up Calabi-Yau monstrosities).
The type is defined by introduction and elimination rules - in other words operationally: the universe is what one can * do *with it .
Actually instead of thinking of the universe as a fixed static object we should be thinking of an embedded agent in a environment-universe.
That is we should be thinking of an * interface *
[cue: holographic principle]
I'm a little skeptical of your contention that all these properties are more-or-less independent. Rather there is a strong feeling that all/most of these properties are downstream of a core of agentic behaviour that is inherent to the notion of true general intelligence. I view the fact that LLMs are not agentic as further evidence that it's a conceptual error to classify them as true general intelligences, not as evidence that ai risk is low. It's a bit like if in the 1800s somebody says flying machines will be dominant weapons of war in the future and get rebutted by 'hot gas balloons are only used for reconnaissance in war, they aren't very lethal. Flying machines won't be a decisive military technology '
I don't know Nate's views exactly but I would imagine he would hold a similar view (do correct me if I'm wrong ). In any case, I imagine you are quite familiar with the my position here.
I'd be curious to hear more about where you're coming from.
Do you have a reference for this statement? I suppose it is related to Joyal's result that a Boolean category is always a poset, preventing a naive categorical analysis of classical proof theory.
You are right to think it is puzzling! I am biased but I am of the opinion there are very deep phenomena lurking under these seemingly unimportant oddities.
To the a first degree the answer mostly boils down to the assymmetry in Set, the category of sets. Set certainly doesn't treat limits and colimits the same, so concepts built on top of it (like limits and colimits for ordinary categories) inherit its assymmetry.
As a very concrete instance of the assymmetry in Set: The initial object 0 in Set is the empty set, which has a unique map to any other set (the empty map). The terminal object 1 in set is the set containing one element. Every set X has a unique map to 1.
What about the other way around? Given a set X, what about the maps 1-> X? They exactly correspond to elements x of X ! Cool! Then what about maps X -> 0 ? Ah. Not so interesting. There are none, unless X=0 is empty.
As pointed out by tailcalled, the opposite category of Set can be described as the category of complete atomic boolean algebras, a quite interesting category unto itself... but not equivalent to Set.
Cool, that's all very good but it doesn't explain why Set was assymetric in the first place.
Is this intrinsic? There are certainly natural categories that are equivalent to their opposites (the category of relations is a good examples).
To really understand what's going on here we need to dig deeper. We need to step outside the traditional framework of set theory, and re-examine the foundations of type theory. If we do, we will find that even the very notion of category itself has an underlying bias, a broken duality. To be continued...
It seems to me Israel is clearly differentiated from these other countries.
South Korea and Taiwan have not mobilized and/or actually fought a real shooting war in the past 70 years.
Ukraine has 'only' been at war since 2014. It Russia has fought some very small border nations but never mobilized or fought anything approaching a major war until Ukraine. Similarly for China. It also seems to be the case that East-asian countries and former Soviet Union countries have depressed fertility rates for whatever reason.
North Korea seems like a case on to its own. Despite having nukes it be a stretch calling it a developed country.
Israel has fought major wars, insurgencies, and/or mobilized every decade or so. There are regular bombings and terrorist attacks. It also permanently occupies a neighbouring state that is similar in size and population. Whatever one thinks about Israel/Palestine and its politics I hope we can all agree that many Israeli feel a sense of being 'permanently under siege'. This seems reflected in the political landscape which was quite leftwing and progressive at its founding and has moved significantly to the right since.
It is of course also is very atypical in its cultural, religious, ethnic makeup, its geopolitical position and socioeconomic place and the large amount of migration so I agree with you one should be careful at drawing too many conclusions.
The elephant in the room doesn't seem to have been mentioned. Israel is (the only developed country) in a permanent state of war.
A simple yet peculiar truth :
A category is not generically isomorphic to its opposite.
Yes!
mmm Good point. Do you have more examples?
Good point.
I do wonder to what degree that may be biased by the fact that there were vastly less academic positions before WWI/WWII. In the time of Darwin and Carnot these positions virtually didn't exist. In the time of Einstein they did exist but they were quite rare still.
How many examples do you know of this happening past WWII?
Shannon was at Bell Labs iirc
As counterexample of field-founding happening in academia: Godel, Church, Turing were all in academia.
Thank you, Thomas. I believe we find ourselves in broad agreement. The distinction you make between lay-legibility and expert-legibility is especially well-drawn.
One point: the confidence of researchers in their own approach may not be the right thing to look at. Perhaps a better measure is seeing who can predict not only their own approach will succed but explain in detail why other approaches won't work. Anecdotally, very succesful researchers have a keen sense of what will work out and what won't - in private conversation many are willing to share detailed models why other approaches will not work or are not as promising. I'd have to think about this more carefully but anecdotally the most succesful researchers have many bits of information over their competitors not just one or two. (Note that one bit of information means that their entire advantage could be wiped out by answering a single Y/N question. Not impossible, but not typical for most cases)
Perhaps.
What a heart-warming story!
Thanks for your skepticism, Thomas. Before we get into this, I'd like to make sure actually disagree. My position is not that scientific progress is mostly due to plucky outsiders who are ignored for decades. (I feel something like this is a popular view on LW). Indeed, I think most scientific progress is made through pretty conventional (academic) routes.
I think one can predict that future scientific progress will likely be made by young smart people at prestigious universities and research labs specializing in fields that have good feedback loops and/or have historically made a lot of progress: physics, chemistry, medicine, etc
My contention is that beyond very broad predictive factors like this, judging whether a research direction is fruitful is hard & requires inside knowledge. Much of this knowledge is illegible, difficult to attain because it takes a lot of specialized knowledge etc.
Do you disagree with this ?
I do think that novel research is inherently illegible. Here are some thoughts on your comment :
1.Before getting into your Nobel prize proposal I'd like to caution for Hindsight bias (obvious reasons).
-
And perhaps to some degree I'd like to argue the burden of proof should be on the converse: show me evidence that scientific progress is very legible. In some sense, predicting what directions will be fruitful is a bet against the (efficiënt ?) scientific market.
-
I also agree the amount of prediction one can do will vary a lot. Indeed, it was itself an innovation (eg Thomas Edison and his lightbulbs !) that some kind of scientific and engineering progress could by systematized: the discovery of R&D.
I think this works much better for certain domains than for others and a to large degree the 'harder' & more 'novel' the problem is the more labs defer 'illegibly' to the inside knowledge of researchers.