Posts
Comments
I see what you're gesturing at but I'm having difficulty translating it into a direct answer to my question.
Cases where language is fuzzy are abundant. Do you have some examples of where a truth value itself is fuzzy (and sensical) or am I confused in trying to separate these concepts?
Can you help me tease out the difference between language being fuzzy and truth itself being fuzzy?
It's completely impractical to eliminate ambiguity in language, but for most scientific purposes, it seems possible to operationalize important statements into something precise enough to apply Bayesian reasoning to. This is indeed the hard part though. Bayes' theorem is just arithmetic layered on top of carefully crafted hypotheses.
The claim that the Earth is spherical is neither true nor false in general but usually does fall into a binary if we specify what aspect of the statement we care about. For example, "does it have a closed surface", "is it's sphericity greater than 99.5%", "are all the points on it's surface between radius * ( 1 +/- epsilon)", "is the circumference of the equator greater than that of the prime meridian".
Synthetically enhancing and/or generating data could be another dimension of scaling. Imagine how much deeper understanding a person/LLM would have if instead of simply reading/training on a source like the Bible N times, they had to annotate it into something more like the Oxford Annotated Bible and that whole process of annotation became training data.
I listened to this via podcast. Audio nitpick: the volume levels were highly imbalanced at times and I had to turn my volume all the way up to hear both speakers well (one was significantly quieter than the other).
Appropriate scaffolding and tool use are other potential levers.
Kudos for referencing actual numbers. I don’t think it makes sense to measure humans in terms of tokens, but I don’t have a better metric handy. Tokens obviously aren’t all equivalent either. For some purposes, a small fast LLM is more way efficient than a human. For something like answering SIMPLEBENCH, I’d guess o1-preview is less efficient while still significantly below human performance.
Is this assuming AI will never reach the data efficiency and energy efficiency of human brains? Currently, the best AI we have comes at enormous computing/energy costs, but we know by example that this isn't a physical requirement.
IMO, a plausible story of fast takeoff could involve the frontier of the current paradigm (e.g. GPT-5 + CoT training + extended inference) being used at great cost to help discover a newer paradigm that is several orders of magnitude more efficient, enabling much faster recursive self-improvement cycles.
CoT and inference scaling imply current methods can keep things improving without novel techniques. No one knows what new methods may be discovered and what capabilities they may unlock.
It's cool that the score voting input can be post-processed in multiple ways. It would be fascinating to try it out in the real world and see how often Score vs STAR vs BTR winners differ.
One caution with score voting is that you don't want high granularity and lots of candidates or else individual ballots become distinguishable enough that people can prove they voted a particular way (for the purpose of getting compensated). Unless marked ballots are kept private, you'd probably want to keep the options 0-5 instead of 0-9 and only allow candidates above a sufficient threshold of support to be listed.
Yes, but with a very different description of the subjective experience -- kind of like getting a sunburn on your back feels very different than most other types of back pain.
Your third paragraph mentions "all AI company staff" and the last refers to "risk evaluators" (i.e. "everyone within these companies charged with sounding the alarm"). Are these groups roughly the same or is the latter subgroup significantly smaller?
I agree. I would not expect the effect on health over 3 years to be significant outside of specific cases like it allowing someone to afford a critical treatment (e.g. insulin for a diabetic person), especially given the focus on a younger population.
This is a cool paper with an elegant approach!
It reminds me of a post from earlier this year on a similar topic that I highly recommend to anyone reading this post: Ironing Out the Squiggles
OP's model does not resonate with my experience either. For me, it's similar to constantly having the flu (or long COVID) in the sense that you persistently feel bad, and doing anything requires extra effort proportional to the severity of symptoms. The difference is that the symptoms mostly manifest in the brain rather than the body.
This is a cool idea in theory, but imagine how it would play out in reality when billions of dollars are at stake. Who decides the damage amount and the probabilities involved and how? Even if these were objectively computable and independent of metaethical uncertainty, the incentives for distorting them would be immense. This only seems feasible when damages and risks are well understood and there is consensus around an agreed-upon causal model.
I also guessed the ratio of the spheres was between 2 and 3 (and clearly larger than 2) by imagining their weight.
I was following along with the post about how we mostly think in terms of surfaces until the orange example. Having peeled many oranges and separated them into sections, they are easy for me to imagine in 3D, and I have only a weak "mind's eye" and moderate 3D spatial reasoning ability.
Even for people who understand your intended references, that won't prevent them from thinking about the evil-spirit association and having bad vibes.
Being familiar with daemons in the computing context, I perceive the term as whimsical and fairly innocuous.
The section on Chevron Overturned surprised me. Maybe I'm in an echo chamber, but my impression was that most legal scholars (not including the Federalist Society and The Heritage Foundation) consider the decision to be the SCOTUS arrogating yet more power to the judicial branch, overturning 40 years of precedent (which was based on a unanimous decision) without sufficient justification.
I consider the idea that "legislators should never have indulged in writing ambiguous law" rather sophomoric. I don't think it's always possible to write law that is complete, unambiguous, and also good policy. Nor do I think Congress is always the best equipped to do so. I don't fully trust government agencies delegated with rulemaking authority, but I have much less trust that a forum-shopped judge in the Northern District of Texas is likely to make better-informed decisions about drug safety than the FDA.
FWIW, I haven't really thought much about Loper as it relates to AI, tech, and crypto specifically. The consequences of activist judges versus the likes of the DOJ, CDC, FDA, and EPA are mostly what come to mind. Maybe it's attention bias given recent SCOTUS decisions versus more limited memory of out-of-control agencies but I feel uneasy tilting the balance of power toward judicial dominance.
This is similar to the quantum suicide thought experiment:
https://en.wikipedia.org/wiki/Quantum_suicide_and_immortality
Check out the Max Tegmark references in particular.
[Epistemic status: purely anecdotal]
I know people who work in the design and construction of data centers and have heard that some popular data center cities aren't approving nearly as many data centers due to power grid concerns. Apparently, some of the newer data center projects are being designed to include net new power generation to support the data center.
For less anecdotal information, I found this useful: https://sprottetfs.com/insights/sprott-energy-transition-materials-monthly-ais-critical-impact-on-electricity-and-energy-demand/
I can definitely imagine them plausibly believing they're sticking to that commitment, especially with a sprinkle of motivated reasoning. It's "only" incremental nudging the publicly available SOTA rather than bigger steps like GPT2 --> GPT3 --> GPT4.
Exactly. All that’s needed for “transcendence” is removing some noise.
I highly recommend the book Noise by Daniel Kahneman et al on this topic.
My hypothesis is that poor performance on ARC is largely due to lack of training data. If there were billions of diverse input/output examples to train on, I would guess standard techniques would work.
Efficiently learning from just a few examples is something that humans are still relatively good at, especially in simple cases where system1and system 2 synergize well. I’m not aware of many cases where AI approaches human level without orders of magnitude more training data than a human ever sees in a lifetime.
I think the ARC challenge can be solved within a year or two, but doing so won’t be super interesting to me unless it breaks new ground in sample efficiency (not trained on billions of synthetic examples) or generalization (e.g. solved using existing LLMs rather than a specialized net).
Ah, I was under the impression that OP was covering both, not only things relevant to Google Sheets.
Ah, I do see the LET function now but I still can't find a reference to the Power Query editor.
There doesn't appear to be an existing term that neatly fits. Here are some possibilities:
One-word terms:
Traumaraderie (portmanteau of "trauma" and "camaraderie")
Two-word terms:
Stress bonding (a technique for bonding rabbits)
Trauma bonding (this is sometimes used to mean bonding between victims even though it also means bonding between victim and abuser in other contexts)
Trauma kinship
Survivor bond
Longer terms:
Friendship forged in fire
Shared trauma bonding
Mutual suffering solidarity
Camradery through adversity
I'm surprised to see no mention of the LET function for making formulas more readable.
Another glaring omission is Power Query in Excel. I find it incredibly useful for connecting to data sources and transforming data. It's easily auditable as it produces steps of code for table transformations rather than working with thousands of cells with individual formulas.
When it comes to writing about spreadsheets, it's just about impossible to even skim the surface without missing anything, especially considering many aspects like array formulas, VBA macros, pivot tables with DAX measures, and Power Query can go super deep. I own a 758-page textbook just on Power Query and multiple books on Power Pivot / DAX.
Some of these depend on how concise you want to be. For example,
"Partial derivative with respect to x of f of x, y" may be shortened to "partial x, f of x, y".
Conciseness is more common when speaking if the written form is also visible (as opposed to purely vocal communication).
AI Explained already has a video out on it:
The New, Smartest AI: Claude 3 – Tested vs Gemini 1.5 + GPT-4
Supposing humanity is limited to Earth, I can see arguments for ideal population levels ranging from maybe 100 million to 100 billion with values between 1 and 10 billion being the most realistic. However, within this range, I'd guess that maximal value is more dependent on things like culture and technology than on the raw population count, just like a sperm whale's brain being ~1000x the mass of an African grey parrot's brain doesn't make it three orders of magnitude more intelligent.
Size matters (as do the dynamic effects of growing/shrinking) but it's not a metric I'd want to maximize unless everything else is optimized already. If you want more geniuses and more options/progress/creativity, then working toward more opportunities for existing humans to truly thrive seems far more Pareto-optimal to me.
Leela now has a contempt implementation that makes odds games much more interesting. See this Lc0 blog post (and the prior two) for more details on how it works and how to easily play odds games against Leela on Lichess using this feature.
GM Matthew Sadler also has some recent videos about using WDL contempt to find new opening ideas to maximize chances of winning versus a much weaker opponent.
I'd bet money you can't beat LeelaQueenOdds at anything close to a 90% win rate.
Why not go all the way and use a constructed language (like Lojban or Ithkuil) that's specifically designed for the purpose?
All the pics and bragging about how wonderful their adventures were really rub me the wrong way. It comes across as incredibly tone deaf to the allegations and focusing on irrelevant things. Hot tubs, beaches, and sunsets are not so important if you’re suffering from deeper issues. Good relationship dynamics are way more important than scenery and perks, especially in a small group setting.
The markets you linked to roughly align with my intuition that GPT-5 before 2025 is likely, although maybe not fully released publicly.
Things to keep in mind:
- We know that GPT-5 is officially in development.
- The hype since ChatGPT went public is huge and competition is substantial.
- Because of hype and competition, investment will be large in terms of dollars and talent.
- There will be strong incentive to show a return on investment rather than holding back and waiting for something worthy of calling "GPT-5".
Other than the amount of investment in next-gen models, most of my intuition is related to the human and marketing factors involved. OpenAI won't want to lose its lead by waiting many years.
That sounds rather tautological.
Assuming ratfic represents LessWrong-style rationality well and assuming LW-style rationality is a good approximation of truly useful instrumental reasoning, then the claim should hold. There’s room for error in both assumptions.
My hunch is that there's sufficient text already if an AI processes it more reflectively. For example, each chunk of text can be fed through a series of LLM prompts intended to enrich it, and then the model trains on the enriched/expanded text.
I definitely like the term "confabulate" more than "hallucinate". It's more accurate and similar to what humans do. My favorite confabulation examples in humans are split-brain experiments.
"split-brain" patients, whose left and right brain hemispheres have been surgically disconnected for medical treatment. Neuroscientists have devised clever experiments in which information is provided to the right hemisphere (for instance, pictures of naked people), causing a change in behavior (embarrassed giggling). Split-brain individuals are then asked to explain their behavior verbally, which relies on the left hemisphere. Realizing that their body is laughing, but unaware of the nude images, the left hemisphere will confabulate an excuse for the body's behavior ("I keep laughing because you ask such funny questions, Doc!").
I found your thread insightful, so I hope you don't mind me pasting it below to make it easier for other readers.
Neel Nanda ✅ @NeelNanda5 - Sep 24
The core intuition is that "When you see 'A is', output B" is implemented as an asymmetric look-up table, with an entry for A->B. B->A would be a separate entry
The key question to ask with a mystery like this about models is what algorithms are needed to get the correct answer, and how these can be implemented in transformer weights. These are what get reinforced when fine-tuning.
The two hard parts of "A is B" are recognising the input tokens A (out of all possible input tokens) and connecting this to the action to output tokens B (out of all possible output tokens). These are both hard! Further, the A -> B look-up must happen on a single token position
Intuitively, the algorithm here has early attention heads attend to the prev token to create a previous token subspace on the Cruise token. Then an MLP neuron activates on "Current==Cruise & Prev==Tom" and outputs "Output=Mary", "Next Output=Lee" and "Next Next Output=Pfeiffer"
"Output=Mary" directly connects to the unembed, and "Next Output=Lee" etc gets moved by late attention heads to subsequent tokens once Mary is output.
Crucially, there's an asymmetry between "input A" and "output A". Inputs are around at early layers, come from input embeddings, and touch the input weights of MLP neurons. Outputs are around more at late layers, compose with the unembedding, and come from output weights of MLPs
This is especially true with multi-token A and B. Detecting "Tom Cruise" is saying "the current token embedding is Cruise, and the prev token space says Tom", while output "Tom Cruise" means to output the token Tom, and then a late attn head move "output Cruise" to the next token
Thus, when given a gradient signal to output B given "A is" it reinforces/creates a lookup "A -> B", but doesn't create "B->A", these are different lookups, in different parameters, and there's no gradient signal from one to the other.
How can you fix this? Honestly, I can't think of anything. I broadly think of this as LLMs working as intended. They have a 1 way flow from inputs to outputs, and a fundamental asymmetry between inputs and outputs. It's wild to me to expect symmetry/flow reversing to be possible
Why is this surprising at all then? My guess is that symmetry is intuitive to us, and we're used to LLMs being capable of surprising and impressive things, so it's weird to see something seemingly basic missing.
LLMs are not human! Certain things are easy for us and not for them, and vice versa. My guess is that the key difference here is that when detecting/outputting specific tokens, the LLM has no notion of a variable that can take on arbitrary values - a direction has fixed meaning
A better analogy might be in-context learning, where LLMs CAN use "variables". The text "Tom Cruise is the son of Mary Lee Pfeiffer. Mary Lee Pfeiffer is the mother of..." has the algorithmic solution "Attend to the subject of sentence 1 (Tom Cruise), and copy to the output"
Unsurprisingly, the model has no issue with reversing facts in context! Intuitively, when I remember a fact A is B, it's closer to a mix of retrieving it into my "context window" and then doing in-context learning, rather than pure memorised recall.
Adversarial gaming doesn't match my experience much at all and suggesting options doesn't feel imposing either. For me at least, it's largely about the responsibility and mental exertion of planning.
In my experience, mutual "where do you want to go" is most often when neither party has a strong preference and neither feels like taking on the cognitive burden of weighing options to come to a decision. Making decisions takes effort especially when there isn't a clearly articulated set of options and tradeoffs to consider.
For practical purposes, one person should provide 2-4 options they're OK with and the other person can pick one option or veto some option(s). If they veto all given options, they must provide their own set of options the first person can choose or veto. Repeat as needed but rarely is more than one round needed unless participants are picky or disagreeable.
Why do you think that the same federal bureaucrats who incompetently overregulate other industries will do a better job regulating AI?
Chevron deference means that judges defer to federal agencies instead of interpreting the laws themselves where the statute is ambiguous. It's not so much a question of overregulation vs underregulation as it is about who is doing the interpretation. For example, would you rather the career bureaucrats in the Environmental Protection Agency determine what regulations are appropriate to protect drinking water or random judges without any relevant expertise?
One consequence of blowing up Chevron deference is that one activist judge in Texas can unilaterally invalidate FDA approval of a drug like mifepristone for the entire country that's been safe, effective, and available on the markets for decades by substituting his own idiosyncratic opinion instead of deferring to the regulatory agency whose entire purpose is to make those kinds of determinations.
Government agencies aren't always competent but the alternative is a patchwork of potentially conflicting decisions from judges ruling outside of their area of expertise.
If you're open to more experimentation, I'd recommend trying playing against Leela Chess Zero using some of the newer contempt parameters introduced in this PR and available in the latest pre-release version. I'm really curious if you'd notice significant style differences with different contempt settings.
Update: The official v0.30.0 release is out now and there is a blog post detailing the contempt settings. Additionally, there is a Lichess bot set up specifically for knight odds games.
Further update: There are now three Lichess bots set up to play odds games on Lichess: LeelaKightOdds, LeelaQueenForKnight, and LeelaQueenOdds. They are currently featured bots on https://lichess.org/player/bots
"Artificial Intention" doesn't sound catchy at all to me, but that's just my opinion.
Personally, I prefer to think of the "Alignment Problem" more generally rather than "AI Alignment". Regardless of who has the most power (humans, AI, cyborgs, aliens, etc.) and who has superior ethics, conflict arises when participants in a system are not all aligned.
This is the most similar question that I could find that already exists.
https://www.metaculus.com/questions/7384/alien-tech-in-solar-system-before-2030/
There are still HR and legal overhead costs involved if you have human operators.
I think part of the answer is also that the space of things low-wage workers can physically do remotely via teleoperation isn't that much larger than the space of things that can be fully automated but still much smaller than the space of things a local human can do. It's a fairly narrow band to exploit, IMO, and the labor cost arbitrage spread is rarely worth the complexity of the extra logistics, capital investment, and maintenance.
As @kave pointed out already, the right side being thicker would make it more clear that it's specifically a delta.
Something like one of these?
https://thenounproject.com/icon/bored-251902/ https://thenounproject.com/icon/annoyed-4979573/
Embodied tasks just aren't an area where comparison makes much sense yet. What kind of perception tasks did you have in mind?
Does this "paradox" still hold in the era of recent multimodal AI? In particular, what are some things that are easy for humans but hard for AI, other than things requiring embodiment? What areas are human mechanical Turks still much better at? (I believe there are areas but pretty fuzzy about what they are.)
I really like the t-AGI concept for a specific task or benchmark but find it difficult to generalize since capabilities are not uniform. How do you classify an AGI that is 1-second AGI at some tasks but 1-day AGI at others (e.g. due to serial processing speed advantages)?
There is indeed the Myth of the flat Earth that is a misconception about the beliefs of scholars in the Middle Ages and some scholars certainly understood the concept of a spherical Earth since at least Eratosthenes. I'm referring to earlier history like ancient Egyptian, Chinese, and pre-Socratic Greek cosmologies. Admittedly, it's not a great example since most of the debates about it are lost to history, and such debates wouldn't involve the same kind of reasoning and evidential standards we use today.
Yes, clearly the less time the human has, the better Leela will do relatively. One thing to note though is that Lichess Elo isn't completely comparable across different time controls. If you look at the player leaderboard, you can see that the top scores for bullet are ~600 greater than for classical, so scores need to be interpreted in context.
Self-Elo inflation is a fair point to bring up and I don't have information on how well it translates.