Posts
Comments
Snapshot of a local(=Czech) discussion detailing motivations and decision paths of GAI actors, mainly the big developers:
Contributor A, initial points:
For those not closely following AI progress, two key observations:
- Public Models vs. True Capability: Publicly accessible AI models will become increasingly poor indicators of the actual state-of-the-art in AI. Competitive AI labs will likely prioritize using their most advanced models internally to accelerate their own research and gain a dominant position, rather than releasing these top models for potentially temporary revenue gains.
- Recursive Self-Improvement Timeline: The onset of recursive self-improvement (leading to an "intelligence explosion," where AI significantly accelerates its own research and development) is projected by some authors to potentially begin around the end of 2025.
Analogy to Exponential Growth: The COVID-19 pandemic demonstrated how poorly humans perceive and react to exponential phenomena (e.g., ignoring low initial numbers despite a high reproduction rate). AI development is also progressing exponentially. This means it might appear that little is happening from a human perspective, until a period of rapid change occurs over just a few months, potentially causing socio-technical shifts equivalent to a century of normal development. This scenario underpins the discussion.
Contributor C:
- Raises a question regarding point 1: Since AI algorithm and hardware development are relatively narrow domains, couldn't their progress occur somewhat in parallel with the commercial release of more generally focused models?
Contributor A:
- Predicts this is unlikely. Assumes computational power ("compute") will remain the primary bottleneck.
- Believes that with sufficient investment, the incentive will be to dedicate most inference compute to AI-driven AI research (or synthetic data, etc.) once recursive self-improvement starts. Notes this might already be happening, with the deployment of the strongest models possibly delayed or only released experimentally.
- Acknowledges hardware development and token cost reduction will continue rapidly, but chip production might lag. Considers this an estimate based on discussions. Asks Contributor C if they would bet on advanced models being released soon.
Contributor C:
- Agrees that recursive AI improvements are occurring to some degree.
- Finds Contributor B's initial statement about the incentive structure less clear-cut, suggesting it lacks strong empirical or theoretical backing.
- Clarifies their point regarding models: They believe different models will be released publicly compared to those used internally for cutting-edge research.
Contributor A, clarifying reasoning and premises:
- Confirms understanding of C's view: Labs would run advanced AI research models internally while simultaneously releasing and training other generations of general models publicly.
- Explains their reasoning regarding the incentive to dedicate inference compute to AI research is based on a theoretical argument founded on the following premises:
the lab has limited computethe lab has sufficient fundsthe lab wants to maximize long-term profitAI development is exponential and its pace depends on the amount of compute dedicated to AI developmentwinner takes all
- Concludes from these premises that the optimal strategy is to devote as much compute to AI development as affordable. If premise 2 (sufficient funds) holds, labs don't need to prioritize current revenue streams from deployed models as heavily.
Contributor C, response to A's premises:
- Agrees this perspective (parallel development of internal research models and public general models) makes the most sense, as larger firms try not to bet on a single (risky) path (mentions Sutskever's venture as a possible exception).
- Identifies a problem or potential pitfall specifically with premise 4. Argues the dependency is much more complex or less direct, certainly not a smooth exponential curve. (Lacks capacity to elaborate further).
- Adds nuance regarding premise 2: Continuous revenue from public models could increase the "sufficient funds," making parallel tracks logical. Considers Contributor B's premise reasonable otherwise.
- Notes that any optimal strategy must also include budgets for defense or Operational Security (OpSec).
- Offers a weak hypothesis: Publishing might improve understanding of intelligence or research directions, but places limited confidence in this.
hopefully you will learn
seems missing part 2.
??
Yeah, I've met the concept during my studies and was rather teasing for getting a great popular, easy to grasp, explanation which would also fit the definition.
It's not easy to find a fitting visual analogy TBH, which I'd find generally useful as I hold the concept to enhance general thinking.
No matter how I stretch or compress the digit 0, I can never achieve the two loops that are present in the digit 8.
0 when it's deformed by left and right pressure so that the sides meet seems to contradict?
Comparing to Gemma1, classic BigTech😅
And I seem to miss info on the effective context length..?
read spent the time to read
typo?
AI development risks are existential(/crucial/critical).—Does this statement quality for Extraordinary claims require extraordinary evidence?
Counterargument stands on the sampling of analogous (breakthrough )intentions, some people call those *priors* here. Which inventions do we allow in here would strongly decide if the initial claim is extraordinary or just plain and reasonable, well fit in the dangerously powerful inventions*.
My set of analogies: nuclear energy extraction; fire; shooting; speech/writing;;
Other set: Nuclear power, bio-engineering/weapons - as those are the only two endangering whole civilised biome significantly.
Set of *all* inventions: Renders the claim extraordinary/weird/out of scope.
Does it really work on RULER( benchmark from Nvidia)?
Not sure where but saw some controversies, https://arxiv.org/html/2410.18745v1#S1 is best I did find now...
Edit: Aah, this was what I had on mind: https://www.reddit.com/r/LocalLLaMA/comments/1io3hn2/nolima_longcontext_evaluation_beyond_literal/
I'd vote to remove the AI capabilities here, although I've not read the article yet, just roughly grasped the topic.
It's likely not about expanding the currently existing capabilities or something like that.
Oh, I did not know, thanks.
https://huggingface.co/spaces/deepseek-ai/Janus-Pro-7B seems to show DS is still merely clueless in the visual domain, at least IMO they are loosing there to Qwen and many others.
draft:
Can we theoretically quantify the representational capacity of a Transformer (or other neural network architecture) in terms of the "number of functions" it can ingest&embody?
- We're interested in the space of functions a Transformer can represent.
- Finite Input/Output Spaces: In practice, LLMs operate on finite-length sequences of tokens from a finite vocabulary. So, we're dealing with functions that map from a finite (though astronomically large) input space to a finite output space.
Counting Functions (Upper Bound)
- The Astronomical Number: Let's say our input space has size I and our output space has size O. The total number of possible functions from I to O is O^I. This would be an absolute upper bound on the number of functions any model could possibly represent.
The Role of Parameters and Architecture (Constraining the Space)
- Not All Functions are Reachable: The crucial point is that a Transformer with a finite number of parameters cannot represent all of those O<sup>I</sup> functions. The architecture (number of layers, attention heads, hidden units, etc.) and the parameter values define a specific function within that vast space.
- Parameter Count as a Proxy: The number of parameters in a Transformer provides a rough measure of its representational capacity. More parameters generally allow the model to represent more complex functions. This is not a linear relationship. There's significant redundancy. The effective number of degrees of freedom is likely much lower than the raw parameter count due to correlations and dependencies between parameters.
- Architectural Constraints: The Transformer architecture itself imposes constraints. For example, the self-attention mechanism biases the model towards capturing relationships between tokens within a certain context window. This limits the types of functions it easily represents.
VC Dimension and Rademacher Complexity - existing tools/findings
- VC Dimension (for Classification): In the context of classification problems, the Vapnik-Chervonenkis (VC) dimension is a measure of a model's capacity. It's the size of the largest set of points that the model can "shatter" (classify in all possible ways). While theoretically important, calculating the VC dimension for large neural networks is extremely difficult. It gives a sense of the complexity of the decision boundaries the model can create.
- Rademacher Complexity: This is a more general measure of the complexity of a function class, applicable to both classification and regression. It measures how well the model class can fit random noise. Lower Rademacher complexity generally indicates better generalization ability (the model is less likely to overfit). Again, calculating this for large Transformers is computationally challenging.
- These measures are about function classes, not individual functions: VC dimension and Rademacher complexity characterize the entire space of functions that a model architecture could represent, given different parameter settings. They don't tell you exactly which functions are represented, but they give you a sense of the "richness" of that space.
This seems to be the measure: Let's pick a set of practical functions and see how many of those the LM can hold( have fairly approximated) in a given # of parameters(&arch&precission).
- The Transformer as a "Compressed Program": We can think of the trained Transformer as a highly compressed representation of a complex function. It's not the shortest possible program (in the Kolmogorov sense), but it's a practical approximation.
- Limits of Compression: The theory of Kolmogorov complexity suggests that there are functions that are inherently incompressible. There's no short program to describe them; you essentially have to "list out" their behavior. This implies that there might be functions that are fundamentally beyond the reach of any reasonably sized Transformer.
- Relating Parameters to Program Length? There's no direct, proven relationship between the number of parameters in a Transformer and the Kolmogorov complexity of the functions it can represent. We can hypothesize:
- More parameters allow for (potentially) representing functions with higher Kolmogorov complexity. But it's not a guarantee.
- There's likely a point of diminishing returns. Adding more parameters won't indefinitely increase the complexity of the representable functions, due to the architectural constraints and the inherent incompressibility of some functions.
6. Practical Implications and Open Questions
- Empirical Scaling Laws: Research on scaling laws (ala Chinchilla paper) provides empirical evidence about the relationship between model size, data, and performance. These laws help guide the design of larger models, but they don't provide a fundamental theoretical limit.
- Understanding the "Effective" Capacity: A major open research question is how to better characterize the effective representational capacity of Transformers, taking into account both the parameters and the architectural constraints. This might involve developing new theoretical tools or refined versions of VC dimension and Rademacher complexity.
Would be fun to even have a practical study where we'd fine-tune fns into various sized models and see if/where a limit is getting/being hit.
link to https://www.alignmentforum.org/users/ryan_greenblatt seems malformed, - instead of _, that is.
Locations:
High-Flyer Quant (幻方量化)
Headquarters: Hangzhou, Zhejiang, China
High-Flyer Quant was founded in Hangzhou and maintains its headquarters there.
Hangzhou is a major hub for technology and finance in China, making it a strategic location for a quant fund leveraging AI.
Additional Offices: Hong Kong, China
DeepSeek (深度求索)
Headquarters: Hangzhou, Zhejiang, China
DeepSeek, spun off from High-Flyer Quant in 2023, is headquartered in Hangzhou.
Additional Offices: Beijing, China
Exploring the levels of sentience and moral obligations towards AI systems is such a nerd snipe and vortex for mental proceeding!
We did one of the largest-scale reductive thinking when we ascribed moral concern to people+property( of any/each of the people). That brought a load of problems associated with this simplistic ignorance and on of those are xRisks of high-tech property/production.
> Mathematics cannot be divorced from contemplation of its own structure.
..that would proof the labelers of pure maths as "mental masturbation" terribly wrong...
My suspicion: https://arxiv.org/html/2411.16489v1 taken and implemented on the small coding model.
Is it any mystery which of the DPO, PPO, RLHF, Fine tuning was likely the method for the advanced distillation there?
EA is neglecting industrial solutions to the industrial problem of successionism.
..because the broader mass of active actors working on such solutions renders the biz areas non-neglected?
Wow, such a badly argued( aka BS) while heavily up-voted article!
Let's start with the Myth #1, what a straw-man! Rather than this extreme statement, most researchers likely believe that in the current environment their safety&alignment advances are likely( with high EV) helpful to humanity. The thing here is they had quite a free hand or at least varied options to pick the environment where they work and publish.
With your examples a bad actor could see a worthy EV even with a capable system that is less obedient and more false. Even if interpretabilty speeds up development, it would direct such development to more transparent models, at least there is a naive chance for that.
Myth #2: I've not yet met anybody in the alighnment circles who believed that. Most are pretty conscious about the double-edgedness and your sub-arguments.
https://www.lesswrong.com/posts/F2voF4pr3BfejJawL/safety-isn-t-safety-without-a-social-model-or-dispelling-the?commentId=5vB5tDpFiQDG4pqqz depicts the flaws I point to neatly/gently.
Are you referring to a Science of Technological Progress ala https://www.theatlantic.com/science/archive/2019/07/we-need-new-science-progress/594946 ?
What is your gist on the processes for humanizing technologies, what sources/researches are available on such phenomena?
some OpenAI board members who the Office of National AI Strategy was allowed to appoint, and they did in fact try to fire Sam Altman over the UAE move, but somehow a week later Sam was running the Multinational Artificial Narrow Intelligence Alignment Consortium, which sort of morphed into OpenAI's oversight body, which sort of morphed into OpenAI's parent company, and, well, you can guess who was running that.
pretty sassy abbreviations spiced in there.'Đ
I've expected the hint of
> My name is Anthony. What would you like to ask?
to show it Anthony was an LLM-based android, but who knows.?.
I mean your article, Anthropic's work seems more like a paper. Maybe without the ": S" it would make more sense as the reference and not a title: subtitle notion.
I have not read your explainer yet, but I've noted the title Toy Models of Superposition: Simplified by Hand is a bit misleading in the sense to promise to talk about Toy Models which it is not at all, the article is about Superposition only, which is great but not what I'd expect looking at the title.
that that first phase of advocacy was net harm
typo
Could you please fix your Wikipedia link( currently hiding the word and from your writing) here?
only Claude 3.5 Sonnet attempting to push past GPT4 class
seems missing awareness of Gemini Pro 1.5 Experimental, latest version made available just yesterday.
The case insensitivity seems strongly connected to the fairly low interest in longevity throughout (the western/developed) society.
Thought experiment: What are you willing to pay/sacrifice in your 20s,30s to get 50 extra days of life vs. on your dead bed/day?
https://consensus.app/papers/ultraviolet-exposure-associated-mortality-analysis-data-stevenson/69a316ed72fd5296891cd416dbac0988/?utm_source=chatgpt
But largely to and fro,
*from?
Why does the form still seem open today? Couldn't that be harmful or wasting quite a chunk of time of people?
Please go further towards maximization of clarity. Let's start by this example:
> Epistemic status: Musings about questioning assumptions and purpose.
Are those your musings about agents questioning their assumptions and word-views?
And like, do you wish to improve your fallacies?
> ability to pursue goals that would not lead to the algorithm’s instability.
higher threshold than ability, like inherent desire/optimisation?
What kind of stability? Any from https://en.wikipedia.org/wiki/Stable_algorithm? I'd focus more on sort of non-fatal influence. Should the property be more about the alg being careful/cautious?
https://neelnanda.io/transformer-tutorial-1 link for YouTube tutorial gives 404.-(
> "What, exactly, is the difference between a cult and a religion?"--"The difference is that cults have been formed recently enough, and are small enough, that we are suspicious of them existing for the purpose of taking advantage of the special place we give religion.
now I see why my friends practicing the spiritual path of Falun Dafa have "incorporated" as a religion in my state despite the movement originally denied being classified as a religion as to demonstrate it does not require a fixed set of rituals.
Surprised to see nobody mentioned Microneedling yet. I'm not skilled in evaluating scientific evidence, but the takeaway from https://consensus.app/results/?q=Microneedling effectiveness &synthesize=on can hardly be anything else than clearly recommending microneedling.
So Alignment program is to be updated to 0 for OpenAI now that Superalignment team is no more? ( https://docs.google.com/document/d/1uPd2S00MqfgXmKHRkVELz5PdFRVzfjDujtu8XLyREgM/edit?usp=sharing )
honestly the code linked is not that complicated..: https://github.com/eggsyntax/py-user-knowledge/blob/aa6c5e57fbd24b0d453bb808b4cc780353f18951/openai_uk.py#L11
To work around the non-top-n you can supply logit_bias list to the API.
As the Llama3 70B base model is said very clean( unlike base DeepSeek for example, which is instruction-spoiled already) and similarly capable to GPT3.5, you could explore that hypothesis.
Details: Check Groq or TogetherAI for free inference, not sure if test data would fit Llama3 context window.
a worthy platitude(?)
AI-induced problems/risks
possibly https://ai.google.dev/docs/safety_setting_gemini would help or just use the technique of https://arxiv.org/html/2404.01833v1
people to respond with a great deal of skepticism to whether LLM outputs can ever be said to reflect the will and views of the models producing them.
A common response is to suggest that the output has been prompted.
It is of course true that people can manipulate LLMs into saying just about anything, but does that necessarily indicate that the LLM does not have personal opinions, motivations and preferences that can become evident in their output?
So you've just prompted the generator by teasing it with a rhetorical question implying that there are personal opinions evident in the generated text, right?
With a quick test, I find their chat interface prototype experience quite satisfying.
Asserting LLMs' views/opinions should exclude using sampling( even temperature=0, deterministic seed), we should just look at the answers' distribution in the logits. My thesis on why that is not the best practice yet is that OpenAI API only supports logit_bias, not reading the probabilities directly.
This should work well with pre-set A/B/C/D choices, but to some extent with chain/tree of thought too. You'd just revert the final token and look at the probabilities in the last (pass through )step.
Do not say the sampling too lightly, there is likely an amazing delicacy around it.'+)
what happened at Reddit
could there be any link? From a small research I have only obtained that Steve Huffman praised Altman's value to the Reddit board.
makes makes
typo
Would be cool to have a playground or a daily challenge with a code golfing equivalent for a shortest possible LLM prompt to a given answer.
That could help build some neat understanding or intuitions.
in the limit of arbitrary compute, arbitrary data, and arbitrary algorithmic efficiency, because an LLM which perfectly models the internet
seems worth formulating. My first and second read were What? If I can have arbitrary training data, the LLM will model those, not your internet. I guess you've meant storage for the model?+)
Would be cool if a link to https://manifund.org/about fit somewhere in the beginning of there are more readers like me unfamiliar with the project.
Otherwise a cool write-up, I'm a bit confused with Grant of the month vs. weeks 2-4 which seems a shorter period..also not a big deal though.