Post AGI effect prediction
post by Juliezhanggg · 2025-02-01T21:16:36.829Z · LW · GW · 0 commentsContents
What differentiate us from AI? Emotion, physical intelligence are not what makes us unique Machine intelligence in the long-run is superior? Math is the foundation of intelligence The ingredients of intelligence Human still are valuable at higher hierarchy work Regulated parts of society will be less affected Machine intelligence in the long-run is superior Human enjoy, AI works? The short-term and long-term effect Scarce resources will remain its value Model providers will be abundent thanks to Deepseek Conclusion None No comments
What differentiate us from AI?
Human's sense of meaning usually are derived from feeling special, on an individual or collective basis. We want to prove ourselves different through tastes, hobbies, and career choices. Difference also show through collective culture and individual values. Biologically, we are all different from our gene sequence. Our tiny difference creates huge leverage that lead some people to enormous wealth while others stay in poverty. It's important to answer how AI is fundamentally different from us, and what's our value proposition longterm?
Emotion, physical intelligence are not what makes us unique
I used to assume that AI can't innovate, nor is conscious and have true understanding. After months of inquiries and research in the AI community and neuroscience community, I have completely shifted my mind. Human is not unique biologically, and a different neural network substrate could lead to equivalent results. I have been trying to answer an important question: what is the essence of intelligence, and will AI have emotions, and be self-conscious like we do? The vision of robotics would automate our society has aspired many people, including my mentor who is a founder devoted all his life engineering robotic hands. Physical intelligence will be taken over by humanoid longterm, and the midterm solutions of specialized robotic system will be eliminated.
Math is the basis of intelligence, and Vitalik gave me a few pointers on his opinion on intelligence. It was quite inspiring he thinks human intelligence is no different from LLM. It feels more human than human if intuition is the marker. RLHF could generate outputs that we find agreeable by having emotions. Bio tissues surely are not the only substrate for emotions/intelligence, and Vitalik believed that emotions are algorithmic. A fun example is to have something running on a clock speed of 1 cycle per minute that's implemented by people passing cards around with their hands to have emergent property equivalent to emotion. I didn't buy into this opinion right away when I heard it, but now after my back-and-forth questioning, I am leaning toward it. Emotion is not unique to humans, neither is intelligence.
Machine intelligence in the long-run is superior?
As we speak of intelligence, superbaby, enhanced intelligence in humans has been a hot topic for a while as a way to de-risk AI. It may sound appealing short term, but silicon-based intelligence will be much smarter than us given the same time of development. No human has ever reached the maximum perfect intelligence score so far, but that highest score is also restrained by biology. Human intelligence, and machine intelligence in a race longer run, the machine will probably be a better substrate because we make progress on compute and inference chips every year with speed, and cost. While biologically, we can only achieve our optimized ceiling of genius level intelligence, and there is a real ceiling to it.
Math is the foundation of intelligence
I had a 6-hour conversation with Gared in lighthaven and he pointed to me that ultimately, the definition of intelligence is the combination of probability theory and optimization theory. Probability stands for the search of hypothesis, and optimization theory is the search of expected utility/results. As he is one of the smartest people I've ever met, it aligned with Vitalik's view that understanding intelligence, requires a mathematical way of seeing what is going on in vector spaces together with the understanding of how math maps to reality.
The ingredients of intelligence
Going up to the industry level of current AI progress we can see more clearly what is going on. Everyone is fighting for more computing, more data, and more capital to win a race for who will get to the most intelligent model that can self-iterate to make itself smarter. This is what's going on between these model deployers. Dario predicted AI smarter than human Nobel prize winner to take place by 2026 which is 1 year later.
Human still are valuable at higher hierarchy work
Yes, midterm, human will leverage AI to boost their generated value. Long-term, when a model can self-iterate without much supervision from humans, that's where the endgame is approaching. Lesser and lesser human agency will be required to correct/give instructions until one day, human is no longer needed in the loop. The way AI replaces humans is gradual, first, the one with less experience or domain expertise (fewer data) then move up the ladder to PhD, scientists, and more C-level executives.
Stephen Wolfram said that even if everything has been automated by computation, we could still have an excellent inner experience or meaningful existence. Automation obsoletes some jobs but enables others. Stephan doesn't think everything will be fully automated because of computational irreducibility. Something unpredictable will always happen, something that automation doesn't cover. If there is anything we care about, we will have to do something to go beyond automation.
Regulated parts of society will be less affected
Marc Andreessen recently posted a tweet that's quite interesting, he thinks AI won't cause unemployment because there is one thing will block the progress of AI, making it illegal for most of the economy. Anything controlled by the government is rising in price including healthcare, education, and housing. The regulated part of products rise, and the less regulated parts fall. This argument also opposes the general perception that AI will take over the economy and people will be needed within the regulated zone. Over over-regulation problem is being solved by special economic zone projects that try to push fewer regulations from local governments, such as Prospera Honduras.
Machine intelligence in the long-run is superior
As we speak of intelligence, superbaby, enhanced intelligence in humans has been a hot topic for a while as a way to de-risk AI. It may sound appealing short term, but silicon-based intelligence will be much smarter than us given the same time of development. No human has ever reached the maximum perfect intelligence score so far, but that highest score is also restrained by biology. Human intelligence, and machine intelligence in a race longer run, the machine will probably be a better substrate because we make progress on compute and inference chips every year with speed, and cost. While biologically, we can only achieve our optimized ceiling of genius level intelligence, and there is a real ceiling to it.
Even humans will always be needed for many reasons above, I do think there is a higher intellectual requirement for human labor. The easier the job is, the faster it will get replaced by automation. Humans will need to enhance their intelligence and be a better version of themselves, which means those who aren't capable of dealing with complex unpredictable reality will retire from the workforce. The types of jobs left for humans are always one ladder above AI. If AI writes scripts for humans, humans will manage and validate them. If agents do repetitive work, humans will manage, and assign roles to agents. Each time there is a technical breakthrough, humans will always move up to a more enjoyable, challenging position. Humans will require new education or training to become better planners, predictors, managers, and mayors of highly complex automation systems through efficient masters of tools including BCI, and AI.
Human enjoy, AI works?
Anthropic CEO Dario's blog revealed his thoughts on the after-effects when geniuses live in a data center (AGI). What's the meaning of human? How do they survive economically? Dario thinks humans do things such as running, walking, art, gaming, dating, and family, not for economic return, but purely for the sake of enjoyment. So, "do what you love" will continue to apply its charm to a post-AGI era. However, doing what you love will not lead to economic success according to Dario. His view aligns with the recent research by Epoch AI that short term the economy will boost, with humans leveraging AI leading to 10x growth. As long as AI IS ONLY 90% better at a job, the other 10% will give humans much leverage. A new job category will be created, which is called the amplifier: human complementing what AI is good at. The human economy will continue to make sense after a while when we reach a country of geniuses in a data center. But at one point, our current economic setup will no longer make sense if AI will be effective and cheap broadly speaking. From hunting, farming, feudalism, and industrialism. We need to start envisioning something entirely new to answer what's next. Universal basic income might only be a small part of the solution. He describes a capitalist economy of AI systems that give out resources to humans following a distribution algorithm. Of economy could run on whuffle points, or humans will continue to matter regardless in unexpected ways. He also calls for brave and dedicated people to stand out and put in a huge amount of effort and struggle to achieve a positive version of the future.
The short-term and long-term effect
The research by Epoch AI suggests AGI could drive wages below the subsistence level. But before that, the wage will boost for a while until it keeps diminishing. The core argument is that capital acts as a production bottleneck. When you add more workers without a corresponding increase in other essential resources, then additional work contributes less. The same as Dario's argument where superintelligence as a commodity will be constrained by the speed of the outside world, the need for data, constraints from humans, and physical laws to make progress in the real world. The boost in wages came from the era after the Industrial Revolution, when the rate of innovation increased, eventually outpacing population growth, and leading to sustained increases in marginal productivity of labor. The introduction of AGI into the economy will also drive innovation at a faster pace, more to compensate for AI's negative impact on wages. The prediction is made that human wages will crash below 2045 when technological progress starts to slow down. Counting on wages alone for future generations to survive is unlikely to be reliable.
Different predictions are made by influential people, each has its reference to the history of industrial evolution and other important timelines, they each integrate their unique understanding of regulation, economy, physical laws, and intelligence to map out the future. The common thread here is that short term after AGI penetrates our economy, humans will maintain its current order until it no longer makes sense with AI being cheaper and more effective comparatively. The Industrial Revolution paved the way for capitalism with the wealth getting concentrated in the hands of a few and gave rise to the new economic theory of laissez-faire capitalism and socialism. It also led to later consumerism, and further Technological Progress as well as a new social class called the middle class. All of these were due to the factory system, with the use of machines that increase production efficiency.
Scarce resources will remain its value
AI will probably do something similar to us this time but make wealth concentrated at the hands of people who leverage the AI workforce to profit which in turn might strengthen capitalism but also give rise to new startups with less capital, less team count to rise to new social class (Everyone can have AI employees work for them). More individuals will be able to make the flip from employees to bosses. Ideas will finally be enough without skills or much capital, passion is more important than knowledge. It could lead to an abundance of content, products, and services and the pursuit of technical-driven innovation will be more in demand. Talent-intensive companies like consulting firms might be more affected by valuation than resource-intensive companies.
Model providers will be abundent thanks to Deepseek
With Deepseek disrupting the AI model space last week, the future of foundational models gets stirred. If companies can get training costs for powerful reasoning models low, maybe some companies can train a Nobel prize-level genius who innovates under a reasonable budget. If more companies opensource their model, there will come a time when model providers become abundant with different variations catering to niche demands. Building applications on top of free effective models has its long lasting significance to the market. I wrote this 5 days ago, and today the chip provider have informed me there is a boost in demand for Nvidia H100, H200 because companies want to deploy their own AI models after the Deepseek disruption of OpenAI and Anthropic.
Conclusion
It's hard to predict what will be human's role in a post AGI timeline, we can only reason and be open-minded with changes given major events, change in policies, and reference to history. The most optimal case is human are needed by each others for entertainment, happiness, lifestyle, and storytelling. Each human can be very different, but each model is not that unique. The nature of model is they scale to serve everyone forever, but the nature of individual is to serve only a few purposes in their lifetime. In the long run, usefulness is only for survival, rare item holds higher value in their existence. We don't only live a life of utility, we also live a life to appreciate art and beauty of the world.
0 comments
Comments sorted by top scores.