Posts

sairjy's Shortform 2020-08-09T16:41:24.953Z
Welcome to Rationality Milano [Edit With Your Details] 2019-10-31T11:04:03.352Z

Comments

Comment by sairjy on Alex Irpan: "My AI Timelines Have Sped Up" · 2020-08-20T10:22:44.317Z · LW · GW

If 65% of the AI improvements will come from compute alone, I find quite surprising that the post author assigns only 10% probability of AGI by 2035. By that time, we should have between 20x to 100x compute per $. And we can also easily forecast that AI training budgets will increase 1000x easily over that time, as a shot to AGI justifies the ROI. I think he is putting way too much credit on the computational performance of the human brain.

Comment by sairjy on Open & Welcome Thread - August 2020 · 2020-08-13T12:33:49.275Z · LW · GW

They seem focused on inferencing, which requires a lot less compute than training a model. Example: GPT-3 required thousands of GPUs for training, but it can run on less than 20 GPUs.

Microsoft built an Azure supercluster for OpenAI and it has 10,000 GPUs.

Comment by sairjy on Open & Welcome Thread - August 2020 · 2020-08-13T08:50:30.435Z · LW · GW

Google won't be able to sell outside of their cloud offering, as they don't have the experience in selling hardware to enterprise. Their cloud offering is also struggling against Azure and AWS, ranking 1/5 of the yearly revenues of those two. I am not saying Nvidia won't have competition, but they seem enough ahead right now that they are the prime candidate to have the most benefits from a rush into compute hardware.

Comment by sairjy on Investment idea: basket of tech stocks weighted towards AI · 2020-08-13T08:42:10.299Z · LW · GW

There is a specific piece of evidence that GPT-3 and the events of the last few years in deep learning added: more compute and data are (very likely) keys to bring transformative AI. Personally, I decide to do a focused bet on who produces the compute hardware. After some considerations, I decided for Nvidia as its seems to be company with the most moats and that will benefit more if deep learning and huge amount of compute is key to transformative AI. AI chip startups are not competitive with Nvidia and Google isn't interested/doesn't know how to sell chips.

Investing into FAANG because of the impacts of transformative AI is not a direct bet on AI: the impacts are hard to understand and predict right now and it is not a given that they will increase their revenues significantly because of AI. They already have a business model, and it isn't focused on AI.

Comment by sairjy on Money creation and debt · 2020-08-13T08:29:50.627Z · LW · GW

As far as I understood money myself, your intuition is correct. All fiat currency are credit money, so that when you are holding a $, either in cash or bank deposit, you are holding someone else liability. The system is balanced, so that total liabilities are equal to total assets at any time. The net value of the entire monetary system in the economy is zero.

That's right, but that's the private sector as a whole. Some part of the private sectors will increase their debt, while others their savings. Clearly that would generate business cycles/boom and bust, and that's the big discussion of macroeconomics in what is the role of governments in damping/preventing them.

Since the Treasury owns the Fed, the profits made by the Fed are channeled back to the Treasury. The ECB is a bit more complex, but it works in a similar way. When a central bank buys government debt, that debt is the facto neutralized.

Comment by sairjy on Money creation and debt · 2020-08-13T08:14:55.422Z · LW · GW

I think he meant savings as cash saving/bank deposits. Since all cash savings/bank deposits are the debt of someone else, for the entire private sector to increase its cash holding/bank deposits the government has to increase its debt.

Comment by sairjy on 10/50/90% chance of GPT-N Transformative AI? · 2020-08-10T08:23:24.197Z · LW · GW

The Scaling Laws for Neural Language Model's paper says that the optimal model size scales 5x with 10x more compute. So to be more precise, using GPT-3 numbers (4000 PetaFLOPs/days for 200 billions parameters), a 100 trillion parameters model would require 4000 ExaFLOPs/days. (using GPT-3 architecture, so no sparse or linear transformer improvements). To be fair, the Scaling Law papers also predicts a breaking down of the scaling laws around 1 trillion parameters.

The peak F16 performance of Fugaku seems to be 2 exaFLOPs. If we are generous and we account for 30% peak hardware utilization in training a transformer model, the same efficiency of an optimized large GPU cluster, it would take around 6000 days (20 years).

Fugaku seems to have cost 1B$, which leads me to believe that GPUs are much better at F16 flops per $ than the ARM SVE architecture they use. In any case, even if we use GPUs, it is clear we are some years away if we don't find a more efficient neural language model architecture.

Comment by sairjy on sairjy's Shortform · 2020-08-09T16:41:25.703Z · LW · GW

After GPT-3, is Nvidia undervalued?

GPT-3 made me update considerably on various beliefs related to AI: it is a piece of evidence for the connectionist thesis, and I think one large enough that we should all be paying attention.

There are 3 clear exponentials trends coming together: Moore's law, the AI compute/$ budget, and algorithm efficiency. Due to these trends and the performance of GPT-3, I believe it is likely humanity will develop transformative AI in the 2020s.

The trends also imply a fastly rising amount of investments into compute, especially if compounded with the positive economic effects of transformative AI such as much faster GDP growth.

In the spirit of using rationality to succeded in life, I start wondering if there is a "Bitcoin-sized" return potential currently untapped in the markets. And I think there is.

As of today, the company that stands to reap the most benefits from this rising investment in compute is Nvidia. I say that because from a cursory look at the deep learning accelerators markets, none of the startups, such as Groq, Graphcore, Cerebras has a product that has clear enough advantages over their GPUs (which are now almost deep learning ASICs anyway).

There has been a lot of debate on the efficient market hypothesis in the community lately, but in this case, it isn't even necessary: Nvidia stock could be underpriced because very few people have realized/believe that the connectionist thesis is true and that enough compute, data and the right algorithm can bring transformative AI and then eventually AGI. Heck, most people, and even smart ones, still believe that human intelligence is somewhat magical and that computers will never be able to __ . In this sense, the rationalist community could have an important mental makeup and knowledge advantage, considering we have been thinking about AI/AGI for a long time, over the rest of the market.

As it stands today, Nvidia is valued at 260 billion dollars. It may appear massively overvalued considering current revenues and income, but the impacts of transformative AI are in the trillions or tens of trillions of dollars, http://mason.gmu.edu/~rhanson/aigrow.pdf, and well the impact of super-human AGI are difficult to measure. If Nvidia can keeps its moats (the CUDA stack, the cutting-edge performance, the invested sunk human capital of tens of thousands of machine learning engineers), they will likely have trillions dollars revenue in 10-15 years (and a multi-trillion $ market cap) or even more if the world GDP starts growing at 30-40% a year.

Comment by sairjy on 10/50/90% chance of GPT-N Transformative AI? · 2020-08-09T10:58:08.030Z · LW · GW

I will use orthonormal definition of transformative AI: I read it as transformative AI would permanently alter world GDP growth rates, increasing them by 3x-10x. There is some disagreement between economists that is the case, i.e the economic growth could be slowed down by human factors, but my intuition says that's unlikely: i.e human-level AI will lead to much higher economic growth.

The assumption that I now think it is likely to be true (90% confident), that's possible to reach transformative AI by using deep learning, a lot of compute and data and the right architecture (which could also be different from a Transformer). Having said that, to scale models 1000x further there is significant engineering effort to be done, and it will take some time (improving model/data parallelism). We are also reaching the point where spending hundreds of millions of dollars of compute will have to be justified, so the ROI of these projects will be important. I have little doubt that they will be useful, but those considerations could slow down the exponential doubling of 3.4 months. For example, to train a 100 trillion parameters GPT model today (roughly 1 zettaflops day), it would require ad-hoc supercomputer with 100,000 A100s GPUs running for 100 days, costing a few billion $. Clearly, such cost would be justified and very profitable if we were sure to get a very intelligent AI system, but from the data we have now we aren't yet sure.

But I do expect 100 trillions models by the end of 2024, and there is a little chance that they could already be intelligent enough to be transformative AIs.

[10/50/90% || 2024/2030/2040] - 50% scenario: Microsoft is the first to release a 1 trillion parameter model in 2020, using it to power Azure NLP API. In 2021 and 2022, there is significant research into better architectures and other companies release their trillion parameters models: to do that they invested into even larger GPUs clusters. Nvidia datacenter revenue grows >100% year over year, and the company breaks the 1 trillion $ marketcap.

By end 2023, a lot of cognitive low-effort jobs, such as call centers, are completely automatized. It is also common to have personal virtual assistant. We also reach Level5 autonomous driving vehicles, but the models are so large that have to run remotely: low-latency 5G connections and many sparse GPUs clusters are the key. In 2024, the first 100 trillion model is unveiled and it can write novel math proofs/papers. This justifies a race towards larger models by all tech companies: billion $ or 10 billion $ scale private supercomputer/datacenter are build to train quadrillion-scale models (2025-2026). The world GDP growth rate start accelerating and it reaches 5-10% per year. Nvidia breaks above 5 trillion $ of marketcap.

By 2028-2030, large nations/blocks such as EU, China and US invest 100s billion $ into 1-5 Gigawatt-class supercomputers, or high-speed networks of them, to train and run their multi-quadrillion-scale model which reach superhuman intelligence and bring transformative AI. The world GDP grows 30%-40% a year. Nvidia is a 25 T$ company.

Comment by sairjy on Open & Welcome Thread - August 2020 · 2020-08-09T10:01:54.972Z · LW · GW

GPT-3 made me update considerably on various beliefs related to AI: it is a piece of evidence for the connectionist thesis, and I think one large enough that we should all be paying attention.

There are 3 clear exponentials trends coming together: Moore's law, the AI compute/$ budget, and algorithm efficiency. Due to these trends and the performance of GPT-3, I believe it is likely humanity will develop transformative AI in the 2020s.

The trends also imply a fastly rising amount of investments into compute, especially if compounded with the positive economic effects of transformative AI such as much faster GDP growth.

In the spirit of using rationality to succeded in life, I start wondering if there is a "Bitcoin-sized" return potential currently untapped in the markets. And I think there is.

As of today, the company that stands to reap the most benefits from this rising investment in compute is Nvidia. I say that because from a cursory look at the deep learning accelerators markets, none of the startups, such as Groq, Graphcore, Cerebras has a product that has clear enough advantages over their GPUs (which are now almost deep learning ASICs anyway).

There has been a lot of debate on the efficient market hypothesis in the community lately, but in this case, it isn't even necessary: Nvidia stock could be underpriced because very few people have realized/believe that the connectionist thesis is true and that enough compute, data and the right algorithm can bring transformative AI and then eventually AGI. Heck, most people, and even smart ones, still believe that human intelligence is somewhat magical and that computers will never be able to __ . In this sense, the rationalist community could have an important mental makeup and knowledge advantage, considering we have been thinking about AI/AGI for a long time, over the rest of the market.

As it stands today, Nvidia is valued at 260 billion dollars. It may appear massively overvalued considering current revenues and income, but the impacts of transformative AI are in the trillions or tens of trillions of dollars, http://mason.gmu.edu/~rhanson/aigrow.pdf, and well the impact of super-human AGI are difficult to measure. If Nvidia can keeps its moats (the CUDA stack, the cutting-edge performance, the invested sunk human capital of tens of thousands of machine learning engineers), they will likely have trillions dollars revenue in 10-15 years (and a multi-trillion $ market cap) or even more if the world GDP starts growing at 30-40% a year.

Comment by sairjy on Open & Welcome Thread—May 2020 · 2020-05-17T15:10:29.443Z · LW · GW

Wow! Beautiful!

Comment by sairjy on New Improved Lottery · 2020-05-17T13:22:50.289Z · LW · GW

There would be some handsome winners, as in the case of Bitcoin early adopters, also for this lottery. You mean average returns? In any case, expected average future returns should be zero for both.

It is similar enough, that no matter what fancy justification or narrative is painted over, most cryptocurrency investors own crypto because they believe it will make them rich. Possibly very fast. And that possibility can strike at any time.

Comment by sairjy on April Coronavirus Open Thread · 2020-04-18T16:01:00.716Z · LW · GW

I am not sure how it is possible that there are reports in the media claiming a low IFR (0.1%) when Lombardy has an official population fatality rate (i.e official COVID19 deaths over total population) of 0.12%, and unofficial one of 0.22% (measuring March and April all cause mortality there are ~10000 excess deaths) and a variability of up to 10x of casualties between towns more or less hit, indicating that only a small fraction (~10-20% imho) of the entire population was infected. I am pretty confident that the IFR is around 1% on average: it’s probably lower for younger people (0.2%) but as as high as 3% for people over 65. Moreover, Lombardy average age is less than the Italian average and the same as Germany. Even if there could be some age distribution difference they can’t explain the variation in the estimated IFR.

Comment by sairjy on New Improved Lottery · 2019-07-18T16:55:35.072Z · LW · GW

This essay had a very good insight for things to come: Bitcoin and other cryptocurrencies fit the above description.