...the authors’ floorplan solutions have been incorporated into the chip designs for Google’s next-generation artificial-intelligence processors. This means that the solutions are good enough for millions of copies to be printed on expensive, cutting-edge silicon wafers. We can therefore expect the semiconductor industry to redouble its interest in replicating the authors’ work, and to pursue a host of similar applications throughout the chip-design process.
My current guess is that this is not a big deal. Surely these AI-optimizations will result in something like 10% improvement in AI-training-FLOPS-per-dollar, not 100%+, so they won't really change timelines or anything else strategically important. And it won't even be 10% improvement every year from now on, but more like 10% this year, a further 5% next year, a further 2.5% the year after that, etc. as the low-hanging fruit from floorplan optimization is picked. OTOH, this plausibly will reduce the time it takes to design new chips by a lot... but I'd be surprised if that was the main bottleneck anyway. I would have thought ramping up production was the main bottleneck.
I know very little about the chip industry though. Anyone care to correct me?
Just made this account to answer this. Source: I've worked in physical design/VLSI and CPU verification, and pretty regularly deal with RTL.
TL;DR - You're right-- it's not a big deal, but it simultaneously means more and less than you think.
Jump to "What It Means" if you already understand the problem.
First, let me talk about about the purpose of floorplanning. The author's mention it a little bit, but it's worth repeating.
Placement optimizations of this form appear in a wide range of science and engineering applications, including hardware design, city planning, vaccine testing and distribution, and cerebral cortex layout.
Much like a city, an SoC (system-on-chip) has lots of agents that transfer data to each other. If a mayor has to get to city hall, the library, the post office, the locksmith, the school, the burger joint, etc., how do you best place the buildings to get the shortest path to each of them? Suppose suddenly the librarian wants to first go to school, then the post office, and also a burger because they have 20% off. How do you position that requirement along with the mayor's requirement? Do you prioritize the mayor? What if he wants a burger too? What if it's not guaranteed the number of paths the mayor will take before returning to city hall Etc. etc.
As you probably know, placement in general is an NP-complete problem. Tools for this exist, and/or you can do it manually, but much like city planning, it gets very complicated very fast. These tools (if you wanna sound cool, call them PnR tools (place-and-route)) take foooreeever to run (it's quite common to let a tool run for a week) and are critical in the holistic design lifecycle-- more on that later.
Enter this paper. Honestly, they don't do any revolutionary stuff-- CNNs, ReLu, weight adjustment-- or rather, it's revolutionary because it's applied to PnR for the first time that I've seen at least (which, in hindsight, is pretty obvious. Pulling up the GUI for the tool, it's literally just a grid, exactly like a city, with its own centers and everything. Still cool nevertheless).
Let's talk about results!
I don't know how to do tables in comments, so bear with the formatting-- here are the results for one test they did:
Note: I left out "Congestion" and "wire length" because those are metrics that tbh don't really matter
Total area (µm 2 )
Total power (W)
Don't worry what wns and tns exactly mean (here are a few resources). Just know that they are essentially a measure of how short a "path" is between "buildings". The smaller it is, the better, because it means our mayor can travel less distance to get his burger.
Area and power are relatively explanatory-- essentially, how big is your city + all the roads you've built, and how much energy does it take to run it all. Again, the smaller the better.
What It Means
These are good results! We've just built roads that are twice as short vs. our manual methods! (23.3 vs. 47.6). But, I want to provide my opinion for why it's even worse than you think (i.e., I don't even think it would provide a 1% increase in perf, much in the same way that increasing CPU GHz doesn't do that much-- it's inherently limited), but also much better
For why it's worse-- consider again city planning. Suppose we take this to the extreme and the burger joint, library, post office, etc are all literally inside the same building as City Hall (i.e. no roads exist). First, his arteries will certainly get clogged passing by a McDonalds, but ultimately-- How much performance/time saved does the mayor really save?
I would argue that, while it depends on how convoluted the city was initially, there's a limit to how much you can shrink the roads and place the buildings. While these planning efforts are very much important to strive for, it's not the real bottleneck.
Furthermore, what if this travel time was time simultaneously being well-spent already? For instance-- perhaps he checked his emails walking to the post office. Maybe he called his mother. Maybe he brought his meeting notes to practice a speech. The point is-- this travel time is not really saved: just reallocated.
Note: CPUs do this a lot, e.g. while a memory request is occurring, they just switch to do some other tasks. This is also (to vastly oversimplify) essentially why frequency scaling no longer had immense payoffs as it did 30 years ago.
Now that I've killed your enthusiasm, let me tell you why it's also better than you think with this quote.
We show that our method can generate chip floorplans that are comparable or superior to human experts in under six hours, whereas humans take months to produce acceptable floorplans for modern accelerators
I mentioned earlier that designers heavily rely on PnR tools not only prior to tapeout, but as tools to iterate over (e.g. can I mux this more efficiently? Do I really need this logic in the critical path? Can this "building" be shifted over? etc.) As these tools take longer as our designs become more complex, it ultimately results in a longer feedback loop-- again, a week sometimes-- and personally, I really like instant gratification, so it's definitely a bit annoying.
And this is why it's potentially better-- it's indicative of a step towards freeing up resources of what I feel is a massive cost to many semiconductor companies. Not just for better and tighter feedback loops, but because these PnR/physical design/EDA tool teams are massive. Like, hundreds of people sometimes. And these people ultimately have the final signoff for lots of tapeouts, and determine timelines for hardware companies.
Go 5 years in the future, and give them a tool that improves engineer productivity 100x? Honestly, that'd be insane. For me personally, but also for my colleagues. (Honestly, not sure what I'd do with that extra time. I currently just cook stuff while I'm blocked- :) )
So, that's why I think it's both better and worse than you think.
Awesome, thanks! And welcome to LW! I found this very helpful and now have some follow-up questions if you don't mind. :)
1. How does this square with Zac's answer below? It on the surface seems to contradict what you say; after all, it proposes 10x-1000x improvements to AI stuff whereas you say it won't even be 1%! I think I can see a way that your two answers can be interpreted as consistent, however: You identify the main benefit of this tech as reducing the clock time it takes for engineers to come up with a new good chip design. So even if the new design is only 1% better than the design the engineers would have come up with, if it happens a lot faster, that's a big deal. Why is it a big deal? Well, as Zac said, it means the latest AI architectures can be quickly supplemented by custom chips, and in general custom chips provide 10x - 1000x speedups. Would you agree with this synthesis?
2. I'd be interested in your best guess for what the median X's and Y's in this sentence are: "In about X years, we'll be in a regime where the latest AI models are run on specialized hardware that provides a factor-of-Y speedup over today's hardware."
3. ETA: Maybe another big implication of this technology is that it'll lower the barrier to entry for new chipmakers? Like, maybe 5 years from now there'll be off-the-shelf AI techniques that let people design cutting-edge new chips, and so China and Russia and India and everyone will have their own budding chip industry supported by generous government subsidies. Or maybe not -- maybe most of the barriers to entry have to do with manufacturing talent rather than design talent?
Circuit design is the main bottleneck for use of field-programmable gate arrays. If fully-automated designs become good enough, we could see substantial gains from having optimising compilers output a gate layout rather than machine code for an xPU or specific accelerator. We already have some such compilers, and this looks like a meaningful step towards handling non-toy-scale problems with them.
The main change here wouldn't be so much training speed - we already have TPUs etc. to accelerate current workloads, and fabricating a new design as ASICs rather than FPGA layouts takes months-to-years at scale - but rather the latency with which we can try out custom hardware for novel ML paradigms such as transformers. What is to transformers as TPUs are to CNNs? Specifically for novel tasks, this could be a 10x-1000x speedup, and 2x-50x speedup for existing workloads... though I understand they're bottlenecked more on data movement between nodes than compute.
TLDR: a small step in a high-long-term-impact trend.
(Source: while I'm not a hardware specialist, I've worked with the PyMTL team at Cornell on verification and validation of their Python-to-Verilog-to-silicon hardware design tools, followed high-level developments in custom compute hardware for around a decade, and worked on peta-scale supercomputing for a few years.)
I think this is incorrect. You might imagine that CPU->GPU and GPU->TPU transitions were steps up a tall log-scale tech ladder, in the way that Moore's-law doublings were, with many more steps still possible in theory. But this is not the case, because the metric these transitions were improving on was "fraction of transistors which are dedicated to useful compute" (as opposed to extracting parallelism from a serial instruction stream, or computing unnecessary low-order bits on overly-wide floating point). This metric has a hard upper limit, at 100%, and I don't think there's even one order of magnitude left between current utilization and that limit.
No, I think we mostly agree - I'd expect TPUs to be with say 4x of practically optimal for the things they do. The remaining one OOM I think is possible for non-novel tasks has more to do with specialisation, eg model-specific hardware design, and that definitely has an asymtote.
The interesting case is if we can get TPU-equivalent hardware days after designing a new architecture, instead of years after, because (IMO) 1,000x speedups over CPUs are plausible.
Thanks! As I understand it, you are saying (a) In general it's not hard to get 10x - 1000x speedups (as measured by flops per dollar? Or better yet, performance per dollar?) for very specific/narrow AI applications, if you design custom hardware for it, and (b) when AIs automate more of the chip design process, it'll take less time and money to design custom hardware for stuff, so e.g. when Transformer 2.0 comes out, less than a year later there'll be specialized hardware for it that makes it even better. Is this a fair summary?
If so, I'd be interested to hear why you said 10x - 1000x, as opposed to 2x or 1.1x. Has specialized hardware given 100x improvements in performance-per-dollar in the past? For neural nets in particular?
Yes, that's a fair summary - though in "not hard ... if you design custom hardware" the second clause is doing a lot of work.
As to the magnitude of improvement, really good linear algebra libraries are ~1.5x faster than 'just' good ones, GPUs are a 5x-10x improvement on CPUs for deep learning, and TPUs 15x-30x over Google's previous CPU/GPU combination (this 2018 post is a good resource). So we've already seen 100x-400x improvement on ML workloads by moving naive CPU code to good but not hyper-specialised ASICs.
Truly application-specific hardware is a very wide reference class, but I think it's reasonable to expect equivalent speedups for future applications. If we're starting with something well-suited to existing accelerators like GPUs or TPUs, there's less room for improvement; on the other hand TPUs are designed to support a variety of network architectures and fully customised non-reprogrammable silicon can be 100x faster or more... it's just terribly impractical due to the costs and latency of design and production with current technology.
For example, with custom hardware you can do bubblesort in O(n) time, by adding a compare-and-swap unit between the memory for each element. Or with a 2D grid of these, you can pipeline your operations and sort lists in O(1) time and O(n) latency! Matching the logical structure of your chip to the dataflow of your program is beyond the scope of this article (which is "just" physical structure), but also almost absurdly powerful.
In the short term: moderately big deal. The chip industry is currently in rather a lot of flux; Intel was supplanted as leader in transistor size by TSMC; Apple is running with their own chip designs; China's monopoly on rare earth mineral processing has come under scrutiny again. This has provoked a boom in new development as a consequence. Even a small improvement in the design and manufacture of these facilities weighs a lot; because the chip industry is so important and so centralized, moderately big deal is essentially the floor for any actual development within it.
In the long term: big deal. This is not an opinion shared by anyone else as far as I can tell, but it feels very clear to me that "people use ML for this application" is the threshold at which the hardware overhang is almost immediately accessible to AGI. At that point the AGI is literally an upgrade operation, as opposed to having to go through the entire process of converting a workflow to something an AI of any type at all can work on. To be more concrete: I expect any kind of AI-driven takeover to control all of the currently-uses-ML industries before taking over any that do not; and I expect that within currently-uses-ML industries the order will be determined largely by how saturated they are with tools of that kind.
Thanks! So would you agree with my suggestion in the comment above that this will lower barriers to entry and allow many new chipmakers into the market?
I'm not yet convinced re AGI takeover. I feel like this sort of chip optimization is the sort of thing more suited to narrow AI than to AGI. Maybe in the long run additional optimizations could be gleaned by drawing on general world-knowledge that includes e.g. understanding of fractals and biological systems and city planning and so forth, but I feel like that would be only marginally better than what a narrow AI trained on simulated chips would produce.
I would agree; anything that cuts months and potentially hundreds of people makes it easier for new entrants. Further, the trend appears strongly in the direction of outsourcing, as even Intel will now build other's designs. I see no reason why this could not be done on a contracting basis as well. The primary obstacle is the low appetite for the private sector to make large investments in physical things. Intel and TSMC's new investments are largely defense motivated.
I agree that this particular sort of chip optimization is suited more for narrow AI than AGI; my claim is rather that anything which employs narrow AI is more vulnerable to AGI takeover. It seems likely to me that AGI would have an interest in production of processing power, so it seems like automating the steps is lowering the threshold.
I also consider that this kind of development is exactly what the CAIS model predicts. If CAIS is a system of narrow AIs, including coordinator/management AIs, why won't a misaligned or malevolent coordinator AI from interacting with already existing narrow AIs? The malevolent case could be as straightforward as an ML redux of Stuxnet.
All of this rests pretty heavily on the crux that once one AI runs a task, it is easy to replace it with another AI; if this effect is weak, or I am completely wrong and it is in fact harder, then the chain of logic falls apart.
I see this as analogous to the points you made in the embodied intellectual property post comments [LW(p) · GW(p)]: what we think we are doing is making more efficient use of resources, but what we are actually doing is engaging in a tradeoff of gaining time and money in exchange for living with a more opaque method of controlling the work. Within this more opaque method, additional risks lie. A more specific analogy to the Portuguese sailing technology commentary [LW(p) · GW(p)] in the Conquistadors post feels achievable, but it isn't coming together for me yet.
One additional thing I'd be interested in is AI-assisted solution of the differential equations behind better masks for EUV lithography. It seems naively like another factor of 2-ish in feature size is just sitting out there waiting to be seized, though maybe I'm misunderstanding what I've heard about switching back to old-style masks with EUV.