--- title: Making Transformers Efficient description: Learn how to make Transformer models more efficient and reduce their size for low-resource hardware. Explore different methods like attention mechanism optimization, pruning attention heads, and assessing the benefits of model pruning. image: https://www.embedl.com/hubfs/43-min-1.png --- # Making Transformers Efficient While the Transformer architecture has taken the natural language processing (NLP) world by storm – witness the amazing ChatGPT, GPT-4 and related technologies – it has also been recognized that they are monster models with over a trillion parameters(!) resulting in huge energy costs and carbon footprint, see the figure.   ![Energy costs and carbon footprint](https://www.embedl.com/hubfs/image-png-2.png)   There is thus a pressing need to make Transformer based models more efficient – by reducing their size and optimize them for low resource hardware. The main mechanism behind the Transformer architecture is the multi-headed attention mechanism ([as we explained in the first of this series of posts](https://www.embedl.com/knowledge/enter-the-transformer)). Thus, a lot of research has already been devoted to reducing the cost of the attention mechanism. There is a continuously updated [survey](https://arxiv.org/pdf/2009.06732.pdf) with a taxonomy of different versions of Transformers optimized in different ways. The most glaring feature about the attention mechanism is that it scales quadratically with the number of tokens in the sentence. Several attempts have been made to address this issue, from simple engineering hacks like keeping smaller windows to more sophisticated methods to achieve *sparsity* - Low rank factorizations of the attention matrix - K-means clustering of tokens - Locality sensitive hashing (LSH) techniques - Orthogonal random features - Kernel methods The survey linked above goes over several such methods. While interesting, a huge sticking point is that none of these methods has yet shown its advantage on real world problems beyond the toy examples in those papers. This is a point we return to at the end. Another way to reduce the size of models is to prune the number of attention heads as in the paper: [Are Sixteen Heads Really Better than One](https://arxiv.org/abs/1905.10650)? This and other papers use different methods to access the importance of different attention heads and come to the conclusion that some heads are really important and play very specific roles whereas the rest can be safely pruned without affecting the performance significantly. Finally, it is important to keep in mind that one needs to be very careful in assessing the benefits of compression methods. Simply using FLOPs as a proxy for efficiency is often quite misleading, a point made for Transformers in the paper [Efficiency Misnormer](https://arxiv.org/abs/2110.12894). They make the point that simply using a single measure is often misleading, especially when that measure is just parameters count. The actual efficiency is a much more complex matter, and of course it is very closely tied to the underlying hardware platform. This is very much the philosophy of Embedl – our methods take a holistic approach considering several different measures including accuracy, latency and cost and is done in a hardware-aware manner..       [Insider](https://www.embedl.com/knowledge/tag/insider), [Blog-](https://www.embedl.com/knowledge/tag/blog-) ##### Like it? Share it: [LinkedIn ](https://www.linkedin.com/shareArticle?mini=true&url=https://www.embedl.com/knowledge/making-transformers-efficient) [Facebook ](https://www.facebook.com/sharer/sharer.php?u=https://www.embedl.com/knowledge/making-transformers-efficient) ### You may also like [Enter the Transformer! ](https://www.embedl.com/knowledge/enter-the-transformer) ##### [Enter the Transformer!](https://www.embedl.com/knowledge/enter-the-transformer) 4 July, 2023 “The Transformer has taken over AI”, says Andrej Karpathy, former Director of AI Research at Tesla, in a recent episode ... [Read this article](https://www.embedl.com/knowledge/enter-the-transformer) [LinkedIn ](https://www.linkedin.com/shareArticle?mini=true&url=https://www.embedl.com/knowledge/enter-the-transformer) [Facebook ](https://www.facebook.com/sharer/sharer.php?u=https://www.embedl.com/knowledge/enter-the-transformer) [Vision Transformers vs CNNs at the Edge ](https://www.embedl.com/knowledge/vision-transformers-vs-cnns-at-the-edge) ##### [Vision Transformers vs CNNs at the Edge](https://www.embedl.com/knowledge/vision-transformers-vs-cnns-at-the-edge) 12 January, 2024 Enter the Transformer “The Transformer has taken over AI”, says Andrej Karpathy, (Former) Director of AI at Tesla, in a ... [Read this article](https://www.embedl.com/knowledge/vision-transformers-vs-cnns-at-the-edge) [LinkedIn ](https://www.linkedin.com/shareArticle?mini=true&url=https://www.embedl.com/knowledge/vision-transformers-vs-cnns-at-the-edge) [Facebook ](https://www.facebook.com/sharer/sharer.php?u=https://www.embedl.com/knowledge/vision-transformers-vs-cnns-at-the-edge) [Amazon releases AZ1 Neural Edge Processor ](https://www.embedl.com/knowledge/amazon-releases-az1-neural-edge-processor) ##### [Amazon releases AZ1 Neural Edge Processor](https://www.embedl.com/knowledge/amazon-releases-az1-neural-edge-processor) 3 October, 2020 On September 24th, Amazon held its annual hardware event, introducing their new generation of smart home hubs, speakers ... [Read this article](https://www.embedl.com/knowledge/amazon-releases-az1-neural-edge-processor) [LinkedIn ](https://www.linkedin.com/shareArticle?mini=true&url=https://www.embedl.com/knowledge/amazon-releases-az1-neural-edge-processor) [Facebook ](https://www.facebook.com/sharer/sharer.php?u=https://www.embedl.com/knowledge/amazon-releases-az1-neural-edge-processor) ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Devdatt Dubhashi", "url" : "https://www.embedl.com/knowledge/author/devdatt-dubhashi" }, "dateModified" : "2024-01-18T09:54:59.652Z", "datePublished" : "2023-07-14T09:04:36.000Z", "headline" : "Making Transformers Efficient", "image" : [ "https://www.embedl.com/hubfs/43-min-1.png" ], "mainEntityOfPage" : { "@id" : "https://www.embedl.com/knowledge/making-transformers-efficient", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://www.embedl.com/hubfs/logotype%20embedl.png" }, "name" : "Embedl" } } ```