# ML Sprint: Transformers Wiki! **URL:** \ **Category:** Show the Community!\ **Created:** [September 10, 2021, 12:17pm UTC](https://community.wandb.ai/t/ml-sprint-transformers-wiki/492 "2021-09-10T12:17:38Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![bhutanisanyam1](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/bhutanisanyam1/32/18_2.png) [@bhutanisanyam1](https://community.wandb.ai/u/bhutanisanyam1)\ **Post date:** [September 10, 2021, 12:17pm UTC](https://community.wandb.ai/t/ml-sprint-transformers-wiki/492/1 "2021-09-10T12:17:38Z")
Hi Everybody! As part of our first community hangout, we’re excited to be hosting a few sprints. This is one of the same: The plan with ML Sprints is to run week-long activities where our community will contribute to projects. This is one of three wikis that we’re inviting you to contribute to! This Wiki is meant to serve as a collection of best resources to learn about Transformer models and their applications. This is a wiki! This means all of you can edit it, please do so! Papers: [Attention Is All You Need (2017)](https://arxiv.org/abs/1706.03762) [End-to-End Object Detection with Transformers (2020)](https://arxiv.org/abs/2005.12872) [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2021)](https://arxiv.org/abs/2010.11929) Blog posts: [The Annotated Transformer](https://nlp.seas.harvard.edu/2018/04/03/attention.html) [Transformer Deep Dive](https://wandb.ai/carlolepelaars/transformer_deep_dive/reports/Transformer-Deep-Dive--VmlldzozODQ4NDQ?galleryTag=posts) [The illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) Explanation videos: [Attention is All you need by Yannic Kilcher](https://www.youtube.com/watch?v=iDulhoQ2pro) [GPT-2 by Yannic Kilcher](https://www.youtube.com/watch?v=u1_qMdb0kYU) [BERT by Yannic Kilcher](https://www.youtube.com/watch?v=-9evrZnBorM) [RoBERTa by Yannic Kilcher](https://www.youtube.com/watch?v=-MCYbmU9kfg) Kaggle Notebooks: [Utilizing Transformer Representations Efficiently](https://www.kaggle.com/rhtsingh/utilizing-transformer-representations-efficiently) [On Stability of Few-Sample Transformer Fine-Tuning](https://www.kaggle.com/rhtsingh/on-stability-of-few-sample-transformer-fine-tuning) [Speeding up Transformer w/ Optimization Strategies](https://www.kaggle.com/rhtsingh/speeding-up-transformer-w-optimization-strategies) ---
**Author:** ![harveenchadha](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/harveenchadha/32/575_2.png) [@harveenchadha](https://community.wandb.ai/u/harveenchadha)\ **Post date:** [September 12, 2021, 7:56am UTC](https://community.wandb.ai/t/ml-sprint-transformers-wiki/492/2 "2021-09-12T07:56:04Z")
If you just want to get the hang of transformers in one post it would definitely be this one from Jay Alammar [The illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) ---
**Author:** ![harveenchadha](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/harveenchadha/32/575_2.png) [@harveenchadha](https://community.wandb.ai/u/harveenchadha)\ **Post date:** [September 12, 2021, 7:59am UTC](https://community.wandb.ai/t/ml-sprint-transformers-wiki/492/3 "2021-09-12T07:59:28Z")
This playlist from Ms. Cofee Bean is informative as well. [The Transformer explained by Ms. Coffee Bean](https://www.youtube.com/playlist?list=PLpZBeKTZRGPNdymdEsSSSod5YQ3Vu0sKY) I specifically like this diagram from Chirs McCormick on what you need to know to understand transformers: ![Screenshot from 2021-09-12 13-32-23](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/592ff23fa447cd2f78c947a05b7f5a01e1944573.png) ---
**Author:** ![aadil](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/aadil/32/152_2.png) [@aadil](https://community.wandb.ai/u/aadil)\ **Post date:** [September 13, 2021, 1:58am UTC](https://community.wandb.ai/t/ml-sprint-transformers-wiki/492/4 "2021-09-13T01:58:09Z")
1. A great lecture by Dr. Rachel Tomas, about the fundamental idea behind Transformers. [YouTube link](https://www.youtube.com/watch?v=AFkGPmU16QA&list=WL&index=31) 2. “Attention is All You Need” paper read through by Yannic Kilcher. [YouTube link](https://www.youtube.com/watch?v=iDulhoQ2pro) ---
**Author:** ![dchanda](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dchanda](https://community.wandb.ai/u/dchanda)\ **Post date:** [September 14, 2021, 2:34pm UTC](https://community.wandb.ai/t/ml-sprint-transformers-wiki/492/5 "2021-09-14T14:34:57Z")
These are some of the paper walk-throughs everyone should go through at least once: 1. [Attention is All you need by Yannic Kilcher](https://www.youtube.com/watch?v=iDulhoQ2pro) 2. [GPT-2 by Yannic Kilcher](https://www.youtube.com/watch?v=u1_qMdb0kYU) 3. [BERT by Yannic Kilcher](https://www.youtube.com/watch?v=-9evrZnBorM) 4. [RoBERTa by Yannic Kilcher](https://www.youtube.com/watch?v=-MCYbmU9kfg) These kernels are good to learn from the application point of view: 1. Different ways to utilize transformer representations [Utilizing Transformer Representations Efficiently](https://www.kaggle.com/rhtsingh/utilizing-transformer-representations-efficiently) 2. Stabilizing training of transformer models [On Stability of Few-Sample Transformer Fine-Tuning](https://www.kaggle.com/rhtsingh/on-stability-of-few-sample-transformer-fine-tuning) 3. Speeding up transformer training [Speeding up Transformer w/ Optimization Strategies](https://www.kaggle.com/rhtsingh/speeding-up-transformer-w-optimization-strategies)