# ResNets - Deep Residual Learning for Image Recognition **URL:** \ **Category:** Paper Reading Group\ **Created:** [September 7, 2021, 8:02pm UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461 "2021-09-07T20:02:44Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![lessing](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/lessing/32/200_2.png) [@lessing](https://community.wandb.ai/u/lessing)\ **Post date:** [September 7, 2021, 8:02pm UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/1 "2021-09-07T20:02:44Z")
Register: [wandb.me/prg](http://wandb.me/prg) ResNets Paper: [[1512.03385] Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385) Forum: [ResNets - Deep Residual Learning for Image Recognition - Paper Reading Group - W&B Community](http://wandb.me/resnets) Rewatch here: [W&B Paper Reading Group: ResNets - YouTube](https://www.youtube.com/watch?v=nspf00KpU-g&list=PLD80i8An1OEG_vpqwQgwH1gIxeb9r30u5&index=8&t=2s) ---
**Author:** ![girijesh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/girijesh/32/39_2.png) [@girijesh](https://community.wandb.ai/u/girijesh)\ **Post date:** [September 8, 2021, 3:16am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/2 "2021-09-08T03:16:06Z")
A general question is how does the same network like Resnet do image classification and also object detection and segmentation? ---
**Author:** ![vinayak\_nayak](https://avatars.discourse-cdn.com/v4/letter/v/439d5e/32.png) [@vinayak\_nayak](https://community.wandb.ai/u/vinayak_nayak)\ **Post date:** [September 8, 2021, 3:24am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/3 "2021-09-08T03:24:41Z")
I presume you’re talking about the pretrained resnet on imagenet task being used subsequently for the tasks you mentioned. The backbone body layers should have learned many generic features corresponding to the 1000 classes i.e. dog’s eyes, cat’s fur etc. which are really good features for many tasks like object detection So, we can use this backbone of conv layers and change the head to do regression of coordinates in place of classification… ---
**Author:** ![ramesh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/ramesh/32/98_2.png) [@ramesh](https://community.wandb.ai/u/ramesh)\ **Post date:** [September 8, 2021, 3:28am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/4 "2021-09-08T03:28:10Z")
Can the ResNet idea of Skip Connection be applied to Feed Forward Linear Network? Or does that not make much sense to see them in use? Or have we stopped building deep feed forward linear models? ---
**Author:** ![ramesh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/ramesh/32/98_2.png) [@ramesh](https://community.wandb.ai/u/ramesh)\ **Post date:** [September 8, 2021, 3:29am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/5 "2021-09-08T03:29:40Z")
Can we also talk about the Training method used. For example, in the paper they talk about having a warm-up LR of 0.01 and later moving to LR of 0.1. This seems really high LR than what we mostly use. Can we review that section? ---
**Author:** ![vinayak\_nayak](https://avatars.discourse-cdn.com/v4/letter/v/439d5e/32.png) [@vinayak\_nayak](https://community.wandb.ai/u/vinayak_nayak)\ **Post date:** [September 8, 2021, 3:29am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/6 "2021-09-08T03:29:56Z")
Are the identity layers not learnable? If not, then how can we claim deeper shouldn’t be worse than shallowe counterpart? If yes, then I understand. ---
**Author:** ![ramesh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/ramesh/32/98_2.png) [@ramesh](https://community.wandb.ai/u/ramesh)\ **Post date:** [September 8, 2021, 3:34am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/7 "2021-09-08T03:34:02Z")
As per my understanding, Identiy layers have no parameters. So there’s nothing to learn. But the Residual path can be learned to be Zeros, making the layer not relevant (Skip). Identify mapping primarily resolves the Vanishing Gradient problem through Identity connection. So Early layers get the gradients even if the network is really deep solving the primary problem introduced by deep networks. ---
**Author:** ![saiamrit](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/saiamrit/32/85_2.png) [@saiamrit](https://community.wandb.ai/u/saiamrit)\ **Post date:** [September 8, 2021, 3:37am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/8 "2021-09-08T03:37:25Z")
I have tried to summarise most of the points that are shared in the “How to read research papers” video by Professor Andrew Ng provides. Anyone interested can have a look at it here. > **[How to read Machine Learning and Deep Learning Research papers](https://saiamrit.github.io/technical-blog/research/reading_papers/2021/07/31/read-papers.html)** > > Tips on preparing Literature survey of a field and how to read a ML / DL research papers. The 3 pass method to read ML or DL research papers is discussed. ---
**Author:** ![ganesh3](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/ganesh3/32/187_2.png) [@ganesh3](https://community.wandb.ai/u/ganesh3)\ **Post date:** [September 8, 2021, 3:40am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/9 "2021-09-08T03:40:47Z")
Can you please explain with an input example of say 224 by 224 images, how the identity x is added back? ---
**Author:** ![vinayak\_nayak](https://avatars.discourse-cdn.com/v4/letter/v/439d5e/32.png) [@vinayak\_nayak](https://community.wandb.ai/u/vinayak_nayak)\ **Post date:** [September 8, 2021, 3:42am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/10 "2021-09-08T03:42:14Z")
Aren’t identity layer and residual path the same? Can you kindly elaborate the difference between the two? ---
**Author:** ![saiamrit](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/saiamrit/32/85_2.png) [@saiamrit](https://community.wandb.ai/u/saiamrit)\ **Post date:** [September 8, 2021, 3:50am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/11 "2021-09-08T03:50:39Z")
As per my understanding. The skip connection learns the identity mapping and anything else that in the main stem of the network is learning(i.e: output of a layer is F(x)+x, so x is what is coming from the identity mapping, the F(x) that is learnt is called the residual thing learnt). Identity is like the major thing that is passed from the input and the F(x) is anything residual or anything additional over the identity that the network learns. So it is basically like we atleast learn an identity mapping in worst case(at a large depth where the gradients are 0 and learning from the main stem is almost 0). Addition to identity we are also learning something more. ---
**Author:** ![girijesh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/girijesh/32/39_2.png) [@girijesh](https://community.wandb.ai/u/girijesh)\ **Post date:** [September 8, 2021, 3:52am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/12 "2021-09-08T03:52:07Z")
How to decide whether to use skip connections ? The intuition behind different blocks? ---
**Author:** ![ramesh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/ramesh/32/98_2.png) [@ramesh](https://community.wandb.ai/u/ramesh)\ **Post date:** [September 8, 2021, 3:54am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/13 "2021-09-08T03:54:18Z")
Can we talk about this? ---
**Author:** ![vinayak\_nayak](https://avatars.discourse-cdn.com/v4/letter/v/439d5e/32.png) [@vinayak\_nayak](https://community.wandb.ai/u/vinayak_nayak)\ **Post date:** [September 8, 2021, 3:54am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/14 "2021-09-08T03:54:58Z")
In fastbook, we saw as we add conv layers, we increase the channels when we have higher strides. Along those lines, in resnet-34 I can see that there’s several conv layers which have same channels in input and output and then they do pooling and increase feature maps. ![image](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/f433eaa8801a9ff63ba90247067cfd069acd4ba5.png) Is it a rule that if we have stride = 1, we should keep same feature maps across conv layers and when we do pooling we increase feature maps… ---
**Author:** ![vrindaprabhu](https://avatars.discourse-cdn.com/v4/letter/v/97f17d/32.png) [@vrindaprabhu](https://community.wandb.ai/u/vrindaprabhu)\ **Post date:** [September 8, 2021, 3:59am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/15 "2021-09-08T03:59:07Z")
Just a side note. Kaiming He is one of my favourite researchers. Most of the papers he is involved in are really great and is used at a massive scale. Few of his contributions - - The He initialisation - Resnet - Faster and Mask RCNN - Focal Loss and Pyramid Networks ---
**Author:** ![deepak\_mewada](https://avatars.discourse-cdn.com/v4/letter/d/22d042/32.png) [@deepak\_mewada](https://community.wandb.ai/u/deepak_mewada)\ **Post date:** [September 8, 2021, 4:00am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/16 "2021-09-08T04:00:12Z")
Could you please again explain what the dashed connection mean? ---
**Author:** ![vinayak\_nayak](https://avatars.discourse-cdn.com/v4/letter/v/439d5e/32.png) [@vinayak\_nayak](https://community.wandb.ai/u/vinayak_nayak)\ **Post date:** [September 8, 2021, 4:02am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/17 "2021-09-08T04:02:28Z")
So essentially, the skip connection is learnable and it tries to learn identity mapping so that most of the input information can flow in to subsequent layers and kind of circumvent the vanishing gradient also. The residual path will try learning what any plain model is trying to do i.e. features like eyes, fur etc. in imagenet (for cats/dogs etc.) Is my understanding correct? ---
**Author:** ![saiamrit](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/saiamrit/32/85_2.png) [@saiamrit](https://community.wandb.ai/u/saiamrit)\ **Post date:** [September 8, 2021, 4:14am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/19 "2021-09-08T04:14:41Z")
Ya there are papers which add one conv layer in the skip connection and keep it minimal but most skip connections are simple identity mapping with no parameters in them and hence there is no learning involved. It’s just a simple path that passes the input as it is. Learning only takes place in the residual branch. ---
**Author:** ![saiamrit](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/saiamrit/32/85_2.png) [@saiamrit](https://community.wandb.ai/u/saiamrit)\ **Post date:** [September 8, 2021, 4:17am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/20 "2021-09-08T04:17:56Z")
[https://theaisummer.com/skip-connections/#conclusion](https://theaisummer.com/skip-connections/#conclusion) hope this helps. ---
**Author:** ![amanarora](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/amanarora/32/59_2.png) [@amanarora](https://community.wandb.ai/u/amanarora)\ **Post date:** [September 9, 2021, 12:50am UTC](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461/21 "2021-09-09T00:50:29Z")
> [@VinayakNayak](#): > > the skip connection is learnable and it tries to learn identity mapping Skip connection is not learnable. This will be clearer when we code things up. [Next page](https://community.wandb.ai/t/resnets-deep-residual-learning-for-image-recognition/461.md?page=2)