FN landola, SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size, 2016
設計目標
- 希望簡化網路複雜度,同時達到public網路識別精度,話句話說,主要是為了降低CNN模型參數數量而設計的。
設計原則
- 替換3x3的卷積kernel為1x1的卷積kernel
- 可以讓參數縮小9x。但是為了不影響識別精度,而是一部分用3x3,一部分用1x1。
- 減少輸入3x3卷積的input feature map數量
- 把原本一層Conv分解為兩層,並且封裝為一個Fire Module
- 減少pooling
- 有許多架構有實作過
- 例如
- GoogLeNet
- Deep Residual Learning
Fire Module
- 一層Conv層變成兩層:squeeze層+expand層,各自帶上ReLU激活層。
- squeeze層
- 1x1卷積kernel,計為s1X1
- s11 < input map number, 即滿足設計原則(2)
- expand層
- 1x1卷積kernel,計為e1X1
- 3x3卷積kernel,計為e3X3
- 在channel維度把1x1和3x3卷積output feature map拼接起來
- squeeze層
Overall Architecture
Input name/type | output size | filter size/stride(if not a fire layer) | depth | s1X1(#1x1 squeeze) | e1X1(#1x1 expand) | e3X3(# 3x3 expand) | s1X1 sparsity | e1X1 sparsity | e3X3 sparsity | # bits | # pararmeter before pruning | # parameter after pruning |
input image | 224x224x3 | - | - | |||||||||
conv1 | 111x111x96 | 7x7/2 (x96) | 1 | 100%(7x7) | 6bit | 14,208 | 14,208 | |||||
maxpool1 | 55x55x96 | 3x3/2 | 0 | |||||||||
fire2 | 55x55x128 | 2 | 16 | 64 | 64 | 100% | 100% | 33% | 6bit | 11,920 | 5,746 | |
fire3 | 55x55x128 | 2 | 16 | 64 | 64 | 100% | 100% | 33% | 6bit | 12,432 | 6,258 | |
fire4 | 55x55x256 | 2 | 32 | 128 | 128 | 100% | 100% | 33% | 6bit | 45,344 | 20,646 | |
maxpool4 | 27x27x256 | 3x3/2 | 0 | |||||||||
fire5 | 27x27x256 | 2 | 32 | 128 | 128 | 100% | 100% | 33% | 6bit | 49,440 | 24,742 | |
fire6 | 27x27x384 | 2 | 48 | 192 | 192 | 100% | 50% | 33% | 6bit | 104,880 | 44,700 | |
fire7 | 27x27x384 | 2 | 48 | 192 | 192 | 50% | 100% | 33% | 6bit | 111,024 | 46,236 | |
fire8 | 27x27x512 | 2 | 64 | 256 | 256 | 100% | 50% | 33% | 6bit | 188,992 | 77,581 | |
maxpool8 | 13x12x512 | 3x3/2 | 0 | |||||||||
fire9 | 13x13x512 | 2 | 64 | 256 | 256 | 50% | 100% | 30% | 6bit | 197,184 | 77,581 | |
conv10 | 13x13x1000 | 1x1/1 (x1000) | 1 | 20%(3x3) | 6bit | 513,000 | 103,400 | |||||
avgpool10 | 1x1x1000 | 13x13/1 | 0 | |||||||||
activations | parameters | compression info | 1,248,424 | 421,098 |
reference:https://arxiv.org/pdf/1602.07360.pdf