g SrSSKrSSKJrJrJr SSKJrJrJ r J r J r J r J r Jr SSKJr SSKrSSKJr SSKJrJr SSKJr SS KJrJrJrJrJrJrJ r J!r!J"r"J#r#J$r$ SS K%J&r& /S Qr'SS jr(0S \("SS9_S\("SS9_S\("SSSS9_S\("SSS9_S\("SSS9_S\("SSS9_S\("S SS9_S!\("S"SS9_S#\("S$SS9_S%\("S&SS9_S'\("S(SS9_S)\("S*S+SSS,S-S.9_S/\("S0SSS1S29_S3\("S0SSS1S29_S4\("S0SSS1S29_S5\("S0SSS1S29_S6\("S0SSS1SS79_r)\"S8S955r*\"S:S;55r+SS<jr,SS=\\-\-4S>\ \.\\.4S?\/S@\\*4SAjjr0\1"S0S\+"\*"SBSSCSDSS-SE9\*"SBSDSFSDSS-SE9\*"SGSHSISDSSJSE9\*"SGSKSISDSSLSE9\*"SGSMSISSSLSE94SNSSOSP9_S\+"\*"SBSSCSDSS-SE9\*"SBSDSFSDSS-SE9\*"SGSHSISDSSJSE9\*"SGSMSISDSSLSE9\*"SGSSISSSLSE94SNSSOSP9_S \+"\*"SBSSQSDSS-SE9\*"SBSRSQSDSS-SE9\*"SGSSSTSDSSJSE9\*"SGSDSUSDSSLSE9\*"SGSSVSSSLSE94SWSSXSP9_S\+"\,"SYSZS[9S\S]S^9_S\+"\,"S_S`9S\S]S^9_S\+"\,"SaS`9S\S]S^9_S\+"\,"SaSMSb9S\S]S^9_S!\+"\,"ScS`9S\S]S^9_S#\+"\,"ScSMSb9S\S]S^9_S%\+"\,"SdS`9S\S]S^9_S'\+"\,"SdSMSb9S\S]S^9_S)\+"\*"SGSDSVSSNSJSE9\*"SGSMSeSDSNSJSE9\*"SGSHSfSDSNSJSE9\*"SGSMSfSDSS-SE94SCSgSShSiSj9_S/\+"\*"SkSSVSSS-\1"5Sl9\*"SGSMSeSDSNSJSE9\*"SGSHSfSDSNSJSE9\*"SGSMSfSDSS-SE94SCSmSShSi\1"SnSo9Sp9_S3\+"\*"SkSRSVSSJSq9\*"SkSMSeSDSJSq9\*"SGSHSrSDSJSq9\*"SGSRShSDSJSq94S]SsSSt\1"SuSnSv9Sw9_S4\+"\*"SGSRSVSSJSq9\*"SGSMSeSDSJSq9\*"SGSHSrSDSJSq9\*"SGSRShSDSJSq94S]SsSSxSy9_S5\+"\*"SGSRSVSSNSJSE9\*"SGSMSeSDSNSJSE9\*"SGSHSrSDSNSJSE9\*"SGSRShSDSNSJSE94S]SsSzSSiSxS{9_S6\+"\*"SGSDSVSSNSJSE9\*"SGSDSeSDSNSJSE9\*"SGSDSrSDSNSJSE9\*"SGSDShSDSNSJSE94S]SsSzSSiS|\1"SuS}9S~9_6r2\&SSj5r3\&SSj5r4\&SSj5r5\&SSj5r6\&SSj5r7\&SSj5r8\&SSj5r9\&SSj5r:\&SSj5r;\&SSj5r<\&SSj5r=\&SSj5r>\&SSj5r?\&SSj5r@\&SSj5rA\&SSj5rB\&SSj5rCS\ \*\\*4S@\\*4SjrDSrE\"SS55rF"SS\R5rHS\F4SjrI"SS\R5rJ"SS\R5rK"SS\R5rL"SS\R5rM"SS\R5rN"SS\R5rO\1"\J\K\L\M\N\OS9rPS\-S\R4SjrQS\ \-\R44SjrR"SS\R5rTSS\F4SjjrUSSjrVSrWS\ \-\ 4S\*S\+4SjrXSS\X4S\+S\YS\.S\ \-\ 4S\ \.S\ \FS\ \ 4SjjrZS\+4Sjr["SS\R5r\SSjr]SSjr^g)aBring-Your-Own-Blocks Network A flexible network w/ dataclass based config for stacking those NN blocks. This model is currently used to implement the following networks: GPU Efficient (ResNets) - gernet_l/m/s (original versions called genet, but this was already used (by SENet author)). Paper: `Neural Architecture Design for GPU-Efficient Networks` - https://arxiv.org/abs/2006.14090 Code and weights: https://github.com/idstcv/GPU-Efficient-Networks, licensed Apache 2.0 RepVGG - repvgg_* Paper: `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 Code and weights: https://github.com/DingXiaoH/RepVGG, licensed MIT In all cases the models have been modified to fit within the design of ByobNet. I've remapped the original weights and verified accuracies. For GPU Efficient nets, I used the original names for the blocks since they were for the most part the same as original residual blocks in ResNe(X)t, DarkNet, and other existing models. Note also some changes introduced in RegNet were also present in the stem and bottleneck blocks for this model. A significant number of different network archs can be implemented here, including variants of the above nets that include attention. Hacked together by / copyright Ross Wightman, 2021. N) dataclassfieldreplace)TupleListDictOptionalUnionAnyCallableSequence)partialIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)build_model_with_cfg) ClassifierHead ConvBnActBatchNormAct2dDropPath AvgPool2dSame create_conv2d get_act_layerconvert_norm_actget_attnmake_divisible to_2tuple)register_model)ByobNet ByoModelCfg ByoBlockCfgcreate_byob_stem create_blockc 0USSSSS[[SSS. UE$) N)r()r)g?bilinearz stem.convzhead.fc) url num_classes input_size pool_sizecrop_pct interpolationmeanstd first_conv classifierr)r+kwargss 8C:\Users\Terry\Desktop\FGVC-HERBS\timm\models\byobnet.py_cfgr7,s04}SYJ%.B!     gernet_szjhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-ger-weights/gernet_s-756b4751.pth)r+gernet_mzjhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-ger-weights/gernet_m-0873c53a.pthgernet_lzjhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-ger-weights/gernet_l-f31e2e8d.pth)r'r<)r=)r+r-r. repvgg_a2znhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-repvgg-weights/repvgg_a2-c1ee6d2b.pth)zstem.conv_kxk.convzstem.conv_1x1.conv)r+r3 repvgg_b0znhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-repvgg-weights/repvgg_b0-80ac3f1b.pth repvgg_b1znhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-repvgg-weights/repvgg_b1-77ca2989.pth repvgg_b1g4zphttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-repvgg-weights/repvgg_b1g4-abde5d92.pth repvgg_b2znhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-repvgg-weights/repvgg_b2-25b7494e.pth repvgg_b2g4zphttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-repvgg-weights/repvgg_b2g4-165a85f2.pth repvgg_b3znhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-repvgg-weights/repvgg_b3-199bc50d.pth repvgg_b3g4zphttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-repvgg-weights/repvgg_b3g4-73c370bf.pth resnet51qzkhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/resnet51q_ra2-d47dcc76.pthz stem.conv1)r' rG?)r+r3r-r.test_input_sizer/ resnet61qzstem.conv1.convbicubic)r3r-r.r0 geresnet50t gcresnet50t gcresnext26tsbat_resnext26ts)r3r-r.r0min_input_sizec\rSrSr%\\\R4\S'\ \S'\ \S'Sr \ \S'Sr \ \\ \ 4\S'S r\\S 'Sr\ \\S 'Sr\ \\\4\S 'Sr\ \\S 'Sr\ \\\4\S'Sr\ \\\4\S'Srg)r"ntypedcsNgsrHbr attn_layer attn_kwargsself_attn_layerself_attn_kwargs block_kwargs)__name__ __module__ __qualname____firstlineno__r strnnModule__annotations__intrWrXr r rYfloatrZr[rr r\r]r^__static_attributes__r_r8r6r"r"ns RYY  F F AsJ)-BsH}%&-BN!%J $,0K$sCx.)0%)OXc])15htCH~.5-1L(4S>*1r8r"cJ\rSrSr%\\\\\S44S4\S'Sr\ \S'Sr \ \S'Sr \ \ \S 'S r \\S 'S r\\S 'Sr\\S'Sr\\S'Sr\\S'Sr\ \S'Sr\ \S'Sr\ \ \S'\"SS9r\\S'Sr\ \ \S'\"SS9r\\S'\"S S9r\\ \4\S!'S"r g)#r!.blocksconv1x1 downsample3x3 stem_typemaxpool stem_pool stem_chsrH width_factorr num_featuresTzero_init_last_bnFfixed_input_sizerelu act_layer batchnorm norm_layerNrZc[5$Ndictr_r8r6ByoModelCfg.sdfr8)default_factoryr[r\c[5$rrr_r8r6rrs46r8r]c[5$rrr_r8r6rrsr8r^r_)!r`rarbrcrr r"rgrordrqrsr rurhrvrirwrxboolryr{r}rZrr[rr\r]r^rr rjr_r8r6r!r!s % U;+;%<<=sB CCJIs(Ix}(HcL%L#"t""d"Is!J!!%J $n=K=%)OXc])">BdB#(#HL$sCx.Hr8r!c^SnSnTS:aU4Sjn[[XU5VVVs/sHupn[SXU-US9PM snnn5nU$s snnnf)N)@r<rc*>US-S-S:XaUT-$S$)NrrVrr_)chsidxgroupss r6r_rep_vgg_bcfg..s a1}7IcVm&Pq&Pr8rep)rSrTrUrX)tuplezipr")rTwfrrU group_sizebcfgs ` r6 _rep_vgg_bcfgrs[AJ zP X[\]bdXefXeHART+5ARJGXef gD KgsA typeseveryfirstreturnc :[U5S:Xde[U[5(a+[[ U(aSOUX!55nU(dUS- /n[ U5 /n[ U5H$nXa;aUSOUSnU[ SUSS.UD6/- nM& [U5$)z&interleave 2 block types in stack rVrr)rSrTr_)len isinstancerhlistrangesetr"r)rrrTrr5rmi block_types r6interleave_blocksrs u:??%U15!;<UGEJ F 1X!"U1Xq ;>J!>v>?? =r8basicrrV)rSrTrUrWrXrYbottleig?@rti )rmrursrw0r'r)ii0r< i)rVrr)?rrg@)rTrrr)rmrqru)rHrHrH@)r)@rrg@)rr)rrr@)rrrrriquad2isilu)rmrurqrsrwr{edge)rSrTrUrWrXrYr^quadT) extra_conv)rmrurqrsrwr{r^)rSrTrUrWrYitieredger=)extent extra_params)rmrurqrsrZr[gc)rmrurqrsrZrr)rmrurqrsrwr{rZbat) block_size)rmrurqrsrwr{rZr[c [SSU0UD6$)zGEResNet-Large (GENet-Large from official impl) `Neural Architecture Design for GPU-Efficient Networks` - https://arxiv.org/abs/2006.14090 pretrained)r;_create_byobnetrr5s r6r;r;^  G* G GGr8c [SSU0UD6$)zGEResNet-Medium (GENet-Normal from official impl) `Neural Architecture Design for GPU-Efficient Networks` - https://arxiv.org/abs/2006.14090 r)r:rrs r6r:r:frr8c [SSU0UD6$)zEResNet-Small (GENet-Small from official impl) `Neural Architecture Design for GPU-Efficient Networks` - https://arxiv.org/abs/2006.14090 r)r9rrs r6r9r9nrr8c [SSU0UD6$)zURepVGG-A2 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r>rrs r6r>r>v  H: H HHr8c [SSU0UD6$)zURepVGG-B0 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r?rrs r6r?r?~rr8c [SSU0UD6$)zURepVGG-B1 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r@rrs r6r@r@rr8c [SSU0UD6$)zWRepVGG-B1g4 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)rArrs r6rArA  JZ J6 JJr8c [SSU0UD6$)zURepVGG-B2 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)rBrrs r6rBrBrr8c [SSU0UD6$)zWRepVGG-B2g4 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)rCrrs r6rCrCrr8c [SSU0UD6$)zURepVGG-B3 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)rDrrs r6rDrDrr8c [SSU0UD6$)zWRepVGG-B3g4 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)rErrs r6rErErr8c [SSU0UD6$) r)rFrrs r6rFrF  H: H HHr8c [SSU0UD6$)rr)rJrrs r6rJrJrr8c [SSU0UD6$)rr)rLrrs r6rLrL  JZ J6 JJr8c [SSU0UD6$)rr)rMrrs r6rMrMrr8c [SSU0UD6$)rr)rNrrs r6rNrNs  Lz LV LLr8c [SSU0UD6$)rr)rOrrs r6rOrOs  N Nv NNr8stage_blocks_cfgc [U[5(dU4n/n[U5H6up#U[UR5Vs/sH n[ USS9PM sn- nM8 U$s snf)Nr)rT)rr enumeraterrTr)r block_cfgsrcfg_s r6expand_blocks_cfgrsd & 1 1,.J,-%,?,Qwsa(,?? . @sA"c.U(dgX-S:XdeX-$)Nrrr_)rchannelss r6 num_groupsrs$ $)))%%r8c|\rSrSr%\r\\S'\r \\S'\ Rr \\S'Sr \\\S'Sr\\\S'Srg) LayerFni conv_norm_actnorm_actactNattn self_attnr_)r`rarbrcrrr rgrrreReLUrrr rrjr_r8r6rrsB'M8''Hh'GGC#D(8 #$(Ix!(r8rc:^\rSrSrSS\4U4SjjjrSrSrU=r$) DownsampleAvgilayersc>>[[U] 5 U=(d [5nUS:XaUOSnUS:dUS:a1US:Xa US:a[O[ R nU"SUSSS9UlO[ R"5UlURXSUS9Ul g)z/AvgPool Downsampling as in 'D' ResNet variants.rrVTF) ceil_modecount_include_pad apply_actN) superr__init__rrre AvgPool2dpoolIdentityrconv) selfin_chsout_chsstridedilationrr avg_stride avg_pool_fn __class__s r6rDownsampleAvg.__init__s mT+-$79'1}V! A:A+5?x!|-QSQ]Q]K#AzTUZ[DI DI((!y(Q r8cBURURU55$rrrrxs r6forwardDownsampleAvg.forwardsyy1&&r8r)rrFN) r`rarbrcrrrrj __classcell__rs@r6rrs" RW^ R R''r8rrc US:Xa [S0UD6$UR"URS5URS54SS0UD6$)Navgrr kernel_sizerr_)rrpop)downsample_typerr5s r6create_downsampler sI%&v&&##FJJx$8&**Y:Oi]^ibhiir8cV^\rSrSrSrS S\4U4SjjjrS S\4SjjrSr Sr U=r $) BasicBlockiz#ResNet Basic Block - kxk + kxk rc 2>[[U] 5 U =(d [5n [ X'-5n[ Xn5nX:wdUS:wd USUS:wa[ XX$USSU S9UlO[R"5UlU RXX4USS9Ul U (d U Rc[R"5OU RU5Ul U RXX5SXSS9Ul U (a U Rc[R"5OU RU5UlU S:a [U 5O[R"5UlU (a[R"5UlgU R#SS 9Ulg) NrrFrrrrrr)rr)rr drop_blockrTinplace)rrrrrrrshortcutrerr conv1_kxkr conv2_kxk attn_lastr drop_pathr)rrrr rrr bottle_ratioror linear_outrrdrop_path_ratemid_chsrrs r6rBasicBlock.__init__sJ j$(*$79 !78J0  ! x{hqk/I-7T\]^T_0DMKKMDM--f{dlmndo-p%.&++2EBKKM6;;W^K_ -- kQKqv.x.76;;;NTZT_T_`gTh5Cb5H.1bkkm$.2;;=FJJtJ4Lr8rxcU(a=[RRURRR 5 UR UR4H&n[US5(dMUR5 M( gNreset_parameters reinitzeros_rbnweightrrhasattrr%rrxrs r6 init_weightsBasicBlock.init_weights/V  GGNN4>>,,33 4YY/Dt/00%%'0r8cURU5nURU5nURU5nURU5nUR U5nUR X-5nU$r)rrrrrrrrrs r6rBasicBlock.forward6s^==# NN1  NN1  IIaL NN1  HHQ\ "r8)rrrrrrr) r'rrrNrHr TFNNrF r`rarbrc__doc__rrrr-rrjr r s@r6rrs@locgMHOMM0(d(  r8rcV^\rSrSrSrS S\4U4SjjjrS S\4SjjrSr Sr U=r $) BottleneckBlockiCz3ResNet-like Bottleneck Block - 1x1 - kxk - 1x1 rc  >[[U] 5 U =(d [5n [ X&-5n[ X5nX:wdUS:wd USUS:wa[ XX$USSU S9UlO[R"5UlU RXS5Ul U RXX4USUU S9Ul U RXX4USUU S9Ul U (aU RXX5SUU S9Ul O[R"5Ul U (d U Rc[R"5OU RU5UlU RXSSS9UlU (a U Rc[R"5OU RU5UlUS:a [#U5O[R"5UlU (a[R"5UlgU R'S S 9Ulg) NrrFrrrrr)rrrrrTr)rr8rrrrrrrerr conv1_1x1r conv2b_kxkr conv3_1x1rrrr)rrrr rrrrrorrrrrr r!rrs r6rBottleneckBlock.__init__Gs ot-/$79 !78J0  ! x{hqk/I-7T\]^T_0DMKKMDM--fqA-- k8A;j.2 -- k8A;j.2 $22+ F_i3kDO!kkmDO%.&++2EBKKM6;;W^K_ --gU-S.76;;;NTZT_T_`gTh5Cb5H.1bkkm$.2;;=FJJtJ4Lr8rxcU(a=[RRURRR 5 UR UR4H&n[US5(dMUR5 M( gr$) rer'r(r=r)r*rrr+r%r,s r6r-BottleneckBlock.init_weightshr/r8c<URU5nURU5nURU5nURU5nUR U5nUR U5nUR U5nURU5nURX-5nU$r) rr;rr<rr=rrrr1s r6rBottleneckBlock.forwardos==# NN1  NN1  OOA  IIaL NN1  NN1  NN1  HHQ\ "r8) rrrr;rr<r=rr) r'rr3rHNr FFFNNrr4r5r s@r6r8r8CsAosjn13M`gMMB(d(  r8r8cV^\rSrSrSrS S\4U4SjjjrS S\4SjjrSr Sr U=r $) DarkBlocki~aDarkNet-like (1x1 + 3x3 w/ stride) block The GE-Net impl included a 1x1 + 3x3 block in their search space. It was not used in the feature models. This block is pretty much a DarkNet block (also DenseNet) hence the name. Neither DarkNet or DenseNet uses strides within the block (external 3x3 or maxpool downsampling is done in front of the block repeats). If one does want to use a lot of these blocks w/ stride, I'd recommend using the EdgeBlock (3x3 /w stride + 1x1) for more optimal compute. rc 0>[[U] 5 U =(d [5n [ X&-5n[ X~5nX:wdUS:wd USUS:wa[ XX$USSU S9UlO[R"5UlU RXS5Ul U (d U Rc[R"5OU RU5Ul U RXX4USXSS9Ul U (a U Rc[R"5OU RU5UlU S:a [U 5O[R"5UlU (a[R"5UlgU R#SS9Ulg) NrrFrrrrrrrTr)rrDrrrrrrrerrr;rrrrrrrrrr rrrrrorrrrr r!rrs r6rDarkBlock.__init__sB i')$79 !78J0  ! x{hqk/I-7T\]^T_0DMKKMDM--fqA%.&++2EBKKM6;;W^K_ -- k8A;U.D/86;;;NTZT_T_`gTh5Cb5H.1bkkm$.2;;=FJJtJ4Lr8rxcU(a=[RRURRR 5 UR UR4H&n[US5(dMUR5 M( gr$r&r,s r6r-DarkBlock.init_weightsr/r8cURU5nURU5nURU5nURU5nUR U5nUR U5nUR X-5nU$r)rr;rrrrrr1s r6rDarkBlock.forwardj==# NN1  IIaL NN1  NN1  NN1  HHQ\ "r8)rrrr;rrr) r'rr3rHNr TFNNrr4r5r s@r6rDrD~s@pthl "MMTMM0(d(  r8rDcV^\rSrSrSrS S\4U4SjjjrS S\4SjjrSr Sr U=r $) EdgeBlockiaEdgeResidual-like (3x3 + 1x1) block A two layer block like DarkBlock, but with the order of the 3x3 and 1x1 convs reversed. Very similar to the EfficientNet Edge-Residual block but this block it ends with activations, is intended to be used with either expansion or bottleneck contraction, and can use DW/group/non-grouped convs. FIXME is there a more common 3x3 + 1x1 conv block to name this after? rc ,>[[U] 5 U =(d [5n [ X&-5n[ X~5nX:wdUS:wd USUS:wa[ XX$USSU S9UlO[R"5UlU RXX4USXS9Ul U (d U Rc[R"5OU RU5Ul U RXSSS9Ul U (a U Rc[R"5OU RU5UlU S:a [U 5O[R"5UlU (a[R"5UlgU R#SS 9Ulg) NrrFrr:rrTr)rrOrrrrrrrerrrr conv2_1x1rrrrrGs r6rEdgeBlock.__init__sD i')$79 !78J0  ! x{hqk/I-7T\]^T_0DMKKMDM-- [(1+.3&/&++2EBKKM6;;W^K_ --gU-S.76;;;NTZT_T_`gTh5Cb5H.1bkkm$.2;;=FJJtJ4Lr8rxcU(a=[RRURRR 5 UR UR4H&n[US5(dMUR5 M( gr$) rer'r(rQr)r*rrr+r%r,s r6r-EdgeBlock.init_weightsr/r8cURU5nURU5nURU5nURU5nUR U5nUR U5nUR X-5nU$r)rrrrQrrrr1s r6rEdgeBlock.forwardrMr8)rrrrrQrr) r'rr3rHNr FFNNrr4r5r s@r6rOrOs@ptX\13MNUMM0(d(  r8rOcT^\rSrSrSrS S\4U4SjjjrS S\4SjjrSr Sr U=r $) RepVggBlockizRepVGG Block. Adapted from impl at https://github.com/DingXiaoH/RepVGG This version does not currently support the deploy optimization. It is currently fixed in 'train' mode. rc F>[[U] 5 U =(d [5n [ Xq5n X:H=(a US:H=(a USUS:Hn U (aU R USS9OSUlU RXX4USXSS9UlU RXSXLSS9Ul U Rc[R"5OU RU5Ul U S:aU (a [U 5O[R"5UlU RSS 9Ulg) NrrFrrF)rrrrTr)rrXrrrridentityrconv_kxkconv_1x1rrerrrr)rrrr rrrrrorrr r use_identrs r6rRepVggBlock.__init__s k4)+$79J/%T&A+T(1+RS:T EN5ATX ,, [(1+E-C ,,Vain,o %+[[%8BKKMfkk'>R 5Cb5HY.1\^\g\g\i::d:+r8rxcUR5Hzn[U[R5(dM$[RR UR SS5 [RR URSS5 M| [URS5(aURR5 gg)Ng?rr%) modulesrre BatchNorm2dr'normal_r*biasr+rr%)rrxms r6r-RepVggBlock.init_weightss|A!R^^,,"b12.  4990 1 1 II & & ( 2r8c>URc$URU5URU5-nOIURU5nURU5URU5-nURU5nX-nUR U5nUR U5nU$r)rZr\r[rrr)rrrZs r6rRepVggBlock.forward s ==  a 4==#33A}}Q'H a 4==#33Aq!A A IIaL HHQKr8)rrr\r[rrZ) r'rr3rHNNNrr4r5r s@r6rXrXs:ptXZ,(/,, )d)  r8rXcV^\rSrSrSrS S\4U4SjjjrS S\4SjjrSr Sr U=r $) SelfAttnBlockizHResNet-like Bottleneck Block - 1x1 - optional kxk - self attn - 1x1 rc H>[[U] 5 U ce[X&-5n[ UU5nX:wdUS:wd USUS:wa[ XX$USSU S9UlO[R"5UlU RUUS5Ul U (aU RUUX4USUUS9Ul SnO[R"5Ul U c0O[U S9nU R"U4SU0UD6Ul U (aU RU5O[R"5UlU RUUSSS9UlUS :a [#U5O[R"5UlU (a[R"5UlgU R'S S 9Ulg) NrrFrr:) feat_sizerrrTr)rrjrrrrrrerrr;rrrr post_attnr=rrr)rrrr rrrrrorr post_attn_narlrrr r!r opt_kwargsrs r6rSelfAttnBlock.__init__st mT+-!!! !78J0  ! x{hqk/I-7T\]^T_0DMKKMDM--fgqA #11+xPQ{*26DNF[[]DN$,R$2K ))'O&OJO5A1r{{}--gwU-S5Cb5H.1bkkm$.2;;=FJJtJ4Lr8rxcU(a=[RRURRR 5 [ URS5(aURR5 ggr$) rer'r(r=r)r*r+rr%)rrxs r6r-SelfAttnBlock.init_weights:sL  GGNN4>>,,33 4 4>>#5 6 6 NN + + - 7r8cURU5nURU5nURU5nURU5nUR U5nUR U5nUR U5nURX-5nU$r)rr;rrrmr=rrr1s r6rSelfAttnBlock.forward@sy==# NN1  NN1  NN1  NN1  NN1  NN1  HHQ\ "r8)rr;rr=rrmrr) r'rr3rHNr FFTNNNrr4r5r s@r6rjrjs@osdhIKM MM>.d.  r8rj)rrdarkrrrrblock_fncU[U'gr)_block_registry)rrvs r6register_blockryXs "*OJr8blockc [U[R[45(aU"S0UD6$U[;d SU35e[U"S0UD6$)NzUnknown block type (r_)rrerfrrx)rzr5s r6r$r$\sR%"))W-..v O #C';E7%CC # 5 ! +F ++r8c8^\rSrSrSS\4U4SjjjrSrU=r$)Stemicrc >[TU]5 US;deU =(d [5n [U[[ 45(a[ U5nUn O0[U5V s/sHn [X(U --5PM sn SSS2n X@l /Ul Sn S/S/US- --n US:Xa U(dSU S'UcUOUnS/Xg- -S/U--nUnSn[[XU55H}un unnnU(a U RO[nS U S-3nU S :a*US:a$URR[!UUU S 95 UR#UU"UUUUS 95 UnUU-nUn M U(agS UR%5;aSURR[!UUU S 95 UR#S[&R("SSS55 US-nSn URR[!UUU S 95 UU:Xdegs sn f)N)rVrrhrVrrFTrrnum_chs reductionmodule)r rmaxrr')rrrrrrrrroundr feature_inforrrrappendr add_modulelowerre MaxPool2d)rrrr rrnum_repnum_act chs_decayrrur prev_feat stem_stridesstem_norm_actsprev_chs curr_stridechrWnalayer_fn conv_namers r6r Stem.__init__es $79 ge} - -'lGHAFwPAgQ67PQUSUQUVH  saSGaK00 Q;t L $_''G$56$'9II 'HN(STNA{Ar/1v++}Hq1ugI1uQ!!((h+^g)hi OOIx"+^_'` aH 1 K!IU ETZZ\)    $ $T(kZc%d e OOFBLLAq$9 : 1 KI   h+V_!`af$$$AQs!G=)rr)r'rrrr'Ng?N)r`rarbrcrrrjr r s@r6r}r}csFOKO*%AH*%*%r8r}c `U=(d [5nUS;deSU;aSU;aSOSn[XSXcUS9nOSU;a[USU-S -US-U4X5S 9nOvS U;a [XSS X5S 9nOcSU;a [XSUS9nOQSU;a&U(a [XSSX5S9nO7URXSSS9nO%U(a [XSSX5S9nOURXSSS9n[ U[5(a:UR Vs/sH n[ USRXHS/5S9PM" n nXy4$[ USUS9/n Xy4$s snf)N)rhrrrdeepr7x7rprrrVr)rrrrrr'r=)rrrrH)rrrrr)rrrr)r)rrrr.r)rr)rr}rXrrrrjoin) rrrq pool_type feat_prefixrrstemfrs r6r#r#sh  wyF T TT T )+!FQX^_ Y FQ[A-w!|WEIe 9 FQ#I] ) 61VD )  AIUD''1'ED AIUD''1'ED$VZVgVghVgQRQsxxk0J'KLVg h  W+NO  is0'D+cRUcS$[UVs/sHo"U-PM sn5$s snfr)r)rlrrWs r6reduce_feat_sizers+$4Q%i0Pifi0P*QQ0Ps$c&UbUOUnU=(d 0$)a!Override model level attn/self-attn/block kwargs w/ block level NOTE: kwargs are NOT merged across levels, block_kwargs will fully replace model_kwargs for the block if set to anything that isn't None. i.e. an empty block_kwargs dict will remove kwargs set at model level for that block r_)r^ model_kwargs out_kwargss r6override_kwargsrs".!9|J  r8r^ block_cfg model_cfgcUSnURc URbvUR(dSnOY[URUR5nUR=(d URnUb[[ U5/UQ76OSn[ X4S9nUR c URbvUR(dSnOY[UR UR 5nUR=(d URnUb[[ U5/UQ76OSn[ X6S9nX0S'UR[URUR55 g)Nr)rr) r[rZrrrrr]r\updater^)r^rr layer_fnsrZr[r\r]s r6update_block_kwargsrs5X&I(I,@,@,L##J))*?*?AVAVWK"--E1E1EJHRH^*!5D DdhJI7 !!-1J1J1V(("O.y/I/I9KeKef '77T9;T;TO".&h&?SBRS48 IA & (>(> @V@VWXr8rr  output_stride stem_featrlblock_kwargs_fncU=(d [5n/nURVs/sHn[U5PM n nU V V s/sH(n [U V s/sHoRPM sn 5PM* n n n [ R "SU[U 55RU 5V s/sHoR5PM nn SnUSnUSnUn/n[U 5GHjunnUSRnUS:waU(aURU5 UU:a US:aUU-nSnUU-nUS;aSOSn/n[U5Hunn[URUR-5nURn[!U["5(a U"UU5n[%UUUS:XaUOSUU4UUR&UR(UUUUS9 nUR*S;aUUS 'U"UUUS 9 U[-UR*40UD6/- nUnUnUS:dMUS:XdM[/UU5nM U[0R2"U6/- n[%UUS U3S 9nGMm URU5 [0R2"U6U4$s snfs sn fs sn n fs sn f) Nrrrr)rrVrV) rrrrrrror rrrl)rrzstages.r)rrmrsumrTtorchlinspacesplittolistrrWrrrUrvrXrr rrYrorSr$rre Sequential)rr rrrlrrrrWr stage_bcsbcdepthsrdprr net_striderrstages stage_idxstage_block_cfgsrfirst_dilationrm block_idxrrrr^s r6create_byob_stagesrs  wyFL03 ; 1#A& J;?I Jz)c),)B44),-zF J$~~aVMSSTZ[ \[!88:[C \H;'J#HI F'0'<# #!!$&& Q;9    *  &6A:  HFf &&0a$-.>$? Iy$Y[[33C3C%CDG"J*h//'; !*avQ((3%&\\>>"9~i8 L~~/,5 [) LI M |INNClCD DF%NHzi1n,Y? 1%@4 2==&)**Z'R[Q\H]^ M(=P " ==& !< //c<, J \s#I/I9I4$ I9$I?4I9c[UR5n[URUS9n[ [ URUS9nUR (a)[ [UR 540URD6OSnUR(a)[ [UR540URD6OSn[X2XUS9nU$)N)r}r{)rrrrr) rr{rr}rrrZrr[r\r]r)rrrrrrrs r6 get_layer_fnsr#s  &C3>>SIHI#..CPMCF>>78CNN+ ?s ?W[DRUReRe!4!45N9M9MNkoI]3enoH Or8cX^\rSrSrSrS S\4U4SjjjrSrS SjrSr Sr S r U=r $) r i-a'Bring-your-own-blocks' Net A flexible network backbone that allows building model stem + blocks via dataclass cfg definition w/ factory functions for module instantiation. Current assumption is that both stem and blocks are in conv-bn-act order (w/ block ending in act). rc >[TU]5 X lXl[ U5n UR (a UcS5eUb [ U5OSn /Ul[[UR=(d URSRUR-55n [X),    #E#*:*:S=M=M*M$N OD $228=N=NPQRDO (  kkmDO  **jn[6QZf gi i#4#4#4kdhdrdrs &&(DAq !Q )Aq.))1BC r8c.URR$r)rfc)rs r6get_classifierByobNet.get_classifier\syy||r8cL[URXURS9Ulg)Nr)rrwrr)rr,rs r6reset_classifierByobNet.reset_classifier_s"4#4#4kdhdrdrs r8clURU5nURU5nURU5nU$r)rrrrs r6forward_featuresByobNet.forward_featuresbs/ IIaL KKN OOA r8cJURU5nURU5nU$r)rrrs r6rByobNet.forwardhs$  ! !! $ IIaLr8)rrrrr,rwrr)r&r'r rtTNrr)r ) r`rarbrcr6r!rrrrrrjr r s@r6r r -sAikUW%DK%D%DNt r8r cz[U[R5(aURSURS-UR-nX R -nUR RRS[R"SU- 55 URb%URRR5 gg[U[R5(aa[RRUR SSS9 URb*[RRUR5 gg[U[R 5(aS[RR#UR 5 [RRUR5 gg)Nrrrrg{Gz?)r1r2)rreConv2dr  out_channelsrr*datarbmathsqrtrczero_Linearr'r(raones_)rdrfan_outs r6rrns!RYY--"Q]]1%55FHH  a3=!9: 66  FFKK     Aryy ! ! s5 66  GGNN166 "  Ar~~ & &  ahh qvv 'r8c V[[X4[U[U[ SS9S.UD6$)NT)flatten_sequential) default_cfgr feature_cfg)rr default_cfgs model_cfgsr)variantrr5s r6rr~s;   )W%D1     r8)rh))rrr)rHrHrHrHrr4r_)rhrhrN)rV)_r6r dataclassesrrrtypingrrrr r r r r functoolsrrtorch.nnre timm.datarrhelpersrrrrrrrrrrrrrregistryr__all__r7rr"r!rrdrhrrrrr;r:r9r>r?r@rArBrCrDrErFrJrLrMrNrOrrrrfrrrr8rDrOrXrjrxryr$rr}r#rrrrirrr rrr_r8r6rs@ 4 11NNN A)XXXX$ W5 xz5 xz 5  x F4 5 |?A5 |?A5" |?A#5(4 ~?A)5. |?A/544 ~?A55: |?A;5@4 ~?AA5J yMV%5K5R$&`ikS5V4$&`ikW5Z4$&`ik[5`T$&`ika5dt$&`i$&e5 p  2 2  2  II I,PUS#X',S$s)^'<HL ;$j  WSA! C WSA! C XcQ1 G XcQ1 D XcQ1 D   j WSA! C WSA! C XcQ1 G XcQ1 D XcQ1 D   j2 WR1r B WR1r B XcQ1 G XcQ1 D XcQ1 D   3jL}1FGMjV 12Wj` 01ajj 0;kjt 34uj~ 3A>jH 01IjR 0;Sj` XcQ2$ G XcQ2$ G XdaB4 H XdaA# F   aj| VqC1sQUQW X XcQ2$ G XdaB4 H XdaA# F  T* }j\ VqC1 > VqC1 > XdaD A XdaD A  5]j~ XcQ4 @ XcQ4 @ XdaD A XdaD A   jX XcQ2$ G XcQ2$ G XdaB4 H XdaB4 H   Yjv XcQ2$ G XcQ2$ G XdaB4 H XdaB4 H  A&wj ZHHHHHHIIIIIIKKIIKKIIKKII II KK KK MM OO k8K;P.P(QVZ[fVg& )) )'BII'"jwj--`8bii8v3 3l2 2j+"))+\4BII4n     +c+RYY+,c299n-,,%2==,%^ ^e FR Yd38nYYYdYH$($(.A 90 90*/90@C90PTUXZ]U]P^90C=90!90"(+ 90x{>bii>B  r8