čpe}@dZddlmZmZddlmZmZmZmZddl m Z ddl m Z gZ dadZid edd d d dedd d d dedd d d dedd d d dedd d d dedd d d dedd d d dedd d d deddd d dedddeddd d dedd d d dedd d d dedd d d dedd d d!edd d Zedbid"eed#d$d%ddd&'ed#d(d)d*dd&'ed+dd*d,d*dd&-ed.d$d/d*dd&'fd0d1d2dd d3e4d5eed#d$d%d*dd&'ed#d(d)d*dd&'ed+dd6d,d*dd&-ed.d$d/ddd&'fd0d1ddd d7d3e8 d9eed#d$d%dd:d&'ed#d(d)d*d:d&'ed+dd*d,d*d:d&-ed.d$d/d*d:d&'fd0d1d2dd d7d;d3e< deed.d$d0ddd='ed.d$d>d*dd='ed.d?d%d*dd='ed.d$d)d*dd='fd0d@d2ddAedBd$CDdeed#d$d0ddd='ed#d$d>d*dd='ed.d?d%d*dd='ed.d$d)d*dd='fd0d1d2ddAedBd$CDdeed#d*d%ddd&'ed#d*d)d*dd&'ed+dd*d,d*dd&-ed.d*d/d*dd&'fd0d1d2ddAedBd*CDdeed#d$d%ddd&'ed#d(d)d*dd&'ed+dd6d,d*dd&-ed.d$d/d*dd&'fd0d1d2dd7dAedBd*CEdeed#d*d%dd:d&'ed#d*d)d*d:d&'ed+dd*d,d*d:d&-ed.d*d/d*d:d&'fd0d1d2dd7d;dAedBd*CF deed#d*d%ddd&'ed#d*d)d*dd&'ed+dd*d,d*dd&-ed.d*d/d*dd&'fd0d1d2ddGeDdeed#d$d%ddd&'ed#d(d)d*dd&'ed+d$d6d,d*dd&-ed.d$d/d*dd&'fd0d1d2ddGeDdeed#d*d%dd:d&'ed#d*d)d*d:d&'ed+dd*d,d*d:d&-ed.d*d/d*d:d&'fd0d1d2dd7d;dGeF dHeed#d*d%ddd&'ed+dd*d)d*dd&-ed+dd*d,d*dd&-ed.d*d/d*dd&'fd0d1d2dd dIedBJ4dKeed#d$d%ddd&'ed+dd(d)d*dd&-ed+dd*d,d*dd&-ed.d$d/d*dd&'fd0d1d2dd d7dIedBJ8 dLeed#d*d%dd:d&'ed+dd*d)d*d:d&-ed+dd*d,d*d:d&-ed.d*d/d*d:d&'fd0d1d2dd d7d;dIedBJ< deed.d*d%ddd&'ed.d*d)d*dd&'ed.d*d,d*dd&'ed.d*d/d*dd&'fd0d1d2ddMeDd!eed.d$d%ddd&'ed.d(d)d*dd&'ed.d*d,d*dd&'ed.d$d/d*dd&'fd0d1d2dd7dMeEZdcdPZe dddQZe dddRZe dddSZe dddTZe dddUZe dddVZe dddWZe dddXZe dddYZe dddZZe ddd[Ze ddd\Ze ddd]Ze ddd^Z e ddd_Z!e ddd`Z"dNS)ea Bring-Your-Own-Attention Network A flexible network w/ dataclass based config for stacking NN blocks including self-attention (or similar) layers. Currently used to implement experimential variants of: * Bottleneck Transformers * Lambda ResNets * HaloNets Consider all of the models definitions here as experimental WIP and likely to change. Hacked together by / copyright Ross Wightman, 2021. IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) ByoBlockCfg ByoModelCfgByobNetinterleave_blocks)build_model_with_cfg)register_modelc 8|dddddttddddd |S) Ni)r)rg?bicubiczstem.conv1.convzhead.fcF) url num_classes input_size pool_sizecrop_pct interpolationmeanstd first_conv classifierfixed_input_sizemin_input_sizer)rkwargss F/home/terry/CeaseFire/code_minimal_pytorch_ITTI/timm/models/byoanet.py_cfgr!s:4}SYI%.B'y!]       botnet26t_256T)rr$)r%)rrrrbotnet50ts_256eca_botnext26ts_256 halonet_h1)rrrrhalonet_h1_c4c5 halonet26t halonet50tseca_halonext26tslambda_resnet26t)rr.)rrrrlambda_resnet50t)rreca_lambda_resnext26tsswinnet26t_256swinnet50ts_256eca_swinnext26ts_256 rednet26t)rrr rednet50ts botnet26tbottlerr$g?)typedcsgsbri)r7 self_attni)typeseveryr9r:r;r<r=r@i@tieredmaxpool bottleneck)blocksstem_chs stem_type stem_pool num_featuresrself_attn_layerself_attn_kwargs botnet50tssilu) rGrHrIrJrKr act_layerrLrMeca_botnext26tseca) rGrHrIrJrKrrQ attn_layerrLrMg?r. 7x7halor%) block_size halo_size)rGrHrIrJrKrLrM)rGrHrIrJrKrQrLrM) rGrHrIrJrKrQrUrLrMlambda swinnet26tswin)win_size swinnet50tseca_swinnext26ts involutionNFc tt||ft||s t|n t|t dd|S)NT)flatten_sequential) default_cfg model_cfg feature_cfg)r r default_cfgs model_cfgsdict)variant cfg_variant pretrainedrs r _create_byoanetrm7s\ *  )-8U*W%%j>UD111       r"c J|ddtdd|i|S)T Bottleneck Transformer w/ ResNet26-T backbone. Bottleneck attn in final stage. img_sizer$r#r6rl)r#r6 setdefaultrmrlrs r r#r#@s4 j#&&&  Y YJ YRX Y YYr"c J|ddtdd|i|S)zT Bottleneck Transformer w/ ResNet50-T backbone. Bottleneck attn in final stage. rpr$r&rNrl)r&rNrqrss r r&r&H4 j#&&&  [ [j [TZ [ [[r"c J|ddtdd|i|S)rorpr$r'rRrl)r'rRrqrss r r'r'Ps5 j#&&&  e ePZ e^d e eer"c tdd|i|S)z HaloNet-H1. Halo attention in all stages as per the paper. This runs very slowly, param count lower than paper --> something is wrong. r(rl)r(rmrss r r(r(Xs  I IJ I& I IIr"c tdd|i|S)z8 HaloNet-H1 config w/ attention in last two stages. r)rl)r)rxrss r r)r)as  N N Nv N NNr"c tdd|i|S)F HaloNet w/ a ResNet26-t backbone, Hallo attention in final stage r*rl)r*rxrss r r*r*h  I IJ I& I IIr"c tdd|i|S)zF HaloNet w/ a ResNet50-t backbone, Hallo attention in final stage r+rl)r+rxrss r r+r+os  J JZ J6 J JJr"c tdd|i|S)r{r,rl)r,rxrss r r,r,v  O O* O O OOr"c tdd|i|S)B Lambda-ResNet-26T. Lambda layers in one C4 stage and all C5. r-rl)r-rxrss r r-r-}rr"c tdd|i|S)zB Lambda-ResNet-50T. Lambda layers in one C4 stage and all C5. r/rl)r/rxrss r r/r/rr"c tdd|i|S)rr0rl)r0rxrss r r0r0s  U U Uf U UUr"c J|ddtdd|i|S) rpr$r1r\rl)r1r\rqrss r r1r1rur"c J|ddtdd|i|S)rrpr$r2r_rl)r2r_rqrss r r2r2s4 j#&&&  ] ] ]V\ ] ]]r"c J|ddtdd|i|S)rrpr$r3r`rl)r3r`rqrss r r3r3s5 j#&&&  g gR\ g`f g ggr"c tdd|i|S)rr4rl)r4rxrss r r4r4s  H H: H H HHr"c tdd|i|S)rr5rl)r5rxrss r r5r5r|r")r )NF)F)#__doc__ timm.datarrbyobnetrrr r helpersr registryr __all__r!rgrirhrmr#r&r'r(r)r*r+r,r-r/r0r1r2r3r4r5rr"r rs}  BAAAAAAAIIIIIIIIIIII))))))$$$$$$ TTb4M]cdddddrD]^deee44B-cijjj  $$2-6Zghhh tt}_lmmm$$2-6Zghhh44B=F[hiii `mnnnMmgmnnnMBBBddr-Tamsttt ddrD]^deee!"ttTm_efff#$DDR$=djkkk%("&III)*$$2-6JJJ+ 2Tyyyk KXcQ1 F F F KXcQ1 F F F  $;1TUV[\ae f f f K[AqT J J J  $y"{ KXcQ1 F F F KXcQ1 F F F  $;1TUV[\ae f f f K[AqT J J J  $#yB K KXcQ2$ G G G KXcQ2$ G G G  $;1TUV[]bf g g g K[Ard K K K  $Cyf{ K[AqQ3 G G G K[AaC H H H K[B#qS I I I K[AaC H H H  a888    gyB K KXbA! D D D KXcQ1 E E E K[B#qS I I I K[AaC H H H  a888    Cy^{ KXcQ1 F F F KXcQ1 F F F  $;1TUV[\ae f f f K[AqT J J J  a888    _yz KXcQ1 F F F KXcQ1 F F F  $;1TUV[\ae f f f K[AqT J J J  a888{yX![ KXcQ2$ G G G KXcQ2$ G G G  $;1TUV[]bf g g g K[Ard K K K  a888Yyz![ KXcQ1 F F F KXcQ1 F F F  $;1TUV[\ae f f f K[AqT J J J       {yV![ KXcQ1 F F F KXcQ1 F F F  $;1TUV[\ae f f f K[AqT J J J       Wyr'; KXcQ2$ G G G KXcQ2$ G G G  $;1TUV[]bf g g g K[Ard K K K   syT{ KXcQ1 F F F  $;1STUZ[`d e e e  $;1TUV[\ae f f f K[AqT J J J  q)))Uyr KXcQ1 F F F  $;1STUZ[`d e e e  $;1TUV[\ae f f f K[AqT J J J  q)))syR![ KXcQ2$ G G G  $;1STUZ\ae f f f  $;1TUV[]bf g g g K[Ard K K K  q)))Syxk K[AaD I I I K[AaD I I I K[AqT J J J K[AqT J J J  $    yyT{ K[AaD I I I K[AaD I I I K[AqT J J J K[AqT J J J  $Uy xZZZZ\\\\ffffJJJJOOOO JJJJ KKKK PPPP PPPP PPPP VVVV \\\\^^^^hhhhIIII JJJJJJr"