8gVddlZddlmZddlmcmZddlmZddlm Z ddl Z ddlm Z ddl m Z dZdZdZed ZGd d ej$ZGd d ej$ZGddej$ZGddej$ZGddej$Zy)N)partial)Callable)Tensor)repeatcfd}|S)Nct|tjjrt|ts t |St t |SN) isinstance collectionsabcIterablestrtupler)xns -/nas3/zsz/FLARE_huggingface/mast3r/modules.pyparsez_ntuple..parses: a11 2:a;M8OVAq\"")rrs` r_ntuplers# Lrc |duSr r)vals rexistsrs d?rc t|r|S|Sr )r)rds rdefaultr s+3$1$rc*eZdZdZdfd ZdZxZS) ResidualBlockzW ResidualBlock: construct a block of two conv layers with residual connections cXtt| tj|||d|d|_tj|||dd|_tjd|_|dz}|dk(r\tj|| |_ tj|| |_ |dk(s,tj|| |_ n|d k(rTtj||_ tj||_ |dk(stj||_ n|d k(rTtj||_ tj||_ |dk(swtj||_ n\|d k(rQtj|_ tj|_ |dk(s tj|_ nt |dk(rd|_ytjtj||d| |j|_y)Nzeros) kernel_sizepaddingstride padding_mode)r#r$r&T)inplacegroup) num_groups num_channelsbatchinstancenone)r#r%)superr__init__nnConv2dconv1conv2ReLUrelu GroupNormnorm1norm2norm3 BatchNorm2dInstanceNorm2d SequentialNotImplementedError downsample)self in_planesplanesnorm_fnr%r#r* __class__s rr0zResidualBlock.__init__,s mT+-YY v;&_f  YYvv;PQ`gh GGD) q[ g &QDJ&QDJQ;\\ZfU  /DJ/DJQ;^^F3  "**62DJ**62DJQ;..v6  DJDJQ;]]_ % % Q;"DO mmBIIiUV_e,fhlhrhrsDOrc$|}|j|j|j|}|j|j|j |}|j |j |}|j||zSr )r6r8r3r9r4r?)r@rys rforwardzResidualBlock.forwardVsp  IIdjjA/ 0 IIdjjA/ 0 ?? &"AyyQr)r)r!)__name__ __module__ __qualname____doc__r0rG __classcell__rDs@rrr's(tT rrcLeZdZdZddej ddddffd ZdZxZS)MlpAMLP as used in Vision Transformer, MLP-Mixer and related networksNTFc t ||xs|}|xs|}t|}t|} |rttj dntj } | |||d|_||_t j| d|_ | |||d|_ t j| d|_ yNr!)r#r)bias) r/r0 to_2tuplerr1r2Linearfc1actDropoutdrop1fc2drop2 r@ in_featureshidden_features out_features act_layer norm_layerrUdropuse_conv drop_probs linear_layerrDs rr0z Mlp.__init__ds #2{ )8[t_ elementwise_affineepsT embed_dim num_heads batch_firstrr_r`rdNr) r/r0r1 LayerNormr8r9attnintrPmlp)r@ hidden_sizer|rt mlp_ratio block_kwargsmlp_hidden_dimrDs rr0zAttnBlock.__init__sr \\+%TR \\+%TR l W[l_kl [945;UVWrc|j}|j|}|j|}|j|||\}}||z}||j |j |z}|Sr )dtyper8torrr9)r@rmaskr attn_output_s rrGzAttnBlock.forwardsj JJqM DDK1a+ Q  O A' 'rr ) rIrJrKr1MultiheadAttentionrModuler0rGrMrNs@rrrrrs4 02/D/D XS"))^, X(rrrc(eZdZdfd ZddZxZS)CrossAttnBlockc Ht|tj|dd|_tj||_tj|dd|_tjd ||dd||_t||z}t||d|_ y) z' Cross attention block FrvrwTrzrr~Nr) r/r0r1rr8 norm_contextr9r cross_attnrrPr)r@r context_dimr|rrrrDs rr0zCrossAttnBlock.__init__s \\+%TR LL5\\+%TR // !YD LX [945;UVWrc|j|}|j|}|j||||\}}||j|j |zS)N) attn_mask)r8rrrr9)r@rcontextrrrs rrGzCrossAttnBlock.forwardsY JJqM##G,GWM Q TXXdjjm444r)r!rsr )rIrJrKr0rGrMrNs@rrrsX" 5rr)torchtorch.nnr1torch.nn.functional functionalF functoolsrtypingrr r itertoolsrrrrrVrrrPrlrrrrrrrs % AJ 7 BII7 t!"))!H!bii!F$ $N5RYY5r