!Mg;"ddlZddlmZmZmZmZmZddlZddl Z ddl m Z ddl m Z ddl mZdgZGdde jZGdd e jZGd d e jZGd d e jZGdde jZdS)N)ListOptionalSequenceTupleUnion) distributed)nn) functionalVectorQuantizer2ceZdZ ddeffd ZdZd efd Z dd ej d e ej e eej ffd Z dde ej d e e ej ej ffdZ dd ej dedeee ee eeffdeed e e ej ejff dZde ej d ej fdZdededej dej d e eej ej ff dZxZS) r ?rN?betac ,t||_|_||_||_|_|dkrCtfdt|pt|jD|_ n|dkrIttdkrtntj|_ n@t!tjfdt|D|_ |dt'jt|j|jfdd|_||_tj|j|j|_dS) Nrc|g|]8}tdkrtntj9Sư>absPhir Identity.0_Cvae quant_resis //home/notantonvoron/switti_demo/models/quant.py z-VectorQuantizer2.__init__..$sN/2*oo.DWSz***"+--rc|g|]8}tdkrtntj9Srrrs rrz-VectorQuantizer2.__init__../sS444:-'Cj)))444r ema_vocab_hit_SV) fill_value)super__init__ vocab_sizer using_znorm v_patch_numsquant_resi_ratio PhiNonSharedrangelenr PhiSharedrrr rPhiPartiallyShared ModuleListregister_buffertorchfull record_hitr Embedding embedding) selfr(rr)rdefault_qresi_countsr*rshare_quant_resi __class__s ` ` rr'zVectorQuantizer2.__init__s ) !,(4 * q  *"#7#Q3t?P;Q;QRRDOO  " '),Z4)?RD*%%%R[]]DOO1 44444!!122 44455DO   JD-..@S Q Q Q    dotyAAr c&|dkr2tj|jjj|dS|dkrR|jjjt| |jz t||jz dSdS)Nr)std) r init trunc_normal_r7weightdatauniform_rr()r8einis rrCzVectorQuantizer2.eini?s !8  G ! !$."7"<$ ! G G G G G AX  N ! & / /T T_,c$ii$/.I       r returnc n|jd|jd|jdt|jd|j S)Nz, znorm=z, beta=z | S=z , quant_resi=)r*r)rr.r+r8s r extra_reprzVectorQuantizer2.extra_reprGsp#UUT-=UUdiUUX[\`\mXnXnUU~B~SUU Ur Ff_BChwc  |j}|tjkr|}|j\}}}}|}|} tj| } tjj d5d} tj j tj|j } tj} t!jD]k\}}jr|| dz krCt%j| ||fddd d dd |n+| dd d dd |}t%j|d }tj|t%jjjjjd zd }n|| dz krCt%j| ||fddd d dd |n+| dd d dd |}tj|dd tjjjjddz}||jjjjddtj|d }| j }j!rtEj#|d }|$|||}|| dz krTt%j|dd dd ||fd%n<|dd dd %}j&|| dz z |}| |z} | |z} j!r|'j(dkr!j)|*|nj(dkrGj)|+d,|-dnFj)|+d,|-dxj(dz c_(| ,|| t%j.| j|+j/t%j.| |zz } m| d| z z} | j|z ,|} dddn #1swxYwYtEj0|1|jdz zj z dz|r"fdt!jD}nd}| || fS)NF)enabledr$)dtypedevicer!areasizemoderdimTrUkeepdimalphar) minlength)async_opbicubicdg?g?gGz?g{Gz?g?g{Gz?cg|]O\}}j|kdzPS)r^)r#floatmeanitem)rsipnmarginr8s rrz,VectorQuantizer2.forward..scB&r*f4;;==BBDDIIKKcQr )2rKr3float32r`shapedetachclone zeros_likecudaampautocastzerosr(rLr.r* enumerater)F interpolatepermutereshape normalizeargmaxr7r@rATsumsquareaddmm_argminbincounttrainingtdist all_reduceview contiguousrwaitr5r#copy_mul_add_mulmse_lossrget_world_sizenumel)r8rH ret_usagesrKBCHW f_no_gradf_restf_hat mean_vq_loss vocab_hit_VSNrcrdrest_NCidx_N d_no_gradhit_Vhandleridx_Bhwh_BChwusagesres` @rforwardzVectorQuantizer2.forwardKs7  EM ! $\\^^F\ 1aMMOO "" (( Z^ $ $U $ 3 3H :H :),L+u{6=KT&''B#D$566? n? nB#; "q&LG fB8&III Aq!,, Q#^^Aq!Q77??AFF  k'r:::G!L!+dn.C.H.JPQ"R"R"RREE"q&LG fB8&III Aq!,, Q#^^Aq!Q77??AFF !& ((a!!! -299;;E!I $$!6!;!=Ra%"L:::EAAGGII=E#.utDDDG **QB//b1f RAMw//771aCCV&!jlll0088Aq!DDOOQQ8rAv7??& =)LLNNN!+S-b177>>>>3.S-b166s;;@@3PPPP-b166t<<AA%))D//RRROOq(OO  ''' 5:v > > C CDI N NQRQ[\aclQmQm mm C"H $LZ)+11&99EQH :H :H :H :H :H :H :H :H :H :H :H :H :H :H :V  " "||~~ Q/ 1o     '(9::FF Ffl**sS(VVVT ms_h_BChwcg}|djd}|jdx}}t|j}|r|d||j||t j} t|jD]\} } || } | t|jdz krtj | ||fd} |j | |dz z | } | | |r| }w| | n|d||j|jd|jdt j} t|jD]o\} } tj | | | fd} |j | |dz z || } | | |r| }Z| | p|S)NrrSrKr!r]rN)rgr*r. new_zerosrr3rfrorprqrrappendri) r8rall_to_max_scalelast_one ls_f_hat_BChwrrrrrrcrdrs r embed_to_fhatzVectorQuantizer2.embed_to_fhats  aL q !!"%%A " # #  0aL**1diAU]*SSE#D$566 8 8B"2D-..22P]6AYOOOF7rAv7?? 6"""8$)MM!((7777 8aL** !!$!!$m +E$D$566 0 0B e2r(KKK7rAv7 " FF 6"""0$)MM!((////r to_fhatr* noise_stdc f|j\}}}}|} | } tj| } g} d|p|jD} | dd|kr| dd|ksJd| dd|d|dt | }t| D]\}\}}||dz krCtj | ||fd  dd d d d|n+| dd d d d|}|5tj d|d zz |ztj||zz}|jrXtj|d }tj|tj|jjjjd zd }ntj|ddtj|jjjddz}|||jjjjddtj|d }||||}||dz krTtj || dd dd ||fd n<|| dd dd }|j||dz z |}| || || |r| n| |||z| S)Nc`g|]+}t|tr||fn|d|df,S)rr!) isinstanceint)rrds rrz7VectorQuantizer2.f_to_idxBl_or_fhat..sK   #2s++ ?RHH"Q%A   r rSrr!zpatch_hws[-1]=z != (H=z, W=)rMrNrQrRrTTrVFrXrYr]) rgrhrir3rjr*r.rorprqrrrsmathsqrt randn_liker)rtrur7r@rArvrwrxryrzrrrrsub_r)r8rHrr*rrrrrrrrf_hat_or_idx_Bl patch_hwsrrcphpwz_NCrrrrs rf_to_idxBl_or_fhatz#VectorQuantizer2.f_to_idxBl_or_fhats\ 1aMMOO "" ((.0  #8t'8   bM!  ! /&/mA&6!&; / / .im . .1 . . . . . / / /^^%i00) ) LBR "q&L? fB8&AAAAq!$$Q^^Aq!Q//77A>>  ayY!^!344t;e>Nt>T>TW`>`` 7{4R000 1;t~'<'A'CKKKKQR"IdkkmmDIIIEIN).5577QMMM   $./46bq! YA666jjB++G"q&L N NN7++33Aq!Q??Q"*,,,^^G,,44Q1a@@KKMM 4T_R26]3F;;F JJv    KK     " "!(G emmArBw.G.G    r gt_ms_idx_Blc Fg}|djd}|j}|jdx}}t|j}|d||||t j}|jd} t|dz D]} tj | ||  dd ||| | ||fd} | |j| |dz z | |j| dz} |tj || | fd ||dddt|rt j|d ndS) NrrSrr!rQr]rNrMrT)rgrr*r.rr3rfr-rprqr7 transpose_rrrr transposecat) r8r next_scalesrrrrrrpn_nextrcrs ridxBl_to_switti_inputz&VectorQuantizer2.idxBl_to_switti_input s O !! $ I!"%%A " # #Q))!Q1EM)JJ(+Q--  B]|B/00Aq!!aGW--V F JJ5trR!V}5f== > > >'Q/G    e7G*<6JJJaB1a     14K0@0@Juy!,,,,dJr rcrrrc|jd}||dz kr|j||dz z tj|||fd}|||tj||j|dz|j|dzfdfS|j||dz z |}||||fS)NrSr!r]rNrM)r*rrprqr)r8rcrrrHWhs rget_next_autoregressive_inputz.VectorQuantizer2.get_next_autoregressive_input<s r " a< .b1f . fB8)DDDA JJqMMM!-'Q/1B261JK  /b1f .v66A JJqMMM%< r )r rNrr)F)TF)NN)__name__ __module__ __qualname__r`r'rCstrrGr3Tensorrrrrrboolrrr LongTensorrrr __classcell__r;s@rr r sD-B-B  -B-B-B-B-B-B^UCUUUU 05d+d+ld+ u|T%[%,6 7d+d+d+d+RNS&&el+& tEL!5</ 0&&&&XIM%) BB BBxc5c?.B(CDE B E? B eEL%"223 4 BBBBJK$u|2DKKKKK8   ',| =B\ x %u|3 4        r c(eZdZfdZfdZxZS)rcd}t|||d|dzt||_dS)NrRr!rQ) in_channels out_channels kernel_sizestridepadding)r&r'r resi_ratio)r8 embed_dimrksr;s rr'z Phi.__init__QsO  !"!G    j//r c|d|jz t||jzS)Nr!)rrr&rr)r8rr;s rrz Phi.forward\sJzz!do-..1H1H1M1M O2 2   r )rrrr'rrrs@rrrPsQ * * * * *         r rc0eZdZdeffd ZdefdZxZS)r/qresicVt||_dSN)r&r'r)r8rr;s rr'zPhiShared.__init__cs$  r rDc|jSr)r)r8rs r __getitem__zPhiShared.__getitem__gs zr )rrrrr'rrrs@rr/r/bs_ c      r r/cJeZdZdejffd ZdedefdZde fdZ xZ S)r0qresi_lsct||_t|}|dkrt jd|z dd|z z |nt jd|z dd|z z ||_dSNrgUUUUUU?r!r)r&r'rr.nplinspaceticks)r8rKr;s rr'zPhiPartiallyShared.__init__ls   MMAv :BK 1uqy=! 4 4 4UQYEAI q99 r at_from_0_to_1rDc|jtjtj|j|z Sr)rrrzrrrb)r8rs rrzPhiPartiallyShared.__getitem__vs6}RYrvdj>.I'J'JKKPPRRSSr cd|jSNzticks=rrFs rrGzPhiPartiallyShared.extra_repry$ $$$r ) rrrr r1r'r`rrrrGrrs@rr0r0ks       T%TCTTTT%C%%%%%%%%r r0cDeZdZdeffd Zdedeffd ZdefdZ xZ S)r,rct|t|}|dkrtjd|z dd|z z |ntjd|z dd|z z ||_dSr)r&r'r.rrr)r8rrr;s rr'zPhiNonShared.__init__~s{  JJAv :BK 1uqy=! 4 4 4UQYEAI q99 r rrDcttjtj|j|z Sr)r&rrrzrrrb)r8rr;s rrzPhiNonShared.__getitem__sIww"" IbfTZ.899 : : ? ? A A   r cd|jSrrrFs rrGzPhiNonShared.extra_reprrr ) rrrrr'r`rrrrGrrs@rr,r,}s d       % C      %C%%%%%%%%r r,)rtypingrrrrrnumpyrr3rr}r torch.nnr rp__all__Moduler Conv2drr/r0r1r,r rrs| 99999999999999 &&&&&&$$$$$$       ry   D      ")   $ %%%%%%%%$%%%%%2=%%%%%r