o +e1@sddlZddlZddlmZmZmZddlZddlm Z m Z m Z m Z m Z mZddlmZddlZddlZd1dejdede eeeffd d Zd2d dZd3ddZddZddZe    d4de edede ede ejde ejf ddZe     d5de ededede ejde ejd e f d!d"Z!d#d$Z"ded%efd&d'Z# d6d(e ee eeffd)ed*e ej$fd+d,Z% -d7d.e eeee eeffffd/d0Z&dS)8N)Image ImageDraw ImageFont)OptionalUnionTupleListCallableDict)tqdmrrrimagetext text_colorc Cs|j\}}}t|d}tj||||ftjdd}tj}||d|<t||ddd} || dd||| dd} } t||| | f|d|d|S)Ng?dtyper) shapeintnponesuint8cv2ZFONT_HERSHEY_SIMPLEXZ getTextSizeZputText) r rrhwcoffsetimgfontZtextsizeZtext_xZtext_yr!=/data/xianyang/code/FateZero/ptp_utils_null_text_inversion.pytext_under_images   &r#r{Gz?c Cstt|tur t||}n|jdkr|jd|}n|g}d}tj|djtjdd}dd|D|g|}t|}|dj\}}} t||} ||} tj||| |d|| | | ddftjdd} t |D]-} t | D]&}|| | || | || | || |||| ||| |f<quqo|durt j t j |d d t| }||dSdS) NrrrcSsg|]}|tjqSr!)astyperr).0r r!r!r" 0szview_images..rT)exist_ok)typelistlenndimrrrrrrangeosmakedirspathdirnamer fromarraysave)imagesnum_rowsZ offset_ratio save_pathZ num_emptyZ empty_imagesZ num_itemsrrrrnum_colsZimage_ijZpil_imgr!r!r" view_images&s<      > r<Fc Cs|r|j|||ddd}|j|||ddd}nt|gd} |j| ||dd} | d\}}||||} |j| ||d}||}|S)Nr)encoder_hidden_statessamplerr prev_sample)unettorchcatchunk schedulerstep step_callback) model controllerlatentscontexttguidance_scale low_resourcenoise_pred_uncondnoise_prediction_text latents_input noise_predr!r!r"diffusion_stepGs rRcCsVd|}||d}|dddd}|dddd}|dtj}|S) Ng!ޅ@r>rg?rrr)r)decodeclampcpupermutenumpyr&rr)vaerIr r!r!r" latent2imageUs rYcCsV|durtjd|jj|d|df|d}|||jj|d|d|j}||fS)Nr) generator)rArandnr@ in_channelsexpandtodevice)latentrGheightwidthr[ batch_sizerIr!r!r" init_latent^s$re2@promptnum_inference_stepsrLr[racCst||d}}t|} |jdg| dddd} || j|jd} |j|dddd} || j|jd} t|||||| \}}t | | g}|j |t |j j D] }t||||||}qXt|j|}||fS)N max_lengthMptpaddingrlreturn_tensorsr)register_attention_controlr- tokenizerbert input_idsr_r`rerArBrD set_timestepsr timestepsrRrYvqvae)rGrhrHrirLr[rarbrcrd uncond_inputuncond_embeddings text_inputtext_embeddingsrIrJrKr r!r!r"text2image_ldmhs   r}@rMc Cst||d}} t|} |j|d|jjddd} || j|jd} | jjd} |jdg| d| dd }||j|jd}|| g}|sNt |}t |||| || \}}d d i}|j j |fi|t|j jD] }t|||||||}qmt|j|}||fS) NirlTrn)rprl truncationrqrrkrorr)rrr-rsmodel_max_length text_encoderrur_r`rrArBrerDrvr rwrRrYrX)rGrhrHrirLr[rarMrbrcrdr{r|rlryrzrJrIZextra_set_kwargsrKr r!r!r"text2image_ldm_stables4    rcsfddGddd}dur|d fdd d}|j}|D]4}d|dvr8||d dd7}q%d |dvrI||d dd 7}q%d |dvrY||d dd 7}q%|_dS) NcsXjttjjjjurjdnjdddddfdd }|S) NrcSsL|j\}}}|j}||||||}|dddd|||||}|SNrrrr)rheadsreshaperVselftensorrdseq_lendim head_sizer!r!r"reshape_heads_to_batch_dim "zRregister_attention_control..ca_forward..reshape_heads_to_batch_dimcSsL|j\}}}|j}||||||}|dddd|||||}|Srrrr!r!r"reshape_batch_dim_to_headsrzRregister_attention_control..ca_forward..reshape_batch_dim_to_headscs |}|}|}|j\}}} j} |} |du} | r|n|}|} |}| } | } |}td| | j}|durj||d}t |j j }|dddddf | dd}| |||jdd}|| }td||}|}|S)Nzb i d, b j d -> b i jrr)rzb i j, b j d -> b i d)rrto_qto_kto_vrAeinsumscalerfinformaxrepeat masked_fill_softmax) hidden_statesr=attention_maskZcross_attention_kwargsxrJmaskrdsequence_lengthrrqis_crosskvsimZ max_neg_valueattnout)rH place_in_unetrrrto_outr!r"forwards0             z?register_attention_control..ca_forward..forward)NN)rr+rAnnmodules container ModuleList)rrr)rH)rrrrrr" ca_forwards z.register_attention_control..ca_forwardc@seZdZddZddZdS)z3register_attention_control..DummyControllercWs|dSNrr!)rargsr!r!r"__call__sz.DummyController.__call__cSs d|_dSr)num_att_layers)rr!r!r"__init__s z.DummyController.__init__N)__name__ __module__ __qualname__rrr!r!r!r"DummyControllers rcsL|dvr|||_|dSt|dr$|D] \}}||||d}q|S)N)attn1attn2rchildren) module_name)rhasattrnamed_children)Znet_countrrrZnet__)r register_recrr!r"rs  z1register_attention_control..register_recrrdownrupmidN)r@rr)rGrHrZcross_att_countZsub_netsnetr!)rrHrr"rrs 7     rr word_placec s|d}tturfddt|Dn ttur gg}tdkrffdd|Ddd}d\}}tt|D]#}|t||7}|vrW||d|t||kre|d7}d}qBt |S) N csg|] \}}|kr|qSr!r!)r'r:word)rr!r"r(z!get_word_inds..rcsg|] }|gdqS)#)rSstrip)r'item)rsr!r"r(srr)rr) splitr+str enumeraterr-encoder/appendrarray) rrrsZ split_textrZ words_encodecur_lenptrr:r!)rsrr" get_word_indss$      rbounds prompt_ind word_indscCst|tur d|f}t|d|jdt|d|jd}}|dur-t|jd}d|d|||f<d|||||f<d||d||f<|S)Nrrr)r+floatrrrAarange)alpharrrstartendr!r!r"update_alpha_time_word$s .rrmcross_replace_stepsc st|tur d|i}d|vrd|d<t|dtd|}ttdD] }t||d|}q'|D].\}dkrdfddtdtD}t|D]\}} t| dkrct|||| }qRq6| |dtddd|}|S)Ndefault_)gg?rcsg|] }t|qSr!)r)r'r:keypromptsrsr!r"r(>rz2get_time_words_attention_alpha..r) r+dictrAzerosr-r/ritemsrr) r num_stepsrrsZ max_num_wordsZalpha_time_wordsr:rindsindr!rr"get_time_words_attention_alpha1s$    r)r )rr$N)F)rfrgNN)rfr~NNFr)rm)'rWrrAPILrrrrtypingrrrrr r Z tqdm.notebookr datetimer0ndarrayrrr#r<rRrYreno_gradr Generator FloatTensorr}boolrrrrTensorrrr!r!r!r"st  $ !    ,[