a e~^@sddlZddlmmZddlZddlZddlm Z m Z m Z m Z ddl mZmZmZmZddlmZmZddlmZmZmZmZddlmZddlmZddlmZdd l m!Z!m"Z"e#e$Z%Gd d d eZ&dS) N)CallableListOptionalUnion) CLIPTextModel CLIPTokenizerCLIPVisionModelCLIPImageProcessor) AutoencoderKLDiffusionPipeline) deprecateis_accelerate_availableis_accelerate_versionlogging) FrozenDict) DDIMScheduler) randn_tensor)MultiViewUNetModel get_cameracs@eZdZddgZd0eeeeee e e dfdd Z ddZ d d Zd d Zd dZd1ddZd2ddZeddZd3e dddZddZddZd4ddZd d!Zd"d#Zed$dd%d%dd&d'd$d(d)dd*dd(d+ed,feee j!e"e"e#e"e#ee"e#ee$ej%e&ej%feeee'e"e"ej(gdfe"e"d-d.d/Z)Z*S)5MVDreamPipelinefeature_extractor image_encoderF)vaeunet tokenizer text_encoder schedulerrrrequires_safety_checkerc stt|jdrd|jjdkrdd|d|jjd} tdd| dd t|j} d| d<t| |_t|jd r|jj d urd|d } td d| dd t|j} d| d <t| |_|j |||||||ddt |j jj d|_|j|ddS)N steps_offsetz*The configuration file of this scheduler: z; is outdated. `steps_offset` should be set to 1 instead of a(. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` filezsteps_offset!=1z1.0.0F)Z standard_warn clip_sampleTa has not set the configuration `clip_sample`. `clip_sample` should be set to False in the configuration file. Please make sure to update the config accordingly as not setting `clip_sample` in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` filezclip_sample not set)rrrrrrr)r)super__init__hasattrconfigrr dictrZ_internal_dictr Zregister_moduleslenrZblock_out_channelsvae_scale_factorZregister_to_config) selfrrrrrrrrZdeprecation_message new_config __class__+/data/tang/glrm/mvdream/pipeline_mvdream.pyr#s@       zMVDreamPipeline.__init__cCs|jdS)z Enable sliced VAE decoding. When this option is enabled, the VAE will split the input tensor in slices to compute decoding in several steps. This is useful to save some memory and allow larger batch sizes. N)rZenable_slicingr)r-r-r.enable_vae_slicingTsz"MVDreamPipeline.enable_vae_slicingcCs|jdS)z Disable sliced VAE decoding. If `enable_vae_slicing` was previously invoked, this method will go back to computing decoding in one step. N)rZdisable_slicingr/r-r-r.disable_vae_slicing]sz#MVDreamPipeline.disable_vae_slicingcCs|jdS)a Enable tiled VAE decoding. When this option is enabled, the VAE will split the input tensor into tiles to compute decoding and encoding in several steps. This is useful to save a large amount of memory and to allow the processing of larger images. N)rZ enable_tilingr/r-r-r.enable_vae_tilingdsz!MVDreamPipeline.enable_vae_tilingcCs|jdS)z Disable tiled VAE decoding. If `enable_vae_tiling` was previously invoked, this method will go back to computing decoding in one step. N)rZdisable_tilingr/r-r-r.disable_vae_tilingmsz"MVDreamPipeline.disable_vae_tilingrcCs~trtddrddlm}ntdtd|}|jjdkrZ|jddd tj |j |j |j fD]}|||qjd S) a Offloads all models to CPU using accelerate, significantly reducing memory usage. When called, unet, text_encoder, vae and safety checker have their state dicts saved to CPU and then are moved to a `torch.device('meta') and loaded to GPU only when their specific submodule has its `forward` method called. Note that offloading happens on a submodule basis. Memory savings are higher than with `enable_model_cpu_offload`, but performance is lower. >=z0.14.0r) cpu_offloadzG`enable_sequential_cpu_offload` requires `accelerate v0.14.0` or highercuda:cpuTZsilence_dtype_warningsN)r r accelerater5 ImportErrortorchdevicetypetocuda empty_cacherrr)r)gpu_idr5r<cpu_offloaded_modelr-r-r.enable_sequential_cpu_offloadts  z-MVDreamPipeline.enable_sequential_cpu_offloadcCstrtddrddlm}ntdtd|}|jjdkrZ|jddd tj d }|j |j |j fD]}||||d \}}qn||_d S) a Offloads all models to CPU using accelerate, reducing memory usage with a low impact on performance. Compared to `enable_sequential_cpu_offload`, this method moves one whole model at a time to the GPU when its `forward` method is called, and the model remains in GPU until the next model runs. Memory savings are lower than with `enable_sequential_cpu_offload`, but performance is much better due to the iterative execution of the `unet`. r4z 0.17.0.dev0r)cpu_offload_with_hookz?`enable_model_offload` requires `accelerate v0.17.0` or higher.r6r7Tr8N)Zprev_module_hook)r rr9rDr:r;r<r=r>r?r@rrrfinal_offload_hook)r)rArDr<hookrB_r-r-r.enable_model_cpu_offloads   z(MVDreamPipeline.enable_model_cpu_offloadcCs\t|jds|jS|jD]8}t|drt|jdr|jjdurt|jjSq|jS)z Returns the device on which the pipeline's models will be executed. After calling `pipeline.enable_sequential_cpu_offload()` the execution device can only be inferred from Accelerate's module hooks. _hf_hookexecution_deviceN)r$rr<modulesrIrJr;)r)moduler-r-r._execution_devices   z!MVDreamPipeline._execution_deviceN)do_classifier_free_guidancec Cs|durt|trd}n0|dur4t|tr4t|}ntdt|d|j|d|jjddd}|j}|j|d dd j} | j d |j d krt || s|j | dd|jjdd f} t d |jjd | t|jjdr|jjjr|j|} nd} |j||| d} | d} | j|jj|d} | j \} }}| d|d} | | ||d } |r|durvdg|}nt|t|urtdt|dt|dnNt|tr|g}n:|t|krtd|dt|d|d|d n|}| j d}|j|d|ddd}t|jjdr>|jjjr>|j|} nd} |j|j|| d}|d}|j d}|j|jj|d}|d|d}||||d }t || g} | S)a Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded device: (`torch.device`): torch device num_images_per_prompt (`int`): number of images that should be generated per prompt do_classifier_free_guidance (`bool`): whether to use classifier free guidance or not negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds`. instead. If not defined, one has to pass `negative_prompt_embeds`. instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.FloatTensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.FloatTensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. NrzA`prompt` should be either a string or a list of strings, but got . max_lengthTpt)paddingrPZ truncationreturn_tensorslongest)rRrSz\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: use_attention_mask)attention_maskrdtyper<z?`negative_prompt` should be the same type to `prompt`, but got z != z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) isinstancestrlistr' ValueErrorr=rZmodel_max_length input_idsshaper;equalZ batch_decodeloggerwarningr$rr%rVrWr>rYrepeatview TypeErrorcat)r)promptr<num_images_per_promptrNnegative_prompt batch_size text_inputsZtext_input_idsZuntruncated_idsZ removed_textrWZ prompt_embedsZbs_embedseq_lenrGZ uncond_tokensrPZ uncond_inputZnegative_prompt_embedsr-r-r._encode_prompts              zMVDreamPipeline._encode_promptcCsTd|jjj|}|j|j}|dddd}|dddd }|S)Nrr!g?r) rr%scaling_factordecodesampleclampr7permutefloatnumpy)r)latentsimager-r-r.decode_latentsSs zMVDreamPipeline.decode_latentscCsXdtt|jjjv}i}|r,||d<dtt|jjjv}|rT||d<|S)Neta generator)setinspect signaturerstep parameterskeys)r)r{rzZ accepts_etaextra_step_kwargsZaccepts_generatorr-r-r.prepare_extra_step_kwargs[sz)MVDreamPipeline.prepare_extra_step_kwargsc Cs|||||j||jf} t|trHt||krHtdt|d|d|durbt| |||d}n ||}||jj}|S)Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)r{r<rY) r(r[r]r'r^rr>rZinit_noise_sigma) r)rkZnum_channels_latentsheightwidthrYr<r{rwr`r-r-r.prepare_latentsps$    zMVDreamPipeline.prepare_latentscCszt|jj}|jtjkr,|dtj}|j|ddj }|j ||d}|j|ddj d}|j |dd }t ||fS) NrQ)rS)r<rYT)output_hidden_statesrdim)nextrrrYnpfloat32astypeuint8rZ pixel_valuesr>Z hidden_statesrepeat_interleaver; zeros_like)r)rxr<rirYZ image_embedsr-r-r. encode_images zMVDreamPipeline.encode_imagecCst|jj}t|dddddj|d}d|d}t j |dddd }|j|d }|j |j }||j jj}|j|dd }t||fS) Nrrorr!r<)rbilinearF)mode align_corners)rYr)rrrrYr; from_numpy unsqueezertr>F interpolaterencodeZ latent_distrrr%rprr)r)rxr<rirYZ posteriorrwr-r-r.encode_image_latentss$  z$MVDreamPipeline.encode_image_latentsrZr2g@rgrvzcuda:0)rhrxrr elevationnum_inference_stepsguidance_scalerjrirzr{ output_typecallbackcallback_steps num_framesc( Cs|jj|d|_|jj|d|_|jj|d|_|dk}|jj||d|jj}|durt|tj rp|j tj kstJ|j j|d|_ | ||| \}}|||| \}}|j||| ||d}|d\}}|dur|n|d}||| d|||j || d}|dur"t||ddj|j |d }nt||d dj|j |d }|j| d d }|| | }t|||jj}|j|d }t|D]z\} }!|rdnd}"t|g|"}#|j|#|!}#|#tj|!g||"|#j |d t|g||g||t|g|"d}$|dur@t|g||g||$d<t|g|g|$d<|jjfi|$}%|rv|%d\}&}'|&||'|&}%|jj|%|!|fi|dd id }| t|dks| d|kr| d|jjd kr|| dur| |d kr| | |!|qWdn1s0Y| dkr0|}n*| dkrP| |}|!|}n | |}t"|dr||j#dur||j#$|S)Nrg?)rhr<rirNrjr!rrT)rZ extra_viewrXFrr)total)x timestepscontextrcameraipZip_img return_dictZlatentpilrE)%rr>rrrZ set_timestepsrr[rndarrayrYrrrrrnchunkrrrrr'order progress_bar enumerater;rgZscale_model_inputtensorforwardrupdateryZ numpy_to_pilr$rEZoffload)(r)rhrxrrrrrrjrirzr{rrrrr<rNrZimage_embeds_negZimage_embeds_posZimage_latents_negZimage_latents_posZ_prompt_embedsZprompt_embeds_negZprompt_embeds_posZactual_num_framesrwrrZnum_warmup_stepsrit multiplierZlatent_model_inputZ unet_inputsZ noise_predZnoise_pred_uncondZnoise_pred_textr-r-r.__call__s    0      zMVDreamPipeline.__call__)F)r)r)N)N)+__name__ __module__ __qualname__Z_optional_componentsr rrrrr rboolr#r0r1r2r3rCrHpropertyrMrnryrrrrr;no_gradr<r\rrrintrur Generatorrr FloatTensorr __classcell__r-r-r+r.rs 9       "r)'r;torch.nn.functionalnn functionalrr}rvrtypingrrrr transformersrrrr diffusersr r Zdiffusers.utilsr r rrZdiffusers.configuration_utilsrZdiffusers.schedulersrZdiffusers.utils.torch_utilsrZmvdream.mv_unetrr get_loggerrrbrr-r-r-r.s