B 0c@sddlZddlZddlZddlZddlZddlZddlmZe e Z e dddZ de dddZGd d d Zejdd d ZdS)N)logging)seedcCs4t|dtjd<dtjd<dtjd<dtjd<dS) zP Helper function for reproducible behavior during distributed training. 1ZCUDA_LAUNCH_BLOCKINGz:16:8ZCUBLAS_WORKSPACE_CONFIGTrueZFLAGS_cudnn_deterministicZFLAGS_benchmarkN)set_seedosenviron)rr ,/home/aistudio/ppdiffusers/training_utils.pyenable_full_determinisms    r cCs,|dk r(t|tj|t|dS)z Args: Helper function for reproducible behavior to set the seed in `random`, `numpy`, `paddle`. seed (`int`): The seed to set. N)randomrnppaddle)rr r r r-s  rc@s2eZdZdZdddZd d Zed d Zd S)EMAModelz6 Exponential Moving Average of models weights r?UUUUUU?H.?cCs^t||_|jx|jD] }d|_q"W||_||_||_||_ ||_ d|_ d|_ dS)a @crowsonkb's notes on EMA Warmup: If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). Args: inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. power (float): Exponential factor of EMA warmup. Default: 2/3. min_value (float): The minimum EMA decay rate. Default: 0. TgrN) copydeepcopyaveraged_modeleval parameters stop_gradientupdate_after_step inv_gammapower min_value max_valuedecayoptimization_step)selfmodelrrrrrparamsr r r __init__>s   zEMAModel.__init__cCsNtd||jd}dd||j|j }|dkr:dSt|jt||jS)zN Compute the decay factor for the exponential moving average. rrg)maxrrrrminr)r"r!stepvaluer r r get_decayYs zEMAModel.get_decayc Csi}|j}||j|_x|D]\}}t|tr:q&y ||}Wn>tk r|j dkrn| d nt |}|||<YnX|jr||| |jd||}n(||j|| |jd|j|||<q&Wx|D]\}}|||<qW|j||jd7_dS)Nrfloat32T)r state_dictr*r!r named_parameters isinstancedictKeyErrorndimcastclonerrrcopy_dtypescale_add_ named_buffers load_dict)r"Z new_modelZema_state_dictZ ema_paramskeyparamZ ema_paramr r r r(es*   "     z EMAModel.stepN)rrrrr) __name__ __module__ __qualname____doc__r%r*rno_gradr(r r r r r9s  rworkccstjdkrtj}|dk}d}z2|sNt|d|d|tjdVWd|rt|d|d|dtjXndVdS) Nrrzmain local processz: waiting for the z to perform z: z completed z, releasing all replicas)r distributedget_world_sizeget_rankloggerdebugbarrier)descrankis_main_processmain_process_descr r r main_process_firsts   rL)N)rA) contextlibrrr numpyrrutilsr get_loggerr<rEintr rrcontextmanagerrLr r r r s   L