U lxd@slddlZddlZddlZddlZddlZddlmZddlm Z ddlm Z ddl m Z ddZ dd ZdS) N)Saver)utils)autocast) GradScalerc Csptd|d}t|}g}tt|D]\}} | dddd} | dddd} tdtd ||| | D]"} | ds| |  |j | | <qtd | ddt } || d | d | d | ddd|j j|j jd}| || d }t }|| }|jd|jj}||}td|||||t|jjD]:}|| d | d | d | d| ddd}||7}q\|| d| d| d|| dd}tj||jjd\}}t|jdkrt|}t|d |}|| d| d|| d| d|iq4W5QRX||jj}||}td|tdt ||S)Nz [*] testing...gnamer/z--------z {}/{} - {}z>>unitsf0volumespk_idT)gt_specinferZ infer_speedupmethodzRTF: {} | {} / {}melF)rr_z.wavname_ext)srz_gt.wavz _pred.wavz [test_loss] test_loss:z Real Time Factor)!printevallentorchno_grad enumeratesplitformatkeys startswithtodevicetimerspeeduprshapedata sampling_rateappendrangetrain batch_sizeitemlog_speclibrosaloadto_mono from_numpy unsqueezeZ log_audionpmean)argsmodelvocoder loader_testsaver test_loss num_batchesZrtf_allZbidxr%fnZspeakerkZst_timersignalZed_timeZrun_timeZ song_timeZrtfiloss path_audioaudiorrB"D:\so-vits-svc\diffusion\solver.pytest sn       6  rDcCs t||d}td|i} |d|| t|} ||dt} |jjdkrbtj } n8|jjdkrvtj } n$|jjdkrtj } nt d|jj|d t |jjD]T} t|D]D\}}|||D]"}|d s|||j||<q| tj krH||d |d |d |d|d|ddd}nHt|j| d4||d |d |d |d|d|ddd}W5QRXt|rt dnF| tj kr||n | || || ||j|jjdkrh|jdd}|d | || |j!j"|jj|#||$|%|j |&d|$i|&d|i|j|jj'dkr|jj(r|nd}|j)|||jd|j|jj'}||jj*dkr|j+|dt,|||||}|d ||&d|i|qqdS)N)initial_global_stepr5z--- model size ---z======= start training =======Zfp32fp16Zbf16z [x] Unknown amp_dtype: z;epoch|batch_idx/num_batches|output_dir|batch/s|lr|time|steprr r r r aug_shiftrF)rGrr) device_typedtypez [x] nan loss rlrz_epoch: {} | {:3d}/{:3d} | {} | batch/s: {:.2f} | lr: {:.6} | loss: {:.3f} | time: {} | step: {}z train/lossztrain/lr)postfixz% --- --- loss: {:.3f}. zvalidation/loss)-rrget_network_paras_amountZlog_inforr)rZ amp_dtyperfloat32float16bfloat16 ValueErrorr(epochsrZglobal_step_increment zero_gradrrr r!floatrisnanbackwardstepscaleupdate global_stepZ interval_log param_groupsrenvexpdirZget_interval_timer+get_total_time log_valueZ interval_valZsave_optZ save_modelZinterval_force_saveZ delete_modelrD)r4rEr5 optimizer schedulerr6 loader_trainr7r8Z params_countr:ZscalerrIepochZ batch_idxr%r<r?Z current_lrZoptimizer_saveZ last_val_stepr9rBrBrCr)Xs                 r))osr"numpyr2rr-Zdiffusion.logger.saverrdiffusion.loggerrrZtorch.cuda.amprrDr)rBrBrBrCs    M