U Ú–bêã@s\dZddlmZddlmZmZdd„Zdd„Zdd „Zd d „Z d d „Z dd„Z dd„Z dS)z) Helpers to train with 16-bit precision. éN)Ú_flatten_dense_tensorsÚ_unflatten_dense_tensorscCs:t|tjtjtjfƒr6|jj ¡|j_|jj ¡|j_dS)z/ Convert primitive modules to float16. N) Ú isinstanceÚnnÚConv1dÚConv2dÚConv3dÚweightÚdataÚhalfÚbias©Úl©rú4/home/zsyue/code/python/GradDiff/models/fp16_util.pyÚconvert_module_to_f16 srcCs:t|tjtjtjfƒr6|jj ¡|j_|jj ¡|j_dS)zP Convert primitive modules to float32, undoing convert_module_to_f16(). N) rrrrrr r Úfloatr r rrrÚconvert_module_to_f32srcCs(tdd„|Dƒƒ}t |¡}d|_|gS)zb Copy model parameters into a (differently-shaped) list of full-precision parameters. cSsg|]}| ¡ ¡‘qSr)Údetachr©Ú.0ÚparamrrrÚ !sz&make_master_params..T)rrÚ ParameterÚ requires_grad©Ú model_paramsÚ master_paramsrrrÚmake_master_paramss  ÿ rcCstdd„|Dƒƒ|d_dS)zp Copy the gradients from the model parameters into the master parameters from make_master_params(). cSsg|]}|jj ¡ ¡‘qSr)Úgradr rrrrrrr.sz/model_grads_to_master_grads..rN)rrrrrrÚmodel_grads_to_master_grads(s ÿr cCs4t|ƒ}t|t||ƒƒD]\}}| ¡ |¡qdS)zH Copy the master parameter data back into the model parameters. N)ÚlistÚzipÚunflatten_master_paramsrÚcopy_)rrrZ master_paramrrrÚmaster_params_to_model_params2s ÿ r%cCst|d ¡|ƒS)zD Unflatten the master parameters to look like model_params. r)rrrrrrr#@sr#cCs,|D]"}|jdk r|j ¡|j ¡qdS)N)rÚdetach_Úzero_)rrrrrÚ zero_gradGs  r() Ú__doc__Útorch.nnrZ torch._utilsrrrrrr r%r#r(rrrrÚs