o tDh@s^ddlmZddlZddlmZddlmZmZddlm Z ej ej ej dej dfddZ dS) )partialN)FullyShardedDataParallel)MixedPrecisionShardingStrategy)lambda_auto_wrap_policyTc s4t||ttfdddt|||d||dS)Ncs |jvS)N)blocks)mmodel&/home/user/app/wan/distributed/fsdp.pys zshard_model..) lambda_fn) param_dtype reduce_dtype buffer_dtype)module process_groupsharding_strategyauto_wrap_policymixed_precision device_idsync_module_states)FSDPrrr)r rrrrrrrr r r shard_model s    r) functoolsrtorchtorch.distributed.fsdprrrrtorch.distributed.fsdp.wraprbfloat16float32 FULL_SHARDrr r r r s