ٳfdZddlZddlZddlZddlZddlZddlZddlZddlmZ ddlm Z dZ dZ dZdZd Zd Zd Zd ed eddfdZddZddlZddeefdZdS)zs Author: Luigi Piccinelli Licensed under the CC-BY NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/) N) distributed)multiprocessingcZtjsdStjsdSdS)NFT)dist is_availableis_initialized4/home/hossein/UniDepth/unidepth/utils/distributed.pyis_dist_avail_and_initializedr s3    u   u 4r cHtsdStjSNr)r rget_rankr r r rrs! ( * *q =??r cLtsdStjdS)N)r rbarrierr r r rr s$ ( * *LNNNNNr c&tdkSr)rr r r is_main_processr&s ::?r c|jdkSr)rank)argss r is_rank_zeror*s 9>r ctjr:tjr'tj}tj}nd}d}||fS)Nr)rrrrget_world_sizer world_sizes r get_dist_infor.sS t244}(**   r c tjdkrg|dd}tjd}|$||krt jd|d|d |d tj|d |d d }tj ||dd}dtj vr<|dkr6d}t jd|dt|tj d<dtj vr\|dkrXtj dd}t jd|dt|tj d<dSdSdS)z-Setup multi-processing environment variables.Windowsmp_start_methodforkT) allow_noneNzMulti-processing start method `z*` is different from the previous setting `z`.It will be force set to `zM`. You can change this behavior by changing `mp_start_method` in your config.)forceopencv_num_threadsrworkers_per_gpuOMP_NUM_THREADSrzDSetting OMP_NUM_THREADS environment variable for each process to be z in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.MKL_NUM_THREADSzDSetting MKL_NUM_THREADS environment variable for each process to be ) platformsystemgetmpget_start_methodwarningswarnset_start_methodcv2 setNumThreadsosenvironstr)cfgr current_methodr$r%omp_num_threadsmkl_num_threadss r setup_multi_processesr:8sI%%''"3V<<,===  %.O*K*K MO/OO8FOO,;OOO    O48888!5q99()))gg/33O **/B/B  :$ : : :   ),O(<(< $% **/B/B*..):A>>  :$ : : :   ),O(<(< $%%%+*/B/Br backendportreturnc ttjd}ttjd}tjd}tj}tj||ztjd|d}t|tjd<|tjd<t|tjd<t||ztjd <t|tjd <t|||||z||tjdtj d tj ||| d S)aInitialize slurm distributed training environment. If argument ``port`` is not specified, then the master port will be system environment variable ``MASTER_PORT``. If ``MASTER_PORT`` is not in system environment variable, then a default port ``29500`` will be used. Args: backend (str): Backend of torch.distributed. port (int, optional): Master port. Defaults to None. SLURM_PROCID SLURM_NTASKSSLURM_NODELISTzscontrol show hostname z | head -n1 MASTER_PORT MASTER_ADDR WORLD_SIZE LOCAL_RANKRANKz nvidia-smi -LrN)intr3r4torchcuda device_count set_device subprocess getoutputr5printr*rinit_process_group)r;r<proc_idntasks node_listnum_gpusaddrs r setup_slurmrUes>"*^,--G N+ , ,F +,Iz&&((H J'H,---   P) P P P Q QD #D BJ} $BJ}"6{{BJ|"7X#566BJ|WBJv (  =! /""    G'fEEEEEEr Tc~ &tjrtjsStjtjj}tj|}tj |j  fdt|D}tj | t|}| z }|r7tj|j j}tj|ffdt|D} tj | g} t'| |D]"\} | d| #|rtj| dS| S)N)devicec8g|]}tjSr rH zeros_like).0_ local_sizes r z+sync_tensor_across_gpus..s$III!!*--IIIr )rWdtypec8g|]}tjSr rY)r[r\ts r r^z+sync_tensor_across_gpus..s$FFFqu'**FFFr r)dim)rrrrH atleast_1dgroupWORLDrrtensorsizerWrange all_gathermaxitemzerosr_catzipappend) rarbrmrd group_size all_sizesmax_size size_diffpaddinggather_t_tensorall_tsrgr]s ` @r sync_tensor_across_gpusrwsy*,,y1D1F1Fy A J E"11%88JaffSkk!(;;;JIIIIuZ7H7HIIIIOIz***9~~H *//"3"33I$+iHHH Iq'l # #FFFFE*4E4EFFFOOOQ''' F 22  4 ah (yQ'''' Mr keysctj|tj}tj|tj|}t|dd}d|D}|S)N)protocol)r_rF)rbrmcg|]L}tjt|D]}|MSr )pickleloadsbytescputolist)r[rxkeys r r^z+sync_string_across_gpus..se    <dhhjj&7&7&9&9 : :;;        r )r|dumpsHIGHEST_PROTOCOLrH frombufferuint8torw)rxrWrbkeys_serializedkeys_serialized_tensors r sync_string_across_gpusrsl4&2IJJJO"-oU[QQQTT5A5  *   D Kr )rT)r)__doc__r3r)rLr.r1rHtorch.utils.data.distributedrrrr,r rrrrrr:r5rUrwr|listrr r r rsb   ####%%%%%%''''''  *=*=*=Z F FC FD F F F FF4   $s)      r