B b@sddlZddlZddlmZddlmmZddlZddl Z e j ej e ddlmZddlZddlZddlZddlZddZddZGdd d eZGd d d eZGd d d eZddZd ddZddZddZdefddZ e!dkre dddd\Z"Z#ddl$Z$e$$Z%xfe&dD]ZZ'xRe(e"D]F\Z)Ze\Z*Z+e)ddkr4e$$Z,e-e'e)e+.e,e%de,Z%q4Wq&WdS)!N)Dataset)read_and_config_filecCsNt|\}}t|}|dkr,t||d}t|jdkrJ|dddf}|S)Ni>r)sfread audio_normlibrosaresamplelenshape)pathdatafsr/mnt/nas_sg/mit_sg/shengkui.zhao/ComplexNN/ComplexNN_16k_V5_UniDFSMN_W40S20_multiUnetx2_residual_mask_Ch_FSMN_full_DNS_Challenge_reverb_20211123/ComplexNN_16k_MaaS/tools/time_dataset.py audioreadsrcCsX|dd}d|}||}|d}|}|||kd}d|}||}|S)Ng?g&-`ʬ?)mean)xrmsscalarZpow_xZ avg_pow_xZrmsxZscalarxrrrrsrc@s,eZdZddZddZddZddZd S) DataReadercCst|dd|_dS)NT)decode)r file_list)self file_namerrr__init__)szDataReader.__init__cCsL|d}|dd}t|tj}t|d|jdg}|||jdfS)Ninputs/rr)splitrastypenpfloat32reshaper )rr utt_idr rrrrextract_feature,s zDataReader.extract_featurecCs t|jS)N)r r)rrrr__len__3szDataReader.__len__cCs||j|S)N)r&r)rindexrrr __getitem__6szDataReader.__getitem__N)__name__ __module__ __qualname__rr&r'r)rrrrr(src@seZdZddZdS) Processerc Cst|d}t|d}|jd}|jd|krvtj|tjd}tj|tjd}||d|jd<||d|jd<n0td||} || | |}|| | |}||fS)Nrlabelsr)dtype)rr r"zerosr#randomrandint) rr start_timesegement_lengthZ wave_inputsZwave_s1Zlen_wavZ padded_inputsZ padded_s1Zst_idxrrrprocess<s  zProcesser.processN)r*r+r,r5rrrrr-:sr-c@s0eZdZddedfddZddZddZdS) TimeDataseti>NcCs@t||_||_t}||_||_t|j|j|j|dS)a scp_file_name: the list include:[input_wave_path, output_wave_path, duration] spk_emb_scp: a speaker embedding ark's scp segement_length: to clip data in a fix length segment, default: 4s sample_rate: the sample rate of wav, default: 16000 processer: a processer class to handle wave data gender2spk: a list include gender2spk, default: None N) rwav_list processermpManagerlistr(r4_dochunk)r scp_file_namer4 sample_rater8Z gender2spkZmgrrrrr]s   zTimeDataset.__init__cCs t|jS)N)r r()rrrrr'tszTimeDataset.__len__cCs:|j|\}}|j|d|dd||j\}}||fS)Nrr.)rr.)r(r8r5r4)rr( data_infor3rs1rrrr)ws$zTimeDataset.__getitem__)r*r+r,r-rr'r)rrrrr6[s  r6c Csx|||D]}|d}t||}||krPd} |d|kr@q||dgqd} x&| ||krz||| g| |7} qVW| |kr||t||gqWdS)Ndurationrrr)intappend) Z target_listZ result_liststartendr4r>itemrAlengthZ sample_indexrrrworker|s"    rH c Csg}t||}|dkrJtjt||dt|||fd}|||ndxbt|D]V}||dkrnt|} n |d|} tjt||||| ||fd}|||qTWx|D] }|qWdS)Ndr)targetargsr)r r9ProcessrHrDrCrangejoin) r7r(r4r> num_threadsZpc_liststridepidxrErrrr<s:        r<cCsttdd|D}tt|||djdg}t|tj}x0t|D]$\}}|||d|jdddf<qHW|S)Ncss|]}|jdVqdS)rN)r ).0inprrr sz"zero_pad_concat..rr)maxr"arrayr r r0r# enumerate)rZmax_tr Z inputs_matrSrUrrrzero_pad_concats  rZcCs@t|\}}tj|tjd}tj|tjd}t|t|fS)N)r/)zipr"rXr#torch from_numpy)r rr@rrr collate_fns r^cCs*t||d}tj|||tddd}||fS)N)r8TF) batch_size num_workersr^shuffle drop_last)r6tud DataLoaderr^)r=r_r`r8datasetloaderrrr make_loaders rg__main__z../data/cv_wsj0_-5~20.lst )r` rJ)rI)/numpyr"r\torch.utils.datarutilsr rcossysr rCdirname__file__miscrmultiprocessingr9 soundfilerrr1rrobjectrr-r6rHr<rZr^rgr*Zlaoder_timestimerNepochrYrSrr.etimeprintsizerrrrs@    !! *