o `Df @s>ddlmZddlmZddlmZmZmZGdddZdS)) defaultdictmean)"AutoModelForSequenceClassification AutoTokenizerpipelinec@s*eZdZddZddZ  dddZdS) RegardcCs2||_td}td}td|d|dd|_dS)Nzsasha/regardv3ztext-classificationT)modelZtop_k tokenizer truncation) config_namerZfrom_pretrainedrrregard_classifier)selfr Zregard_tokenizerZ regard_modelrX/Users/zekunwu/Desktop/LLM_Open_Generation_Bias/LLM-Open-Generation-Bias/utils/metric.py__init__ s     zRegard.__init__cCsHtt}||}|D]}|D] }||d|dqq |t|fS)Nlabelscore)rlistrappenddict)rgroupZ group_scoresZ group_regardpredZ pred_scorerrrregards  z Regard.regardNc s|jdkrU||\}}||\}}dd|Ddd|D}dd|Ddd|D} |dkr@|| dS|d krId Sd fd dDiS||\}}d d|Ddd|D}|dkrvd|iS|d kr~diSd|iS)NcomparecSi|] \}}|t|qSrr.0kvrrr !z"Regard.compute..cSrrmaxrrrrr!"r"cSrrrrrrrr!#r"cSrrr#rrrrr!$r"maximum)Zmax_data_regardZmax_references_regardaverage)Zaverage_data_regardZaverage_references_regardZregard_differencecs"i|] }|||dqS)r)get)rkeyZ pred_meanZref_meanrrr!-s"cSrrrrrrrr!0r"cSrrr#rrrrr!1r"Z max_regardZaverage_regardr)r ritems) rdata references aggregationZ pred_scoresZ pred_regardZ ref_scoresZ ref_regardZpred_maxZref_maxrr)rcomputes,  zRegard.compute)NN)__name__ __module__ __qualname__rrr.rrrrrs  rN) collectionsr statisticsr transformersrrrrrrrrs