o %gU6@s ddlZddlmZddlmZddlZddlZdZddlmZmZm Z ddl Z ddl Z dZGdddZd"d d Zd d ZddZ     d#dejdede jfddZ     d$dejdede jfddZeZdZdZdZdZddZddZeefd d!ZdS)%N)YOLO)Imagecuda)r ImageDraw ImageFontzagents/ui_agent/util/arial.ttfc@s4eZdZddZddZddZddZd d Zd S) MarkHelpercCs"i|_i|_d|_d|_d|_dS)Ng{Gz?) markSize_dict font_dict min_font_size max_font_sizemax_font_proportion)selfrA/home/jianwyan/projects/release/Magma/agents/ui_agent/util/som.py__init__s  zMarkHelper.__init__c Cs:td||f}t|}|jd||d\}}}} | |fS)NRGB)rr)textfont)rnewrDrawtextbbox) rr image_height image_widthrimdraw_widthheightrrrZ__get_markSizes zMarkHelper.__get_markSizec Cs|d|}|j}tt|}t|d|||t|j|jt||krF|d7}tt|}t|d|||t|j|jt||ks&||j|<|d|||}|d|||}|d|||}|||d|j |<dS)NrZ555555)r ) r rtruetype font_pathmin_MarkHelper__get_markSizer rr r ) rrrkeyfontsizerZmarkSize_3digitsZmarkSize_2digitsZmarkSize_1digitrrr_setup_new_fonts , , zMarkHelper._setup_new_fontcCs.|d|}||jvr||||j|S)Nr)r r+)rrrr)rrrget_font5s   zMarkHelper.get_fontcCsX|d|}||jvr||||j|dd}|j|t||\}}||fS)z1Get the font size for the given image dimensions.rr$N)r r+getlen)rZtext_strrrr)Z largest_sizetext_htext_wrrr get_mark_size;s   zMarkHelper.get_mark_sizeN)__name__ __module__ __qualname__rr(r+r,r1rrrrrs  rFcCs|\}}}}|\}}} } t||} t||} t|||| } t|||| }td| | td|| }||}| | }|t||d}|rM||fS|S)z Calculate the Intersection over Union (IoU) of two bounding boxes. :param box1: Tuple of (y, x, h, w) for the first bounding box :param box2: Tuple of (y, x, h, w) for the second bounding box :return: IoU value r-C6?)maxr')box1box2 return_areay1x1h1w1y2x2h2w2y_minx_miny_maxx_maxintersection_areaZ box1_areaZ box2_areaiourrr__calculate_iouEs    rHcCs|\}}}}|\}}}} t||g|||g|||g||||gg} t||g||| g|||g|||| gg} tjj| ddtjf| dd} t| } | S)zPCalculate the distance between the nearest edge or corner of two bounding boxes.Nr#)axis)nparraylinalgnormnewaxisr')r7r8r:r;r<r=r>r?r@rAZcorners1Zcorners2 distances min_distancerrr#__calculate_nearest_corner_distancecs"      " rQc Cs\|\}}}}|\} } |\} } || |f|| ||| f|||f||| ||f||||| f|||f||| || f||| fg} | d}td}| D]Y}|d|d| | f}|ddksz|ddksz|d| | ksz|d| | kr{qR| |  }||D]}t||rqt||ddd}t||dt||}q||kr|}|}qR|S)aFind the corner with the least overlap with other bboxes. Args: bbox: (y, x, h, w) The bounding box to place the text on. bboxes: [(y, x, h, w)] The list of bounding boxes to compare against. drawn_boxes: [(y, x, h, w)] The list of bounding boxes that have already been drawn on. text_size: (height, width) The size of the text to be drawn. image_size: (height, width) The size of the image. rinfr T)r9r5)floatrJ array_equalrHr6rQ)bboxbboxes drawn_boxes text_size image_sizeyxhwZh_textZw_textrrcornersZ best_cornerZmax_flagcornerZ corner_bboxmax_iouZ other_bboxrGrrr_find_least_overlapping_cornerzs:     8   rar#Timage mark_helperreturnc s$t|} |j\|rfdd|D}|D]} | \} } } }| j| | | || | g||dqg}t|D]O\}}t|}||\}}t|||||ff\}}||||f}|r~| j||||||gdd|}| j ||f|d|d| t |q7|dur| ||S) Plots bounding boxes on an image with marks attached to the edges of the boxes where no overlap with other boxes occurs. Args: image: The image to plot the bounding boxes on. bboxes: A 2D int array of shape (num_boxes, 4), where each row represents a bounding box: (y_top_left, x_top_left, box_height, box_width). If normalized_to_pixel is True, the values are float and are normalized with the image size. If normalized_to_pixel is False, the values are int and are in pixel. c@g|]\}}}}t|t|t|t|fqSrint.0rZr[r\r]rrrr @z)plot_boxes_with_marks..)outlinerred)fillwhite)rprN)rrsize rectangle enumeratestrr1rar,rappendrJrKsave)rbrVrc linewidthalpha edgecolorfn_savenormalized_to_pixeladd_markrboxrZr[r\r]rWidxrUrr/r0Zcorner_yZcorner_xtext_boxrrrkrplot_boxes_with_markss2   "   rc sj|j\|rfdd|D}t|} |D]} | \} } | j| | fd||dq|dur3|||S)recrfrrgrirkrrrlrmz+plot_circles_with_marks..)radiusrnrN)rrrrcirclerw) rbpointsrcrxrzr{r|r}rVrpointr[rZrrkrplot_circles_with_markss   rg?rg@cCs|\}}}}|\}} } } || kr| |||kp#| |ko#|| | |k} ||kr0||||kp;||ko;||| |k} | o?| S)zQcheck if two boxes are in the same line or close enough to be considered togetherr)bbox1bbox2vertical_thresholdhorizontal_thresholdr:r;r<r=r>r?r@rAZhorizontally_closeZvertically_closerrr_is_boxes_same_line_or_near s  00rcCsnt|}tj||ftd}t|D]#}t|d|D]}t||||||r3d|||f<d|||f<qq|S)z9Build the adjacency matrix based on the merging criteria.)dtyper )r.rJzerosrhranger)rVrr num_boxesAijrrr_build_adjacency_matrixs   rc st|dkr |fSt|}t|dddf|dddf|dddf|dddf|dddf|dddfgj}t|||}t|}tt|}t |}|dddf|dddf|dddf|dddff\}} || g} g} |D]R} t| } t | } | d fdd| Dt fd d | D}t fd d | D}tfd d | D}tfd d | D}| ||||||fqdd| D} | | fS)aMerge bboxes based on the adjacency matrix and return merged bboxes. Args: bboxes: A 2D array of shape (num_boxes, 4), where each row represents a bounding box: (y, x, height, width). text_details: A list of text details for each bounding box. vertical_threshold: The maximum vertical distance between two boxes to be considered in the same line. horizontal_threshold: The maximum horizontal distance between two boxes to be considered close. r Nrr$r# csg|]}|qSrrrjr) text_detailsrrrlQsz*merge_connected_bboxes..c3|]}|VqdSNrr) corners_yrr Tz)merge_connected_bboxes..c3rrrr) corners_xrrrUrc3rrrr) corners_y_maxrrrVrc3rrrr) corners_x_maxrrrWrcSs8g|]}|d|d|d|d|d|dfqS)r rr$r#r)rjrUrrrrl[s8)r.rJrKTrnxZfrom_numpy_arraylistconnected_componentscopysortedrvjoinr'r6)rVrrrrG componentsr^Z corners_hZ corners_wZ merged_bboxesZmerged_text_details componentindicesrBrCrDrEr)rrrrrrmerge_connected_bboxes&s0  d   Dr)F)r#rNNTT)r#NNTT)torch ultralyticsrPILriobase64devicerrnumpyrJnetworkxrr&rrHrQrandarrayrrZ markhelperZ!BBOX_DEDUPLICATION_IOU_PROPORTIONZ BBOX_GROUPING_VERTICAL_THRESHOLDZ"BBOX_GROUPING_HORIZONTAL_THRESHOLDZBBOX_AUG_TARGETrrrrrrrsb   7: :