o pBg@sddlZddlZddlZddlmZddlZddlmZm Z ddlm Z m Z m Z m Z mZm Z ddZddZ  d ded ed e fd d ZdS)N)PretrainedConfigPreTrainedModel) AutoTokenizerAutoModelForCausalLM AutoConfigBitsAndBytesConfigrrc Cstt|dr&tt|dr&tt|ds$tt|dr&dSddlm}m}ddlm}m}|}z| |}Wn|yT}zd }WYd}~nd}~ww|rm||drm||drm||dsk||drmdSd S) Nzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelztokenizer.jsonTr)HfApi file_exists)validate_repo_idHFValidationErrorF) ospexistsjoinhuggingface_hubrr huggingface_hub.utilsr r repo_exists)pathrr r r api valid_hf_repoerE/home/l/lo/longlian/describe-anything/model/language_model/builder.py has_tokenizers<rcCsZt|dd}t|dd}|r+||kr+td|d|tt||}d|d|_|S)Nmax_position_embeddingsmodel_max_lengthzScaling RoPE from z to linear)typefactor)getattrprintfloatmathceil rope_scaling)configZ orig_ctx_lenrscaling_factorrrrcontext_length_extension+s    r&model_name_or_pathr$returnc Ost|}||_||_|durt|tj|g|R|t|jd|}|}t|s6t dt |d}d|vrDt j||jdd} nd|vrUt j||jddd } n t j||jdddd } |jj|_|| fS) N)r$ torch_dtypezGtokenizer found in VLM root folder. Move to ./{VILA}/llm in the future.llmmptright)r padding_sideyiF)rr-use_fast)rr-r/legacy)rfrom_pretrained_attn_implementationrr&reval model_dtyperwarningswarnr rrlowerr$ hidden_size) r'r$attn_implementationrargskwargsllm_cfgr*Zllm_path tokenizerrrrbuild_llm_and_tokenizer5sP      r>)NN)r!r5osos.pathrr torch transformersrrrrrrrr&strr>rrrrs"