o ϝf+@sdZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZdd lmZddlZdd lmZdd lmZdd lmZddlmZddlmZddl m!Z!m"Z"ddl#m$Z$Gddde!Z%Gddde!Z&Gddde!Z'ddZ(e)dkr e*Z+ee+e+j,de-ddde+j,de-ddde+j,d e.d!d"e+j,d#e.d$d"e+j,d%e/d&d"e+j,d'd(d)e+j,d*e-d+gd,d-d.e+j,d/d(d0d1e+j,d2d(d3d1e+j,d4d(d5d1e+j,d6d(d7d1e+j,d8e-d9d:de+0Z1e(e1dSdS);a Chat with a model with command line interface. Usage: python3 -m fastchat.serve.cli --model lmsys/vicuna-7b-v1.5 python3 -m fastchat.serve.cli --model lmsys/fastchat-t5-3b-v1.0 Other commands: - Type "!!exit" or an empty line to exit. - Type "!!reset" to start a new conversation. - Type "!!remove" to remove the last prompt. - Type "!!regen" to regenerate the last message. - Type "!!save " to save the conversation history to a json file. - Type "!!load " to load a conversation history from a json file. N) PromptSession)AutoSuggestFromHistory) WordCompleter)InMemoryHistory) KeyBindings)Console)Live)Markdown)add_model_args) AWQConfig) ExllamaConfig) XftConfig) GptqConfig)ChatIO chat_loop)str_to_torch_dtypec@sReZdZddedefddZdefdd Zd efd d Zd dZdefddZ dS) SimpleChatIOF multilineprefixcCs||_||_dSN) _multiliner)selfrrrF/mnt/public/algm/yuantao_home/workspace/fastchat/fastchat/serve/cli.py__init__)s zSimpleChatIO.__init__returnc Cs|js t|d|jSg}t|d} ||zt}Wnty5}zWYd}~nd}~wwqd|j|S)N: z" [ctrl-d/z on empty line to end]: T )rinputrappendstripEOFErrorjoin)rroleZ prompt_datalineerrrprompt_for_input-s  zSimpleChatIO.prompt_for_inputr$cCst|dddddS)NrrTendflushprintrr$rrrprompt_for_output;szSimpleChatIO.prompt_for_outputcC|d}|D]'}|d}|d}t|d}||kr+td|||ddd|}qtd||dddd|SNrtext Tr(r*r!splitlenr,r#r output_streamZpreoutputsZ output_textZnowrrr stream_output>  zSimpleChatIO.stream_outputr1cC t|dSrr+rr1rrr print_outputJ zSimpleChatIO.print_outputN)Fr) __name__ __module__ __qualname__boolstrrr'r.r;r?rrrrr(s  rc@sleZdZeZeddddZddedefdd Zd e fd d Z d e fddZ ddZ de fddZ dS) RichChatIOescapeZentercCs|jjdSr)ZappZcurrent_buffernewline)Zeventrrr_Qsz RichChatIO._FrmousecCs>ttd|_tgdtdd|_t|_||_ ||_ dS)N)history)z!!exitz!!resetz!!removez!!regenz!!savez!!load$)wordspattern) rr_prompt_sessionrrecompile _completerr_consoler_mouse)rrrJrrrrUs zRichChatIO.__init__rcCsJ|jd|d|jj|jd|jt|jr|jndd}|j|S)N[bold]:F)Z completerrZ mouse_supportZ auto_suggestZ key_bindings) rSr,rOpromptrRrTrrbindings)rr$Z prompt_inputrrrr'_s zRichChatIO.prompt_for_inputr$cCs |jd|ddddS)NrU/|rV)rSr,replacer-rrrr.ls zRichChatIO.prompt_for_outputcCst|jdd>}|D]3}|sq |d}g}|D]}|||dr+|dq|dqtd|}||q Wdn1sHwY|j|S) zStream output from a role.)ZconsoleZrefresh_per_secondr1z```rz rN) rrS splitlinesr startswithr r#updater,)rr9Zliver:r1linesr%Zmarkdownrrrr;os"       zRichChatIO.stream_outputr1cCs|d|igdS)Nr1)r;r>rrrr?szRichChatIO.print_outputN)FF)rArBrCrrXaddrIrDrrEr'r.r;r?rrrrrFNs    &rFc@s>eZdZdefddZdefddZddZd efd d Zd S) ProgrammaticChatIOrcCsd}d}t|} t||kr|| d}||krnz tjd}||}Wn ty0Yqwq |d| }td|d|dd|S)Nrz __END_OF_A_MESSAGE_47582648__ Tr3[!OP:]: r4)r7sysstdinreadr"r,)rr$contentsZ end_sequenceZlen_endZ last_charscharrrrr's$     z#ProgrammaticChatIO.prompt_for_inputr$cCstd|dddddS)NrcrdrTr(r+r-rrrr.sz$ProgrammaticChatIO.prompt_for_outputcCr/r0r5r8rrrr;r<z ProgrammaticChatIO.stream_outputr1cCr=rr+r>rrrr?r@zProgrammaticChatIO.print_outputN)rArBrCrEr'r.r;r?rrrrrbs  rbcCs|jr't|jd|jkrtd|jd|jd|jtjd<|jtjd<|jr3t|j |j d}nd}|j rMt |j |jd}|jd krLtd d |_nd}|jd krZt|j}n|jd krgt|j|j}n|jd krpt}ntd|jzY|jrt|jd}|dd|_t|j|j|j|jt|j|j |j!|j"|j|j#|j$|j%|t&|j'p|j|j(|j)|j*dt+|j,p|j|j-|j.d|||j/|j0|j1|j2 dWdSt3ytdYdSw)N,zLarger --num-gpus (z) than --gpus !ZCUDA_VISIBLE_DEVICESZXPU_VISIBLE_DEVICES) max_seq_lenZ gpu_split)rlZ data_typeZcpuz@xFasterTransformer now is only support CPUs. Reset device to CPUsimplerich programmaticzInvalid style for console: ri N)ckptwbits groupsizeZ act_order)rqrrrs)Z gptq_configZ awq_configexllama_config xft_configrevisionjudge_sent_enddebugrKzexit...)4Zgpusr7r6Znum_gpus ValueErrorosenvironZenable_exllamar Zexllama_max_seq_lenZexllama_gpu_splitZ enable_xftr Zxft_max_seq_lenZ xft_dtypeZdevicer,ZstylerrrFrJrbZupload_file_pathopenrgZconv_system_msgrZ model_pathZmax_gpu_memoryrZdtypeZ load_8bitZcpu_offloadingZ conv_templateZ temperatureZrepetition_penaltyZmax_new_tokensrZ gptq_ckptZ gptq_wbitsZgptq_groupsizeZgptq_act_orderr Zawq_ckptZ awq_wbitsZ awq_groupsizervrwrxZ no_historyKeyboardInterrupt)argsrtruZchatiorrrrmains          r__main__z--conv-templatezConversation prompt template.)typedefaulthelpz--conv-system-msgzConversation system message.z --temperaturegffffff?)rrz--repetition_penaltyg?z--max-new-tokensiz --no-historyZ store_true)actionz--stylerm)rmrnrozDisplay style.)rrZchoicesrz --multilinez2Enable multiline input. Use ESC+Enter for newline.)rrz--mousez:[Rich Style]: Enable mouse support for cursor positioning.z--judge-sent-endzWWhether enable the correction logic that interrupts the output of sentences due to EOS.z--debugz.Print useful debug information (e.g., prompts)z--upload-file-pathrzupload long txt for summary.)2__doc__ZargparserzrPreZprompt_toolkitrZprompt_toolkit.auto_suggestrZprompt_toolkit.completionrZprompt_toolkit.historyrZprompt_toolkit.key_bindingrZ rich.consolerZ rich.liverZ rich.markdownr ZtorchZfastchat.model.model_adapterr Zfastchat.modules.awqr Zfastchat.modules.exllamar Z#fastchat.modules.xfastertransformerr Zfastchat.modules.gptqrZfastchat.serve.inferencerrZfastchat.utilsrrrFrbrrAZArgumentParserZparserZ add_argumentrEfloatintZ parse_argsr~rrrrs              &K( I