o f@sdddlmZmZmZmZddlZddlmZddlmZddl m Z ddl m Z Gddde Z dS) )ListOptionalTupleUnionN)nn)Cache)CausalLMOutputWithPast)LlamaForCausalLMcseZdZfddZ    ddejdeejded e fd d Z           ddejd eedeejdeejdee e e ejfdeejdeejdeedeedeedeedeejde eefffdd ZZS)LlamaskForCausalLMcs(t|td|j|_|dS)N)super__init__r Embedding hidden_sizespecial_tokens post_init)selfconfig __class__(/home/user/app/model/modeling_llamask.pyr s  zLlamaskForCausalLM.__init__N ? input_idsattention_mask max_tokens temperaturecCs<tj|jj|jd}t|D]}|j||d}|ddddddf|}tjjj |dd} tj | dd d} |j \} } }tj | | d| d|j|jd} || ddd| d| f<|dddddf| dd| d| f<d| dd| | f<| dddf} tj|| gdd}| }ttj| |kddr|Sq|S) N)device)rrlogits)dim) num_samples)dtyper)torchtensorr eos_token_idrrangeforwardr functionalsoftmax multinomialsqueezeshapezerosr$catallany)rrrrrZeos_token_tensor_outputsrprobsZ next_tokens batch_sizeseq_lenZ expanded_maskrrrgenerate$s&   (zLlamaskForCausalLM.generaternum_buffer_token position_idspast_key_values inputs_embedslabels use_cacheoutput_attentionsoutput_hidden_states return_dictcache_positionreturnc  sv|jd} |j|}|d}||j}||dkd}||dktd}tj d||||||| | | | d }|S)Nrr"geg) rrr:r;r<r=r>r?r@rArB) r.modelZ embed_tokens unsqueezetor$ masked_fillfloatr r))rrr9rr:r;r<r=r>r?r@rArBr6r4rrrr)Gs(     zLlamaskForCausalLM.forward)NNrr) NrNNNNNNNNNN)__name__ __module__ __qualname__r r% LongTensorrTensorintrHr8rrr FloatTensorboolrrr) __classcell__rrrrr sl  %      r )typingrrrrr%rZtransformers.cache_utilsrZtransformers.modeling_outputsrZtransformers.models.llamar r rrrrs