fh,H d Z ddlZddlmZ ddlmZmZmZmZm Z m
Z
ddl Z ddl
mZ ddlmZmZ ddlmZ dd lmZ erdd
lmZ ej0 e Zd d iZd
Z d\ Z Z d\ Z Z dZ! ed G d de Z"dgZ#y)z Tokenization classes for BLT. N)copyfile)
TYPE_CHECKINGAnyDictListOptionalTuple )import_protobuf)
AddedTokenPreTrainedTokenizer)logging)requires) TextInput
vocab_fileztokenizer.modelu ▁)z[INST]z[/INST])z<>
z
<>
a You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.)
sentencepiece)backendsc
X e Zd ZdZeZddgZ d deee e
f f fd
Z e d Z
d dZd Zd Ze d
Zd Zd d
dee f fd Zd Zd Zd Zd Zd dee dee fdZd dZ d dee deee de dee f fd
Z d dee deee dee fdZ xZ S ) BLTTokenizeruu
Construct a BLT tokenizer. Based on byte-level Byte-Pair-Encoding. The default padding token is unset as there is
no padding token in the original model.
Args:
vocab_file (`str`):
Path to the vocabulary file.
unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`):
The end of sequence token.
pad_token (`str` or `tokenizers.AddedToken`, *optional*):
A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by
attention mechanisms or loss computation.
sp_model_kwargs (`Dict[str, Any]`, `Optional`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
add_bos_token (`bool`, *optional*, defaults to `True`):
Whether or not to add an `bos_token` at the start of sequences.
add_eos_token (`bool`, *optional*, defaults to `False`):
Whether or not to add an `eos_token` at the end of sequences.
clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`):
Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like
extra spaces.
use_default_system_prompt (`bool`, *optional*, defaults to `False`):
Whether or not the default system prompt for BLT should be used.
spaces_between_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not to add spaces between special tokens.
legacy (`bool`, *optional*):
Whether or not the `legacy` behavior of the tokenizer should be used. Legacy is before the merge of #24622
and #25224 which includes fixes to properly handle tokens that appear after special tokens.
Make sure to also set `from_slow` to `True`.
A simple example:
- `legacy=True`:
```python
>>> from transformers import BLTTokenizerFast
>>> tokenizer = BLTTokenizerFast.from_pretrained("huggyblt/blt-7b", legacy=True, from_slow=True)
>>> tokenizer.encode("Hello .") # 869 is '▁.'
[1, 15043, 29871, 1, 869]
```
- `legacy=False`:
```python
>>> from transformers import BLTTokenizerFast
>>> tokenizer = BLTTokenizerFast.from_pretrained("huggyblt/blt-7b", legacy=False, from_slow=True)
>>> tokenizer.encode("Hello .") # 29889 is '.'
[1, 15043, 29871, 1, 29889]
```
Checkout the [pull request](https://github.com/huggingface/transformers/pull/24565) for more details.
add_prefix_space (`bool`, *optional*, defaults to `True`):
Whether or not to add an initial space to the input. This allows to treat the leading word just as any
other word. Again, this should be set with `from_slow=True` to make sure it's taken into account.
input_idsattention_masksp_model_kwargsc V |i n|| _ t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}| %t j d| j d d} | | _ || _ || _ || _
|
| _ | j |j dd | _ |
| _ t! | D d||||||| j | |
| | |
d | y ) NFT)
normalizedspecialz2You are using the default legacy behaviour of the a . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a blt tokenizer from a GGUF file you can ignore this message from_slow) bos_token eos_token unk_token pad_token
add_bos_token
add_eos_tokenr clean_up_tokenization_spacesuse_default_system_prompt spaces_between_special_tokenslegacyadd_prefix_space )r
isinstancestrr logger warning_once __class__r& r r! r" r$ get_spm_processorpopsp_modelr' super__init__)selfr r r r r r r! r" r# r$ r% r& r' kwargsr- s [/fsx/ita_zaporozhets/transformers/src/transformers/models/blt_modellike/tokenization_blt.pyr2 zBLTTokenizer.__init__ sF " &5% DT^^DT U/ /
F $ * *)B& ..vzz+u/MN
0
'' 0 0)E&?*G -
c p t | j j t | j S N)lenr0 encoder* r r3 s r5 unk_token_lengthz BLTTokenizer.unk_token_length s% 4==''DNN(;<==r6 c 4 t j di | j }| j s|r |j | j
|S t
| j
d 5 }|j }t d| j j d }|j j | }|j }d|_ |j j! | |j# }|j% | d d d |S # 1 sw Y |S xY w)NrbzThe new behaviour of z (with `self.legacy = False`)Fr( )spmSentencePieceProcessorr r& Loadr openreadr r- __name__
ModelProto
FromStringNormalizerSpecadd_dummy_prefixnormalizer_spec MergeFromSerializeToStringLoadFromSerializedProto)r3 r tokenizerfr0 model_pb2modelrI s r5 r. z BLTTokenizer.get_spm_processor s ..F1E1EF ;;) NN4?? +
$//4
(AvvxH'*?@W@W?XXu(vwI ((33H=E ' 6 6 8O/4O , ! ! + +O < ..0H - -h 7 ) ) s
!B"D
Dc ~ | j j }d |d< | j j |d< |S )Nr0 sp_model_proto)__dict__copyr0 serialized_model_proto)r3 states r5 __getstate__zBLTTokenizer.__getstate__ s;
""$ j"&--"F"F"H r6 c | j j | t j di | j | _ | j
j
| j y )Nr( )rS updater? r@ r r0 rL rR )r3 ds r5 __setstate__zBLTTokenizer.__setstate__ sG
Q 22JT5I5IJ
--d.A.ABr6 c 6 | j j S )zReturns vocab size)r0 get_piece_sizer; s r5
vocab_sizezBLTTokenizer.vocab_size s }}++--r6 c t | j D ci c] }| j | | }}|j | j |S c c}w )zReturns vocab as a dict)ranger^ convert_ids_to_tokensrY added_tokens_encoder)r3 ivocabs r5 get_vocabzBLTTokenizer.get_vocab sN ;@ ;QR;Qa++A.1;QR
T../ Ss Atextr returnc 2 | j st | dk( rt | |fi |S |j t
d }| j r t
|z }t | |fi |}t | dkD r"|d t
k( r|d | j v r|dd }|S )z
Converts a string to a list of tokens. If `self.legacy` is set to `False`, a prefix token is added unless the
first token is special.
r N)r& r9 r1 tokenizereplaceSPIECE_UNDERLINEr' all_special_tokens)r3 rf r4 tokensr- s r5 rk zBLTTokenizer.tokenize s
;;#d)q.7#D3F3 3|| ,c2 #d*D!$1&1 v;?vay,< <dNeNeAeABZF
r6 c 8 | j s|j t df s!| j j |t
S | j j | j |z t
}t | | j k\ r|| j d S |S )u(
Returns a tokenized string.
We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any
SPIECE_UNDERLINE. For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give
`['H', 'e', 'y']` instead of `['▁He', 'y']`. Thus we always encode `f"{unk_token}text"` and strip the
`unk_token`. Here is an example with `unk_token = ""` and `unk_token_length = 4`.
`self.tokenizer.sp_model.encode(" Hey", out_type = str)[4:]`.
ri )out_typeN) r&
startswithrm r0 r: r* r r9 r< )r3 rf r4 ro s r5 _tokenizezBLTTokenizer._tokenize s ;;doo/?.E F==''s'; ; %%dnnt&;c%J25f+AVAV2Vvd++-.b\bbr6 c 8 | j j | S )z0Converts a token (str) in an id using the vocab.)r0 piece_to_id)r3 tokens r5 _convert_token_to_idz!BLTTokenizer._convert_token_to_id
s }}((//r6 c < | j j | }|S )z=Converts an index (integer) in a token (str) using the vocab.)r0 IdToPiece)r3 indexrv s r5 _convert_id_to_tokenz!BLTTokenizer._convert_id_to_token s
''. r6 c |d j t r| j r |d dd |d< g }d}d}t | D ] \ }}|| j v r>|s|dk7 r| j
r|dz
}|| j j | |z z
}d}g }R|r+|dk( r&| j r|j t s|dz
}|j | d} || j j | z
}|S )z:Converts a sequence of tokens (string) in a single string.r rj N Fri T) rr rm r' enumeratern r& r0 decodeappend)r3 ro current_sub_tokens
out_stringprev_is_specialrc rv s r5 convert_tokens_to_stringz%BLTTokenizer.convert_tokens_to_string s !9 0 1d6K6K q !"
F1I
!&)HAu///&16dkk #%Jdmm 2 23E F NN
"&%'""qAv$2G2GPUP`P`aqPr #%J"))%0"' * dmm**+=>>
r6 filename_prefixc t j j | st j d| d yt j j ||r|dz ndt d z }t j j | j t j j | k7 rBt j j | j rt | j | |fS t j j | j sCt |d 5 }| j j }|j | ddd |fS |fS # 1 sw Y |fS xY w)a
Save the vocabulary and special tokens file to a directory.
Args:
save_directory (`str`):
The directory in which to save the vocabulary.
Returns:
`Tuple(str)`: Paths to the files saved.
zVocabulary path (z) should be a directoryN-r} r wb)ospathisdirr+ errorjoinVOCAB_FILES_NAMESabspathr isfiler rB r0 rU write)r3 save_directoryr out_vocab_fileficontent_spiece_models r5 save_vocabularyz BLTTokenizer.save_vocabulary/ s" ww}}^, LL,^,<zzNNTFFFFNT)Fr8 )NF)!rD
__module__ __qualname____doc__r vocab_files_namesmodel_input_namesr r r* r r2 propertyr< r. rW r[ r^ re r rk rs rw r{ r r r r intboolr r
__classcell__)r- s @r5 r r 7 s` FP *$&67
48%*"'&+ 8
"$sCx.18
t > >" C
. . [ tCy $c$0
2!x} !X]^aXb !6 sx#
9#
3;DI3F#
ko#
c#
L JN 9 3;DI3F
c r6 r )$r r shutilr typingr r r r r r r r? convert_slow_tokenizerr tokenization_utilsr r
utilsr utils.import_utilsr tokenization_utils_baser
get_loggerrD r+ r rm B_INSTE_INSTB_SYSE_SYSDEFAULT_SYSTEM_PROMPTr __all__r( r6 r5 r s * $ B B 5 A * 4 H %!#45 $, u^
%&a& a 'aH
r6