h; " d dl Z d dlZd dlZd dlZd dlmZ d dlmZ d dlm Z m
Z
m Z d dl m
Z
d dlmZmZ d dlmZmZmZmZ d dlmZmZmZmZ d d lm Z ej: e Z ed
G d d
Z G d d Z! G d de Z"y) N)Future) dataclass)SEEK_ENDSEEK_SETBytesIO)Path)LockThread)DictListOptionalUnion)DEFAULT_IGNORE_PATTERNS
CommitInfoCommitOperationAddHfApi)filter_repo_objectsT)frozenc : e Zd ZU dZeed< eed< eed< eed< y)
_FileToUploadzWTemporary dataclass to store info about files to upload. Not meant to be used directly.
local_path path_in_repo
size_limit
last_modifiedN) __name__
__module__ __qualname____doc__r __annotations__strintfloat 4/fsx/qgallouedec/trackio/trackio/commit_scheduler.pyr r s aOr$ r c e Zd ZdZddddddddddd
dedeeef deeef d e e d
e e d e e d e e
d
e e de ee e ef de ee e ef de
de d ddfdZ d dZ
d dZd dZd dZdefdZde e fdZde e fdZy) CommitSchedulera
Scheduler to upload a local folder to the Hub at regular intervals (e.g. push to hub every 5 minutes).
The recommended way to use the scheduler is to use it as a context manager. This ensures that the scheduler is
properly stopped and the last commit is triggered when the script ends. The scheduler can also be stopped manually
with the `stop` method. Checkout the [upload guide](https://huggingface.co/docs/huggingface_hub/guides/upload#scheduled-uploads)
to learn more about how to use it.
Args:
repo_id (`str`):
The id of the repo to commit to.
folder_path (`str` or `Path`):
Path to the local folder to upload regularly.
every (`int` or `float`, *optional*):
The number of minutes between each commit. Defaults to 5 minutes.
path_in_repo (`str`, *optional*):
Relative path of the directory in the repo, for example: `"checkpoints/"`. Defaults to the root folder
of the repository.
repo_type (`str`, *optional*):
The type of the repo to commit to. Defaults to `model`.
revision (`str`, *optional*):
The revision of the repo to commit to. Defaults to `main`.
private (`bool`, *optional*):
Whether to make the repo private. If `None` (default), the repo will be public unless the organization's default is private. This value is ignored if the repo already exists.
token (`str`, *optional*):
The token to use to commit to the repo. Defaults to the token saved on the machine.
allow_patterns (`List[str]` or `str`, *optional*):
If provided, only files matching at least one pattern are uploaded.
ignore_patterns (`List[str]` or `str`, *optional*):
If provided, files matching any of the patterns are not uploaded.
squash_history (`bool`, *optional*):
Whether to squash the history of the repo after each commit. Defaults to `False`. Squashing commits is
useful to avoid degraded performances on the repo when it grows too large.
hf_api (`HfApi`, *optional*):
The [`HfApi`] client to use to commit to the Hub. Can be set with custom settings (user agent, token,...).
Example:
```py
>>> from pathlib import Path
>>> from huggingface_hub import CommitScheduler
# Scheduler uploads every 10 minutes
>>> csv_path = Path("watched_folder/data.csv")
>>> CommitScheduler(repo_id="test_scheduler", repo_type="dataset", folder_path=csv_path.parent, every=10)
>>> with csv_path.open("a") as f:
... f.write("first line")
# Some time later (...)
>>> with csv_path.open("a") as f:
... f.write("second line")
```
Example using a context manager:
```py
>>> from pathlib import Path
>>> from huggingface_hub import CommitScheduler
>>> with CommitScheduler(repo_id="test_scheduler", repo_type="dataset", folder_path="watched_folder", every=10) as scheduler:
... csv_path = Path("watched_folder/data.csv")
... with csv_path.open("a") as f:
... f.write("first line")
... (...)
... with csv_path.open("a") as f:
... f.write("second line")
# Scheduler is now stopped and last commit have been triggered
```
NF)
everyr repo_typerevisionprivatetokenallow_patternsignore_patternssquash_historyhf_apirepo_id folder_pathr) r r* r+ r, r- r. r/ r0 r1 r returnc | xs t | | _ t | j j | _ |xs d| _ | | _ |
g }
nt |
t r|
g}
|
t z | _ | j
j rt d| j
d | j
j dd | j j |||d }
|
j | _ || _ || _ || _ i | _ |dkD st d | d t+ | _ || _ | | _ t2 j5 d
| j
d | j d | j. d
t7 | j8 d | _ | j: j= t? j@ | jB d| _" y )N)r- z0'folder_path' must be a directory, not a file: ''.T)parentsexist_ok)r2 r, r* r9 r z)'every' must be a positive integer, not 'zScheduled job to push 'z' to 'z' every z minutes.)targetdaemonF)#r apir
expanduserresolver3 r r.
isinstancer r r/ is_file
ValueErrormkdir create_repor2 r* r+ r-
last_uploadedr lockr) r0 loggerinfor
_run_scheduler_scheduler_threadstartatexitregister _push_to_hub_CommitScheduler__stopped)selfr2 r3 r) r r* r+ r, r- r. r/ r0 r1 repo_urls r% __init__zCommitScheduler.__init__j s /U / ,779AAC ( .B , " O
- . /O .1H H # # % B4CSCSBTTVW
td ; 88''W D (
'' "
qy Hr RS SF
, %d&6&6%7vdll^8TXT^T^S__h i
"(t/B/B4!P $$&))* r$ c d| _ y)ziStop the scheduler.
A stopped scheduler cannot be restarted. Mostly for tests purposes.
TN)rN rO s r% stopzCommitScheduler.stop s
r$ c | S Nr# rS s r% __enter__zCommitScheduler.__enter__ s r$ c ` | j j | j y rV )triggerresultrT )rO exc_type exc_value tracebacks r% __exit__zCommitScheduler.__exit__ s r$ c | j | _ t j | j dz | j
ryE)z7Dumb thread waiting between each scheduled push to Hub.< N)rY last_futuretimesleepr) rN rS s r% rH z CommitScheduler._run_scheduler s6 #||~D JJtzzB '~~ r$ c L | j j | j S )zTrigger a `push_to_hub` and return a future.
This method is automatically called every `every` minutes. You can also call it manually to trigger a commit
immediately, without waiting for the next scheduled commit.
)r<
run_as_futurerM rS s r% rY zCommitScheduler.trigger s xx%%d&7&788r$ c z | j ry t j d | j }| j rQt j d | j
j
| j | j | j |S # t $ r }t j d| d }~ww xY w)Nz((Background) scheduled commit triggered.z$(Background) squashing repo history.)r2 r* branchz Error while pushing to Hub: ) rN rF rG push_to_hubr0 r< super_squash_historyr2 r* r+ Exceptionerror)rO valuees r% rM z CommitScheduler._push_to_hub s >> >? $$&E"" BC-- LLDNN4== . L LL.qc2
s A.B B: B55B:c
| j 5 t j d t | j j d D ci c]< }|j
r*|j | j j |> }}| j r | j j d dnd}g }t |j | j | j D ] }|| }|j }| j j# | | j | |j$ k7 sP|j' t) |||z |j* |j$ ddd t- dk( rt j d yt j d
|D cg c]8 }t/ t1 |j2 |j4 |j : } }t j d
| j6 j9 | j: | j< | d| j> }
|D ]% } | j@ | j | j2 < ' |
S c c}w # 1 sw Y
xY wc c}w )a
Push folder to the Hub and return the commit info.
This method is not meant to be called directly. It is run in the background by the scheduler, respecting a
queue mechanism to avoid concurrent commits. Making a direct call to the method might lead to concurrency
issues.
The default behavior of `push_to_hub` is to assume an append-only folder. It lists all files in the folder and
uploads only changed files. If no changes are found, the method returns without committing anything. If you want
to change this behavior, you can inherit from [`CommitScheduler`] and override this method. This can be useful
for example to compress data together in a single file before committing. For more details and examples, check
out our [integration guide](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#scheduled-uploads).
z-Listing files to upload for scheduled commit.z**/*/r6 )r. r/ N)r r r r r z4Dropping schedule commit: no changed file to upload.z9Removing unchanged files since previous scheduled commit.)r )path_or_fileobjr z%Uploading files for scheduled commit.zScheduled Commit)r2 r*
operationscommit_messager+ )!rE rF debugsortedr3 globr@ relative_toas_posixr stripr keysr. r/ statrD getst_mtimeappendr st_sizelenr
PartialFileIOr r r<
create_commitr2 r* r+ r ) rO pathrelpath_to_abspathprefixfiles_to_uploadrelpathr rz file_to_uploadadd_operations commit_infofiles r% rh zCommitScheduler.push_to_hub s[ & YY LLH I
#$$))&1 " D <<> !1!12;;=tC
" <@;L;L))//45Q7RTF 46O."'') # 2 2 $ 4 4
0 8
!(&&**:6>))*5 F#**%'1)/')9'+||*.-- H 1 $ LLO P PQ #2
#2
-"--.:S:S! , 8 8
#2
<=hh,,LLnn%-]] -
$D262D2DD t / $}" YT
s+ ;? $$( ? ? 39% ?
S%Z ? sm
? C=? 3-? $ ? }? !tCy#~!67? "%S 3"78? ? ! ?
? B 9 9hz 2 &UXj 1 Ur$ r' c e Zd ZdZdeeef deddfdZd fd Z defdZ
defd Z d
ef fd Z defd Z
efd
ededefdZddee defdZ xZS )r a A file-like object that reads only the first part of a file.
Useful to upload a file to the Hub when the user might still be appending data to it. Only the first part of the
file is uploaded (i.e. the part that was available when the filesystem was first scanned).
In practice, only used internally by the CommitScheduler to regularly push a folder to the Hub with minimal
disturbance for the user. The object is passed to `CommitOperationAdd`.
Only supports `read`, `tell` and `seek` methods.
Args:
file_path (`str` or `Path`):
Path to the file to read.
size_limit (`int`):
The maximum number of bytes to read from the file. If the file is larger than this, only the first part
will be read (and uploaded).
file_pathr r4 Nc t | | _ | j j d | _ t |t j | j j j | _ y )Nrb)
r
_file_pathopen_fileminosfstatfilenor~ _size_limit)rO r r s r% rQ zPartialFileIO.__init__I sN y/__))$/
z288DJJ4E4E4G+H+P+PQr$ c T | j j t |
S rV )r closesuper__del__)rO __class__s r% r zPartialFileIO.__del__N s
w r$ c < d| j d| j dS )Nz)r r rS s r% __repr__zPartialFileIO.__repr__R s& ''8 TEUEUDVVW X
r$ c | j S rV )r rS s r% __len__zPartialFileIO.__len__W s r$ namec j |j d s|dv rt | | S t d| d )N_)readtellseekz PartialFileIO does not support 'r7 )
startswithr __getattribute__NotImplementedError)rO r r s r% r z PartialFileIO.__getattribute__Z sB ??3 4 ,
$
7+D1 1!$DTF""MNNr$ c 6 | j j S )z!Return the current file position.)r r rS s r% r zPartialFileIO.tellc s zz r$ _PartialFileIO__offset_PartialFileIO__whencec |t k( rt | |z }t }| j j || }|| j
kD r%| j j | j
S |S )zChange the stream position to the given offset.
Behavior is the same as a regular file, except that the position is capped to the size limit.
)r r r r r r )rO r r poss r% r zPartialFileIO.seekg s_
x 4y8+H Hjjooh1 !! !::??4#3#34 4
r$ _PartialFileIO__sizec | j j }||dk r| j |z
}nt || j |z
}| j j | S )zRead at most `__size` bytes from the file.
Behavior is the same as a regular file, except that it is capped to the size limit.
r )r r r r r )rO r currenttruncated_sizes r% r zPartialFileIO.readv s[
**//# >VaZ ! - - 7N !)9)9G)C DNzz~..r$ r ))r r r r r r r r! rQ r r r r r r r r
bytesr
__classcell__)r s @r% r r 6 s $R%T "2 R R R
!
#
OS O!c ! 3;
S
C
s
/8C= /% /r$ r )#rK loggingr rb concurrent.futuresr dataclassesr ior r r pathlibr threadingr r
typingr r r
r huggingface_hub.hf_apir r r r huggingface_hub.utilsr getLoggerr rF r r' r r# r$ r% r s % ! * * " . . 6 8 $ $ P PfL/G L/r$