Update README.md

README.md

@@ -2,4 +2,103 @@
 base_model:
 - Wan-AI/Wan2.1-T2V-1.3B
 pipeline_tag: text-to-video
----
+---
+
+
+# UltraVideo: High-Quality UHD 4K Video Dataset
+
+-----
+
+<p align="center">
+    🤓 <a href="https://xzc-zju.github.io/projects/UltraVideo/">Project</a> &nbsp&nbsp | 📑 <a href="https://arxiv.org/abs/2506.13691">Paper</a> &nbsp&nbsp | 🤗 <a href="https://huggingface.co/datasets/APRIL-AIGC/UltraVideo">Hugging Face (UltraVideo Dataset))</a>&nbsp&nbsp | 🤗 <a href="https://huggingface.co/datasets/APRIL-AIGC/UltraVideo-Long">Hugging Face (UltraVideo-Long Dataset))</a>&nbsp&nbsp | 🤗 <a href="https://huggingface.co/APRIL-AIGC/UltraWan">Hugging Face (UltraWan-1K/4K Weights)</a>&nbsp&nbsp
+<br>
+
+-----
+
+[**UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captions**](https://arxiv.org/abs/2506.13691)
+
+- 🎋 **Click below image to watch the 4K demo video.**
+- 🤓 **First open-sourced UHD-4K/8K video datasets with comprehensive structured (10 types) captions.**
+- 🤓 **Native 1K/4K videos generation by UltraWan.**
+
+[![](assets/ultravideo.png)](https://www.youtube.com/watch?v=KPh62pfSHLQ)
+
+## TODO
+- [x] Release UltraVideo-Short
+- [x] Release UltraVideo-Long for long video generation and understanding.
+- [ ] Release structured caption by our PPL for [Open-Sora-Plan](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.1.0).
+
+
+## Quickstart
+
+1. Refer to [DiffSynth-Studio/examples/wanvideo](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/wanvideo) for environment preparation.
+``` sh
+pip install diffsynth==1.1.7
+```
+2. Download [Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) model using huggingface-cli:
+``` sh
+pip install "huggingface_hub[cli]"
+huggingface-cli download --repo-type model Wan-AI/Wan2.1-T2V-1.3B --local-dir ultrawan_weights/Wan2.1-T2V-1.3B --resume-download
+```
+3. Download [UltraWan-1K/4K](https://huggingface.co/APRIL-AIGC/UltraWan) models using huggingface-cli:
+``` sh
+huggingface-cli download --repo-type model APRIL-AIGC/UltraWan --local-dir ultrawan_weights/UltraWan --resume-download
+```
+4. Generate native 1K/4K videos.
+``` sh
+==> one GPU
+LoRA_1k: CUDA_VISIBLE_DEVICES=0 python infer.py --model_dir ultrawan_weights/Wan2.1-T2V-1.3B --model_path ultrawan_weights/UltraWan/ultrawan-1k.ckpt --mode lora --lora_alpha 0.25 --usp 0 --height 1088 --width 1920 --num_frames 81 --out_dir output/ultrawan-1k
+LoRA_4k: CUDA_VISIBLE_DEVICES=0 python infer.py --model_dir ultrawan_weights/Wan2.1-T2V-1.3B --model_path ultrawan_weights/UltraWan/ultrawan-4k.ckpt --mode lora --lora_alpha 0.5 --usp 0 --height 2160 --width 3840 --num_frames 33 --out_dir output/ultrawan-4k
+```
+``` sh
+==> usp with 6 GPUs
+LoRA_1k: CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 torchrun --standalone --nproc_per_node=6 infer.py --model_dir ultrawan_weights/Wan2.1-T2V-1.3B --model_path ultrawan_weights/UltraWan/ultrawan-1k.ckpt --mode lora --lora_alpha 0.25 --usp 1 --height 1088 --width 1920 --num_frames 81 --out_dir output/ultrawan-1k
+LoRA_4k: CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 torchrun --standalone --nproc_per_node=6 infer.py --model_dir ultrawan_weights/Wan2.1-T2V-1.3B --model_path ultrawan_weights/UltraWan/ultrawan-4k.ckpt --mode lora --lora_alpha 0.5 --usp 1 --height 2160 --width 3840 --num_frames 33 --out_dir output/ultrawan-4k
+```
+5. Official Inference
+``` sh
+==> one GPU
+ori_1k: CUDA_VISIBLE_DEVICES=0 python infer.py --model_dir ultrawan_weights/Wan2.1-T2V-1.3B --mode full --usp 0 --height 1088 --width 1920 --num_frames 81 --out_dir output/ori-1k
+
+==> usp with 6 GPUs
+ori_1k: CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 torchrun --standalone --nproc_per_node=6 infer.py --model_dir ultrawan_weights/Wan2.1-T2V-1.3B --mode full --usp 0 --height 1088 --width 1920 --num_frames 81 --out_dir output/ori-1k
+```
+
+## UltraVideo Dataset
+1. Download [UltraVideo](https://huggingface.co/datasets/APRIL-AIGC/UltraVideo) dataset.
+``` sh
+huggingface-cli download --repo-type dataset APRIL-AIGC/UltraVideo --local-dir ./UltraVideo --resume-download
+```
+2. Users must follow [LICENSE_APRIL_LAB](https://github.com/xzc-zju/UltraVideo/blob/main/license-april-lab.txt) to use this dataset.
+
+<p align="center">
+    <img src="assets/dataset_comparison.png" width="600"/>
+<p>
+
+<p align="center">
+    <img src="assets/statistic.png" width="600"/>
+<p>
+
+## VBench-Style Prompts of UltraVideo
+The used VBench-style prompts in UltraVideo in the paper for reference:`assets/ultravideo_prompts_in_VBench_style.json`
+
+## License Agreement
+1. Users must follow [LICENSE_APRIL_LAB](https://github.com/xzc-zju/UltraVideo/license-april-lab.txt) to use UltraVideo dataset.
+2. Users must follow [Wan-Video/Wan2.1/LICENSE.txt](https://github.com/Wan-Video/Wan2.1/blob/main/LICENSE.txt) to use Wan-related models. 
+
+
+## Acknowledgements
+We would like to thank the contributors to the [Wan2.1](https://github.com/Wan-Video/Wan2.1), [Qwen](https://huggingface.co/Qwen), [umt5-xxl](https://huggingface.co/google/umt5-xxl), [diffusers](https://github.com/huggingface/diffusers) and [HuggingFace](https://huggingface.co) repositories, for their open researches.
+
+## Citation
+
+If you find our work helpful, please cite us.
+
+```
+@article{ultravideo,
+  title={UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captions},
+  author={Xue, Zhucun and Zhang, Jiangning and Hu, Teng and He, Haoyang and Chen, Yinan and Cai, Yuxuan and Wang, Yabiao and Wang, Chengjie and Liu, Yong and Li, Xiangtai and Tao, Dacheng}, 
+  journal={arXiv preprint arXiv:2506.13691},
+  year={2025}
+}
+```