Spaces:
Running
Running
Commit
Β·
261046a
1
Parent(s):
a2c1456
Update README with new repositories (synthetic-data, deduplication, openai-oss)
Browse files- Add 3 new repositories to the script collections table
- Update featured scripts section with more diverse examples
- Include CPU-friendly deduplication example
- Add synthetic data generation with CoT reasoning example
README.md
CHANGED
@@ -40,6 +40,9 @@ hf jobs uv run --flavor l4x1 \
|
|
40 |
| [classification](https://huggingface.co/datasets/uv-scripts/classification) | Text classification with guaranteed valid outputs | β
|
|
41 |
| [dataset-creation](https://huggingface.co/datasets/uv-scripts/dataset-creation) | Create datasets from PDFs and files | β |
|
42 |
| [vllm](https://huggingface.co/datasets/uv-scripts/vllm) | High-performance inference with vLLM | β
|
|
|
|
|
|
|
|
43 |
|
44 |
## π― Why UV Scripts?
|
45 |
|
@@ -64,16 +67,27 @@ hf jobs uv run --flavor l4x1 \
|
|
64 |
your-images extracted-text
|
65 |
```
|
66 |
|
67 |
-
###
|
68 |
|
69 |
-
|
70 |
|
71 |
```bash
|
72 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
73 |
hf jobs uv run --flavor l4x1 \
|
74 |
-
https://huggingface.co/datasets/uv-scripts/
|
75 |
-
--
|
76 |
-
--
|
77 |
```
|
78 |
|
79 |
## π Getting Started with HF Jobs
|
|
|
40 |
| [classification](https://huggingface.co/datasets/uv-scripts/classification) | Text classification with guaranteed valid outputs | β
|
|
41 |
| [dataset-creation](https://huggingface.co/datasets/uv-scripts/dataset-creation) | Create datasets from PDFs and files | β |
|
42 |
| [vllm](https://huggingface.co/datasets/uv-scripts/vllm) | High-performance inference with vLLM | β
|
|
43 |
+
| [synthetic-data](https://huggingface.co/datasets/uv-scripts/synthetic-data) | Generate high-quality synthetic data with CoT reasoning | β
|
|
44 |
+
| [deduplication](https://huggingface.co/datasets/uv-scripts/deduplication) | Remove duplicates using semantic similarity | β |
|
45 |
+
| [openai-oss](https://huggingface.co/datasets/uv-scripts/openai-oss) | Generate responses with visible reasoning traces | β
|
|
46 |
|
47 |
## π― Why UV Scripts?
|
48 |
|
|
|
67 |
your-images extracted-text
|
68 |
```
|
69 |
|
70 |
+
### Deduplicate Datasets (CPU-Friendly!)
|
71 |
|
72 |
+
Remove duplicates using semantic similarity - no GPU needed:
|
73 |
|
74 |
```bash
|
75 |
+
# Fast semantic deduplication on CPU
|
76 |
+
uv run https://huggingface.co/datasets/uv-scripts/deduplication/raw/main/semantic-dedupe.py \
|
77 |
+
your-dataset text your-dataset-clean \
|
78 |
+
--method duplicates --threshold 0.9
|
79 |
+
```
|
80 |
+
|
81 |
+
### Generate Synthetic Training Data
|
82 |
+
|
83 |
+
Create high-quality synthetic data with chain-of-thought reasoning:
|
84 |
+
|
85 |
+
```bash
|
86 |
+
# Generate synthetic math problems with reasoning
|
87 |
hf jobs uv run --flavor l4x1 \
|
88 |
+
https://huggingface.co/datasets/uv-scripts/synthetic-data/raw/main/cot-self-instruct.py \
|
89 |
+
--seed-dataset math-examples --output-dataset synthetic-math \
|
90 |
+
--task-type reasoning --num-samples 1000
|
91 |
```
|
92 |
|
93 |
## π Getting Started with HF Jobs
|