Limit for max_new_tokens
#21
by
hanshupe
- opened
I got it running with max_new_tokens=1024 but get the error below when I increase it. Is 1024 the maximum?
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.