lucidrains / lion-pytorch

🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch

lucidrains/lion-pytorch Issues

Same amount of VRAM is taken as in AdamW
Updated 6 days ago6
Always getting NaNs in long training
Updated a month ago5
Using Triton with PyTorch 2.0 for AMP training results in tensors containing inf values.
Updated 3 months ago
This issue actually still persists. My python environment:
Closed 3 months ago
Loss explodes when resuming using trion implementation.
Closed a year ago3
Strange Results on first step
Closed 7 months ago11
AMD ROCM versions
Updated 9 months ago
Convergence guarantees for Lion
Updated 10 months ago
Do you have the actual weights trained from the paper?
Updated a year ago
add an 8-bit version with bitsandbytes
Updated a year ago5
Performance experiments over AdamW
Closed a year ago1
Instability when resuming trains
Closed a year ago22
Does the Lion optimizer work with grad accumulation?
Closed a year ago1
Learning rate scaling for distributed training?
Updated a year ago3
Strange Results on first step
Closed a year ago1
Add the implementation to official pytorch repo
Closed a year ago3
KeyError in update_fn_kernel when use_triton=True
Closed a year ago7
What is the best learning rate you have found? for lora and dreambooth ty
Closed a year ago
any new update?
Closed a year ago4
Adaptive learning-rate optimization
Closed a year ago2
AttributeError: module 'trinton.language' has no attribute 'constexpr'
Closed a year ago6
How does this optimizer perform in you tasks?
Closed a year ago2
Did you increase the decoupled weight decay simultaneously when decreasing the learning rate?
Closed a year ago4