NVIDIA / NeMo-Aligner

Scalable toolkit for efficient model alignment

NVIDIA/NeMo-Aligner Issues

Results do not reproduce between self-hosted and hosted rewards model.
Updated 25 days ago
Tutorial / Example for Single Node FP8 Inference?
Updated a month ago
Multiple training file support
Updated a month ago1
how to fine-tune Qwen1.5 models based on Nemo
Closed 2 months ago4
add packed dataset
Updated a month ago1
how to fine-tune models with multi-nodes
Closed 2 months ago2
`cfg` in RLHFdataset doesn't have `length_params`
Closed 2 months ago1
Some code related to `train_valid_test_num_samples` may be wrong / unused
Updated 2 months ago
Amend SPIN to be able to handle the cast of rollout_MBS < DP_size
Updated 3 months ago
Docker build failing. Also, is there a .nemo reward model file available?
Updated 3 months ago4
Can you please support context parallel?
Updated 3 months ago
SFT does not work `max_steps`
Updated 3 months ago2
SFT `val_check_interval` should accept float input
Updated 3 months ago
triton server CANNOT work with nemo under python3.10, make it impossible to train a ppo using reward/critic server
Closed 3 months ago4
DPO crashes with `micro_batch_size > 1`
Closed 3 months ago
SFT is broken with container 24.01.01
Updated 3 months ago1
cannot load reward model from SFT model because of missing keys
Updated 3 months ago8
`GPTSFTModel.generate()` crashes with PP>2
Closed 3 months ago
Can you support KTO?
Updated 3 months ago1
Better (and more consistent) handling of examples that exceed the maximum sequence length
Updated 4 months ago
SFT may crash if input data exceeds the context length
Updated 4 months ago1
Support converting HF reward models to .nemo
Updated 5 months ago1
random samplers keeps state
Updated 5 months ago
add support for multi epoch training
Closed 5 months ago5
PPOTrainer ignores limit_val_batches when running validation
Closed 5 months ago2
Add support for `drop_last=False`
Updated 6 months ago4
Changing `num_rollout_samples` modifies the validation set in PPO
Updated 6 months ago
Is there a limit on the largest PPO model size and max context length the package can support to scale out of box? Is it scalable when we add more GPU nodes to the training job? Thanks! Sent from my iPhoneOn Jan 17, 2024, at 11:12 AM, HeyyyyyyG ***@***.***> wrote:
Closed 6 months ago1
What's the largest PPO model size and context length that have been trained successfully with this library? Can you also share some performance metrics (i.e. GPU count, training time) if possible?
Closed 6 months ago
`max_epochs=null` is not supported
Updated 6 months ago1
Force rampup_batch_size=None in config
Updated 6 months ago3
Implement the KTO algorithm
Updated 6 months ago1
Padding impacts parallel_logits computation, affecting PPO logprobs
Updated 7 months ago1
Add the SFT tutorial
Updated 7 months ago7
The learning rate schedule is generally incorrect when `max_steps` is not set
Updated 7 months ago
Consider swapping parameters less often in DPO
Updated 7 months ago
More helpful error message when failing to connect to critic server
Updated 7 months ago
GPTSFTChatDataset loss_mask becomes all False when prompt length > max_seq_length
Updated 7 months ago
Create a Template class to standardize prompt styles
Updated 7 months ago1
Support overwriting model config nodes instead of merging them
Updated 7 months ago1
code cleanup proposal
Updated 7 months ago
Provide helper scripts / update DPOModelDataset to truncate seq length in DPO
Updated 7 months ago1
Unification of preference dataset format
Updated 7 months ago
Log probs in PPO do not account for custom `sampling_params`
Updated 7 months ago
README Paths are absolute
Updated 8 months ago
The shuffle=True argument in SFT is not used
Updated 8 months ago
Add support for iterating multiple times on the same batch of rollout data in PPO
Updated 8 months ago
PPO is missing `limit_val_batches`
Updated 8 months ago
Overriding `cfg.model.encoder_seq_length` and `cfg.model.max_position_embeddings` is potentially dangerous
Updated 8 months ago
More sensible default optimizer settings
Updated 8 months ago