bf16 might be freezing your weight updates — Lacuna
lacuna.tiptreesystems.com
If you fine-tune a model with weights stored in bf16 and let torch.optim.AdamW write to them directly, you get interesting behaviour. bf16 can only represent…