reproduction of BAR

[toyot-li]

Hi @yucornetto Thanks for your great work.

I only have one machine with 8 GPUs and I tried to reproduced BAR-B with the official tokenizer and given config. My result is gFID=1.27.

Do you think the gap to the reported 1.13 gFID is due to the number of machines (the dafault recipe is two nodes)?

If yes, would you suggest how to tune the hyperparams (e.g. LR) or just set gradient accumulation=2, for a better reproduction?

Thanks in advance.

[yucornetto]

Hi, good to know the experiments ran well on your side.

On your questions:

(1) If the total batch size remains 2048, using single node should be fine

(2) Beyond training, I guess different runs may have slightly different optimal sampling hparams (cfg, temp) so you may want to tune it for your trained model as well

[toyot-li]

@yucornetto Thanks for your suggestions!

(1) Since there is only one machine, the batch size is halved, i.e. 1024, otherwise OOM occurs. Consequently, you would suggest changing gradient_accumulation_steps=2?

(2) From your experience, tuning these inference hparams (cfg, temp, trading off quality vs. diversity) is more important than training hparams, such as (LR , bsz) pair?

[yucornetto]

(1) In that case maybe you should use gradient_accumulation_steps or enable use_checkpoint to have an effective batch size 2048, if your aim is to reproduce the results

(2) To be fair, it's hard to say which part matters more than others

[toyot-li]

Thanks a lot! I would first try grad accumulation or grad checkpointing for reproduction.

In addition, for LR-bsz coupling tuning, would you suggest a linear relationship or sqrt and so on?

[toyot-li]

@yucornetto I borrow another machine for reproduction. After training with 2 nodes, I reach an even worse gFID 1.93.

[yucornetto]

Hi, thanks for the update, it could be hard to figure out what is wrong in your reproduction, and it is weird you got close results with 1 node but much worse results with 2 nodes. I attached our training loss curve here and hope it may facilitate you to debug.

train_log_csv.csv

[toyot-li]

Thanks for sharing the log. I think the training curve is similar to my runs.

I would fix a typo that the 1node bs128 training yields gFID 1.72. It's better than bs256 but not so far.

I would investigate more about the ckpt and eval procedure.