upon every restart of finetune i see:
"train data seems to have changed. restarting shuffled epoch."
i looked up where it happens, added debugging line and it turned out that
train->shuffle_samples_hash returns zero.
so, it's either not being stored or not being read. so, finetuning always starts from scratch and can not be continued.
at least that's how it is for me :).
upon every restart of finetune i see:
"train data seems to have changed. restarting shuffled epoch."
i looked up where it happens, added debugging line and it turned out that
train->shuffle_samples_hash returns zero.
so, it's either not being stored or not being read. so, finetuning always starts from scratch and can not be continued.
at least that's how it is for me :).