Question about LPIPS and SSIM

[theodoruszq]

For LPIPS

When evaluating LPIPS metric, the model inference result will be different for each instance for there existing random init weights.

I don't know it is the precision problem or not, keep the input same and torch.load the same weight again and run inference, the result value is different.

Here is my screencaption of two different instances

The two values should be the same, but they are not. After set manual seed to 0, they are amost identical.

For SSIM

the code ssim_all[v] += ssim(cur_gt_frame, cur_comp_frame, multichannel=True) may be better to change as ssim_all[v] += ssim(cur_gt_frame, cur_comp_frame, multichannel=True, data_range=1), because in the skimage library, it considers np.float32 as (-1, 1) range, but actually it is (0, 1) in our situation.

Of course, the minor problems do not affect the conclusions in the paper~

Thank you~

[rszeto]

Hi @sydney0zq,

Sorry about the terribly late response. The LPIPS issue has been fixed in commit de25552. I will try to investigate the SSIM part as well, though it might lie outside the scope of maintenance if it does not affect results.