Can't reproduce the CoQA ablation study

[sunan0519]

Hi there,

I've been attempting to reproduce the ablation study on CoQA dev set. I've tried:

1. perform experiment on GraphFlow model(2-His) and got 78.7% f1 score(78.3% in paper).
2. set "use_bert"=False in config file, the performance dropped significantly to around 70% (which is actually 76.0% in paper).
3. set "use_bert"=True and remove RGNN, the performance dropped slightly, which is also inconsistent with the result present in paper.

The question is that whether set "use_bert"=False means removing bert embeddings? If it is, is there anything else I should do to reproduce these results? Any advice would be appreciated.

Thanks,
Nan