Question about performance

[xixy]

Thank you for releasing the source code for this paper.

I run the code with the following commands, which I think is correct:

python train.py --gpu 1 --mode DMCNN
python train.py --gpu 1 --mode HMEAE

The performance is not as high as the paper reported, I get the following performance for DMCNN:
test best Precision: 0.5258620689655172 test best Recall:0.48348745046235136 test best F1:0.5037852718513421

And performance of HMEAE(CNN) is even worse.

I guess maybe it's because of the random data split.
other_files = [file for dir in self.dirs for file in self.source_files[dir] if dir!='nw']+nw[40:] random.shuffle(other_files) random.shuffle(other_files) test_files = nw[:40] dev_files = other_files[:30] train_files = other_files[30:]

Could you please release or email me([e-mail]) the data split you used for experiment (I mean the three files: train.json/test.json/det.json). I have ACE data and license for it.

Thank you very much

[Ziqi Wang]

The experiment was done a year ago and I didn't record the split that time, so I don't know the specific split I used then.

But it is not hard to get a reasonable split, all you need to do is to be patient and split the data until DMCNN's performance is relative stable.

I just uploaded one feasible split and my training logs, you should get similar results using such split.

[Rongzhi Zhang]

The experiment was done a year ago and I didn't record the split that time, so I don't know the specific split I used then.

But it is not hard to get a reasonable split, all you need to do is to be patient and split the data until DMCNN's performance is relative stable.

I just uploaded one feasible split and my training logs, you should get similar results using such split.

Thank you for the information. The proposed hierarchy module in this paper is very interesting. I also implemented your released code.

The following performance is averaged by 5 times:
DMCNN: 0.5155
HMEAE(CNN): 0.5019

I also wonder the data split and I am a little confused about "split the data until DMCNN's performance is relatively stable" in your reply.
Could you kindly provide more specific information to achieve the results as reported?

Thanks.

[Xiaozhi Wang]

We have provided a feasible split in https://github.com/thunlp/HMEAE/blob/master/logs/split.json. You can try it.
Another issue is that the original ACE 2005 files have a few obvious errors such as wrong span boundaries, different offsets for the same words, which cannot be directly handled by the script. As we cannot directly provide the cleaned files due to the license limit, we recommend to manually check the processed files while the number of errors is small.