device-side assert triggered
LXYTSOS opened this issue · comments
I got RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorCopy.c:70 when calculate the loss(loss = loss_fn(output, target)), something wrong with my label?
I printed the target out and found out that before the error shows, the shape of the target tensor is 2x9, but the batch size is 10, so the target tensor's shape is supposed to be 10x9. And the number of my train/valid samples are not multiple of 10, is that why I got this error?
@LXYTSOS I think so, since the config file in the json format specify the batch size to be 10, your target shape should always be 10x9, where 9 is the grid size, i.e. 3x3=9 patches within the grid.
I got another question, some of the ndpi files are labeled more than one class, not just tumor region, but some normal tissue labeled with muscle, normal, nec, etc. So when retrieve the label, will it think it is positive because it's in the normal polygons?
I have a xml annotation looked like this:<Annotations> <Annotation Name="Muscle" Type="Polygon" PartOfGroup="None" Color="#F4FA58"> here are coords </Annotation> <Annotation Name="Cancer" Type="Polygon" PartOfGroup="None" Color="#F4FA58"> here are coords </Annotation> <Annotation Name="Nec" Type="Polygon" PartOfGroup="None" Color="#F4FA58"> here are coords </Annotation> </Annotations>
only the annotation named 'Cancer' is the positive zone, the rest annotations and the rest of the whole image file are negative. And when convert the xml to json, I put all the cancer annotations into the positive block and the others in negative block. Say I got a patch image that is cut from the 'Nec'(annotation's name) region, so what's the result of if self._annotations[pid].inside_polygons((x, y), True):label = 1else: label = 0(image_producer.py), it's 1 or 0?
@LXYTSOS oh, you mean your own dataset outside of Camelyon16 ? If I understand incorrectly and you are using Camelyon16 data, can you point me to the specific one, e.g. Tumor_XXX.tif?
yeah you are right~, I'm use my own dataset outside of Camelyon16. Some of my ndpi files were labeled with just tumor region, and some were labeled other normal tissues besides the tumor parts.
It seems that my model predicts every part as tumor
@LXYTSOS I'm not quite familiar with your own datatype, e.g. what is ndpi file. But basically, the annotation format of Camelyon16 are polygons composed of coordinates at level 0 for each tumor regions. You can also find out more implementation details at annotation.py. Does your annotation format following the same principles?
yeah, my annotation format following the same principles, ndpi is a kind of image file format, just like the tif file you used here, but one annotation file might contains both positive and negative regions.
@LXYTSOS I think for the purpose of tumor detection, you only need the annotation of tumor(positiver) regions, and can assume all other place are background/normal(negative) regions.
Thanks for your very detailed reply, I'll try that.