您好，请问我运行python3 test_SemanticKITTI.py的时候，内存一直标高，疑似内存泄露，该怎么办?
ksama777 opened this issue · comments
您好，我运行python3 train_SemanticKITTI.py的时候是没问题的，但是到test这一步 就疑似内存泄露，卡死了，怎么办?
Hello, There is no problem when I run python3 train_ SemanticKITTI.py . but when it comes to test, it is suspected that memory leaks and gets stuck. What should we do?
您好!
在test_SemanticKitti.py 中, 您可以先尝试标注掉
tqdm.__init__ = partialmethod(tqdm.__init__, disable=True)
以启动tqdm来监测进度, 如果进度确实没有变化的话, 首先可以尝试减小batch_size 防止OOM,
另外, 该段代码中也包含了原版的voting逻辑, 本身的test时间会较长, 而且会将整个validation set全部缓存, 如果确认是内存泄漏, 可以尝试在
后进行切片来减小一次性读入的test数据样本, 并反复执行多次来回避内存泄漏的问题.
如果还有什么问题欢迎继续提问.
您好，感谢您的耐心回复！
情况大概是 我运行后 占的memory就一直在增加，是不是只是因为我memory不够用呀 。 如这两张图 ，memory一直在加，加满，然后就会卡死。
然后标注掉您说的tqdm以及batch_size我1,2,6,16都试过了，还是一样的情况。
请问您运行的配置是什么？ 以及运行这个代码需要的最低配置呢？
…------------------ 原始邮件 ------------------
发件人: "charlesChen02/RandLA-pytorch" ***@***.***>;
发送时间: 2022年9月21日(星期三) 下午5:06
***@***.***>;
***@***.******@***.***>;
主题: Re: [charlesChen02/RandLA-pytorch] 您好，请问我运行python3 test_SemanticKITTI.py的时候，内存一直标高，疑似内存泄露，该怎么办? (Issue #3)
您好!
在test_SemanticKitti.py 中, 您可以先尝试标注掉
tqdm.__init__ = partialmethod(tqdm.__init__, disable=True)
以启动tqdm来监测进度, 如果进度确实没有变化的话, 首先可以尝试减小batch_size 防止OOM,
另外, 该段代码中也包含了原版的voting逻辑, 本身的test时间会较长, 而且会将整个validation set全部缓存, 如果确认是内存泄漏, 可以尝试在
https://github.com/charlesChen02/RandLA-pytorch/blob/7bae1d79895910acbb5165bbce8010a255c01351/dataset/semkitti_testset.py#L30
后进行切片来减小一次性读入的test数据样本, 并反复执行多次来回避内存泄漏的问题.
如果还有什么问题欢迎继续提问.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
我运行的配置是单张32G V100.
您可以先尝试在此处进行切片
如
self.data_list = sorted(self.data_list)[:500]
来检查一下是否能够正常运行?
切片果然有用了！
感谢大佬
请问python3 visualize_SemanticKITTI.py这里 后面是输入什么？
我直接python3 visualize_SemanticKITTI.py 好像不行
(randlanet) ***@***.***:/media/wx/HDD/DQ/RandLA-pytorch-main$ python3 visualize_SemanticKITTI.py
usage: ./visualize.py [-h] --dataset DATASET [--config CONFIG]
[--sequence SEQUENCE] [--predictions PREDICTIONS]
./visualize.py: error: the following arguments are required: --dataset/-d
…------------------ 原始邮件 ------------------
发件人: "charlesChen02/RandLA-pytorch" ***@***.***>;
发送时间: 2022年9月22日(星期四) 上午6:42
***@***.***>;
***@***.******@***.***>;
主题: Re: [charlesChen02/RandLA-pytorch] 您好，请问我运行python3 test_SemanticKITTI.py的时候，内存一直标高，疑似内存泄露，该怎么办? (Issue #3)
您的图片好像没有正常显示,
我运行的配置是单张32G V100.
您可以先尝试在此处进行切片
https://github.com/charlesChen02/RandLA-pytorch/blob/7bae1d79895910acbb5165bbce8010a255c01351/dataset/semkitti_testset.py#L30
如
self.data_list = sorted(self.data_list)[:500]
来检查一下是否能够正常运行?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
选项有
--dataset path to dataset for visualization
--config dataset config, default utils/semantic-kitti.yaml
--sequence sequence to visualize
--predictions location for predictions
e.g:
python3 evaluate_SemanticKITTI.py --dataset dataset/sequences_0.06/
--predictions runs/supervised/predictions/ --sequences 08
感谢您的耐心指导， 但我这里好像有个问题，我predictions的文件夹里面 有.npy和.label文件 是不是只能放.label的？ 还是说要把它们都放在sequences那个文件夹里面对应的11 12 13 的文件夹里？ 我这里格式应该没问题吧？
(randlanet) ***@***.***:/media/wx/HDD/DQ/RandLA-pytorch-main$ python3 evaluate_SemanticKITTI.py --dataset /media/wx/HDD/DQ/semantic_kitti/dataset/sequences_0.06/ --sequences 11 --predictions /media/wx/HDD/DQ/RandLA-pytorch-main/result/11/predictions/
********************************************************************************
INTERFACE:
Eval What: all
Data: /media/wx/HDD/DQ/semantic_kitti/dataset/sequences_0.06/
Predictions: /media/wx/HDD/DQ/RandLA-pytorch-main/result/11/predictions/
Sequences: ['11']
Config: utils/semantic-kitti.yaml
Limit: None
********************************************************************************
Opening data config file utils/semantic-kitti.yaml
Ignoring xentropy class 0 in IoU evaluation
[IOU EVAL] IGNORE: [0]
[IOU EVAL] INCLUDE: [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
0
0 0
Evaluating sequences
0it [00:00, ?it/s]
Validation set:
Acc avg 0.000
IoU avg 0.000
IoU class 1 [car] = 0.000
IoU class 2 [bicycle] = 0.000
IoU class 3 [motorcycle] = 0.000
IoU class 4 [truck] = 0.000
IoU class 5 [other-vehicle] = 0.000
IoU class 6 [person] = 0.000
IoU class 7 [bicyclist] = 0.000
IoU class 8 [motorcyclist] = 0.000
IoU class 9 [road] = 0.000
IoU class 10 [parking] = 0.000
IoU class 11 [sidewalk] = 0.000
IoU class 12 [other-ground] = 0.000
IoU class 13 [building] = 0.000
IoU class 14 [fence] = 0.000
IoU class 15 [vegetation] = 0.000
IoU class 16 [trunk] = 0.000
IoU class 17 [terrain] = 0.000
IoU class 18 [pole] = 0.000
IoU class 19 [traffic-sign] = 0.000
********************************************************************************
0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000
…------------------ 原始邮件 ------------------
发件人: "charlesChen02/RandLA-pytorch" ***@***.***>;
发送时间: 2022年9月24日(星期六) 上午7:04
***@***.***>;
***@***.******@***.***>;
主题: Re: [charlesChen02/RandLA-pytorch] 您好，请问我运行python3 test_SemanticKITTI.py的时候，内存一直标高，疑似内存泄露，该怎么办? (Issue #3)
选项有
--dataset path to dataset for visualization --config dataset config, default utils/semantic-kitti.yaml --sequence sequence to visualize --predictions location for predictions
e.g:
python3 evaluate_SemanticKITTI.py --dataset dataset/sequences_0.06/
--predictions runs/supervised/predictions/ --sequences 08
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
在data_prepare中, 会将原先的.label文件处理成统一的.npy文件, 同时在test SemanticKitti中, 如果没有配置 --index-to-label 的话, 生成的prediction会是.npy文件, 如果配置的话会转换成.label文件.
对于semanticKitti 数据集来说, sequence 11-21 是作为test使用的, 你所下载的本地文件是不包含对应的.label文件的, 如果你需要对这些sequence进行测试的话可以使用test kitti生成对应的.label文件后, 将生成的结果上传到semantickitti codalab中进行test才能得到对应的performance.
你可以使用sequence 00-10 其中的一个sequence 作为evaluate的对象, 建议是使用 seq 08, 因为00-07, 09-10 是作为training set使用的
您好，很抱歉再次打扰您， 请问我通过test kittii生成了对应的11-21的.label 并把它们放到测试网站上测试 结果网站说failed 报这个错 该怎么办？
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
File "/tmp/codalab/tmpdkODev/run/program/evaluate_semantics.py", line 171, in
assert(len(label_names) == len(pred_names))
AssertionError
还有我的上一封邮件您看了吗 ，用08 来evaluate的结果还是都是0
期待您的指导 万分感谢
…------------------ 原始邮件 ------------------
发件人: "charlesChen02/RandLA-pytorch" ***@***.***>;
发送时间: 2022年9月24日(星期六) 中午12:25
***@***.***>;
***@***.******@***.***>;
主题: Re: [charlesChen02/RandLA-pytorch] 您好，请问我运行python3 test_SemanticKITTI.py的时候，内存一直标高，疑似内存泄露，该怎么办? (Issue #3)
在data_prepare中, 会将原先的.label文件处理成统一的.npy文件, 同时在test SemanticKitti中, 如果没有配置 --index-to-label 的话, 生成的prediction会是.npy文件, 如果配置的话会转换成.label文件.
对于semanticKitti 数据集来说, sequence 11-21 是作为test使用的, 你所下载的本地文件是不包含对应的.label文件的, 如果你需要对这些sequence进行测试的话可以使用test kitti生成对应的.label文件后, 将生成的结果上传到semantickitti codalab中进行test才能得到对应的performance.
你可以使用sequence 00-10 其中的一个sequence 作为evaluate的对象, 建议是使用 seq 08, 因为00-07, 09-10 是作为training set使用的
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
抱歉回复晚了,
关于之前那封邮件的问题, 我认为应该是你指定的 result-dir路径有点问题, 这是我做test与eval对应的设定
python -u test_SemanticKITTI.py --checkpoint_path pre_train/checkpoint.tar --result_dir result/res
python -u evaluate_SemanticKITTI.py --predictions result/res
简单来说, 你只需要使 eval中prediction指定的目录与test中result_dir目录相同即可
如果您需要上传到网站上, 请记得在测试时使用--index-to-label 将prediction转换回.label文件,
网站上预测的逻辑是与repo中eval的逻辑类似的, 可以在本地代码中确认一下报错原因
您好 我照您说的 用了您给的eg目录 还是全是0 另外我在test的时候 加上了--index_to_label 之后才上传网站的，还是那个错
很头痛 另外我试了试用您给的预训练的checkpoint 也是全是0
有没有其他错误可能？
您回复了我就很感激了 我还怕打扰到您
感谢
python3 evaluate_SemanticKITTI.py --dataset /media/wx/HDD/DQ/semantic_kitti/dataset/sequences_0.06 --predictions result/res/08/predictions --sequence 08
********************************************************************************
INTERFACE:
Eval What: all
Data: /media/wx/HDD/DQ/semantic_kitti/dataset/sequences_0.06
Predictions: result/res/08/predictions
Sequences: ['08']
Config: /media/wx/HDD/DQ/RandLA-pytorch-main/utils/semantic-kitti.yaml
Limit: None
********************************************************************************
Opening data config file /media/wx/HDD/DQ/RandLA-pytorch-main/utils/semantic-kitti.yaml
Ignoring xentropy class 0 in IoU evaluation
[IOU EVAL] IGNORE: [0]
[IOU EVAL] INCLUDE: [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
0
0 0
Evaluating sequences
0it [00:00, ?it/s]
Validation set:
Acc avg 0.000
IoU avg 0.000
IoU class 1 [car] = 0.000
IoU class 2 [bicycle] = 0.000
IoU class 3 [motorcycle] = 0.000
IoU class 4 [truck] = 0.000
IoU class 5 [other-vehicle] = 0.000
IoU class 6 [person] = 0.000
IoU class 7 [bicyclist] = 0.000
IoU class 8 [motorcyclist] = 0.000
IoU class 9 [road] = 0.000
IoU class 10 [parking] = 0.000
IoU class 11 [sidewalk] = 0.000
IoU class 12 [other-ground] = 0.000
IoU class 13 [building] = 0.000
IoU class 14 [fence] = 0.000
IoU class 15 [vegetation] = 0.000
IoU class 16 [trunk] = 0.000
IoU class 17 [terrain] = 0.000
IoU class 18 [pole] = 0.000
IoU class 19 [traffic-sign] = 0.000
********************************************************************************
0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000
…------------------ 原始邮件 ------------------
发件人: "charlesChen02/RandLA-pytorch" ***@***.***>;
发送时间: 2022年9月30日(星期五) 下午4:31
***@***.***>;
***@***.******@***.***>;
主题: Re: [charlesChen02/RandLA-pytorch] 您好，请问我运行python3 test_SemanticKITTI.py的时候，内存一直标高，疑似内存泄露，该怎么办? (Issue #3)
抱歉回复晚了,
关于之前那封邮件的问题, 我认为应该是你指定的 result-dir路径有点问题, 这是我做test与eval对应的设定
python -u test_SemanticKITTI.py --checkpoint_path pre_train/checkpoint.tar --result_dir result/res python -u evaluate_SemanticKITTI.py --predictions result/res
简单来说, 你只需要使 eval中prediction指定的目录与test中result_dir目录相同即可
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
您好 我这次用08来evaluate 结果还是全是0 是我路径不对吗？ 我现在08/predictions 里面 都是.label的文件
(randlanet) ***@***.***:/media/wx/HDD/DQ/RandLA-pytorch-main$ python3 evaluate_SemanticKITTI.py --dataset /media/wx/HDD/DQ/semantic_kitti/dataset/sequences_0.06/ --sequences 08 --predictions /media/wx/HDD/DQ/RandLA-pytorch-main/result/08/predictions/
********************************************************************************
INTERFACE:
Eval What: all
Data: /media/wx/HDD/DQ/semantic_kitti/dataset/sequences_0.06/
Predictions: /media/wx/HDD/DQ/RandLA-pytorch-main/result/08/predictions/
Sequences: ['08']
Config: utils/semantic-kitti.yaml
Limit: None
********************************************************************************
Opening data config file utils/semantic-kitti.yaml
Ignoring xentropy class 0 in IoU evaluation
[IOU EVAL] IGNORE: [0]
[IOU EVAL] INCLUDE: [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
0
0 0
Evaluating sequences
0it [00:00, ?it/s]
Validation set:
Acc avg 0.000
IoU avg 0.000
IoU class 1 [car] = 0.000
IoU class 2 [bicycle] = 0.000
IoU class 3 [motorcycle] = 0.000
IoU class 4 [truck] = 0.000
IoU class 5 [other-vehicle] = 0.000
IoU class 6 [person] = 0.000
IoU class 7 [bicyclist] = 0.000
IoU class 8 [motorcyclist] = 0.000
IoU class 9 [road] = 0.000
IoU class 10 [parking] = 0.000
IoU class 11 [sidewalk] = 0.000
IoU class 12 [other-ground] = 0.000
IoU class 13 [building] = 0.000
IoU class 14 [fence] = 0.000
IoU class 15 [vegetation] = 0.000
IoU class 16 [trunk] = 0.000
IoU class 17 [terrain] = 0.000
IoU class 18 [pole] = 0.000
IoU class 19 [traffic-sign] = 0.000
********************************************************************************
0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000
…------------------ 原始邮件 ------------------
发件人: "charlesChen02/RandLA-pytorch" ***@***.***>;
发送时间: 2022年9月24日(星期六) 中午12:25
***@***.***>;
***@***.******@***.***>;
主题: Re: [charlesChen02/RandLA-pytorch] 您好，请问我运行python3 test_SemanticKITTI.py的时候，内存一直标高，疑似内存泄露，该怎么办? (Issue #3)
在data_prepare中, 会将原先的.label文件处理成统一的.npy文件, 同时在test SemanticKitti中, 如果没有配置 --index-to-label 的话, 生成的prediction会是.npy文件, 如果配置的话会转换成.label文件.
对于semanticKitti 数据集来说, sequence 11-21 是作为test使用的, 你所下载的本地文件是不包含对应的.label文件的, 如果你需要对这些sequence进行测试的话可以使用test kitti生成对应的.label文件后, 将生成的结果上传到semantickitti codalab中进行test才能得到对应的performance.
你可以使用sequence 00-10 其中的一个sequence 作为evaluate的对象, 建议是使用 seq 08, 因为00-07, 09-10 是作为training set使用的
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>