DigitalExpression 'No space left on device'
LiuYin3 opened this issue · comments
Hi,
I'm using DigitalExpression to generate DGE matrix, but I always get an error saying no space left on device.
My command is: DigitalExpression I=Puck_200115_08.bam O=Puck_200115_08_2.dge.txt.gz SUMMARY=dge_summary.txt MIN_NUM_TRANSCRIPTS_PER_CELL=10 TMP_DIR=/home/users/temp/dir
INFO 2021-04-17 23:07:50 BarcodeListRetrieval Looking for cell barcodes that have at least 10 transcripts
INFO 2021-04-17 23:08:24 BamTagHistogram Processed 10,000,000 records. Elapsed time: 00:00:33s. Time for last 10,000,000: 33s. Last read position: 14:70,583,444
INFO 2021-04-17 23:09:00 BamTagHistogram Processed 20,000,000 records. Elapsed time: 00:01:10s. Time for last 10,000,000: 36s. Last read position: 1:24,613,336
INFO 2021-04-17 23:09:34 BamTagHistogram Processed 30,000,000 records. Elapsed time: 00:01:43s. Time for last 10,000,000: 33s. Last read position: 4:69,925,318
INFO 2021-04-17 23:10:11 BamTagHistogram Processed 40,000,000 records. Elapsed time: 00:02:20s. Time for last 10,000,000: 37s. Last read position: 8:83,571,855
INFO 2021-04-17 23:10:43 BamTagHistogram Processed 50,000,000 records. Elapsed time: 00:02:52s. Time for last 10,000,000: 32s. Last read position: /
INFO 2021-04-17 23:11:01 BamTagHistogram Processed 60,000,000 records. Elapsed time: 00:03:10s. Time for last 10,000,000: 17s. Last read position: /
INFO 2021-04-17 23:11:19 BamTagHistogram Processed 70,000,000 records. Elapsed time: 00:03:28s. Time for last 10,000,000: 18s. Last read position: /
INFO 2021-04-17 23:11:20 BarcodeListRetrieval Looking for cell barcodes that have at least 10 reads
INFO 2021-04-17 23:11:26 BarcodeListRetrieval Selected 154108 core barcodes
INFO 2021-04-17 23:11:50 UMIIterator Processed 1,000,000 records. Elapsed time: 00:00:23s. Time for last 1,000,000: 23s. Last read position: 10:128,089,145
INFO 2021-04-17 23:12:15 UMIIterator Processed 2,000,000 records. Elapsed time: 00:00:49s. Time for last 1,000,000: 25s. Last read position: 11:75,765,316
INFO 2021-04-17 23:12:42 UMIIterator Processed 3,000,000 records. Elapsed time: 00:01:15s. Time for last 1,000,000: 26s. Last read position: 12:17,284,198
INFO 2021-04-17 23:13:08 UMIIterator Processed 4,000,000 records. Elapsed time: 00:01:41s. Time for last 1,000,000: 26s. Last read position: 13:37,949,116
INFO 2021-04-17 23:13:34 UMIIterator Processed 5,000,000 records. Elapsed time: 00:02:07s. Time for last 1,000,000: 26s. Last read position: 14:66,361,557
INFO 2021-04-17 23:14:00 UMIIterator Processed 6,000,000 records. Elapsed time: 00:02:33s. Time for last 1,000,000: 25s. Last read position: 15:98,132,235
INFO 2021-04-17 23:14:22 UMIIterator Processed 7,000,000 records. Elapsed time: 00:02:56s. Time for last 1,000,000: 22s. Last read position: 17:20,965,827
INFO 2021-04-17 23:14:47 UMIIterator Processed 8,000,000 records. Elapsed time: 00:03:21s. Time for last 1,000,000: 25s. Last read position: 18:20,673,742
INFO 2021-04-17 23:15:11 UMIIterator Processed 9,000,000 records. Elapsed time: 00:03:44s. Time for last 1,000,000: 23s. Last read position: 19:5,802,155
INFO 2021-04-17 23:15:39 UMIIterator Processed 10,000,000 records. Elapsed time: 00:04:12s. Time for last 1,000,000: 28s. Last read position: 1:58,412,646
INFO 2021-04-17 23:16:01 UMIIterator Processed 11,000,000 records. Elapsed time: 00:04:34s. Time for last 1,000,000: 21s. Last read position: 1:182,314,437
INFO 2021-04-17 23:16:23 UMIIterator Processed 12,000,000 records. Elapsed time: 00:04:56s. Time for last 1,000,000: 22s. Last read position: 2:117,280,006
INFO 2021-04-17 23:16:45 UMIIterator Processed 13,000,000 records. Elapsed time: 00:05:18s. Time for last 1,000,000: 21s. Last read position: 2:181,147,820
INFO 2021-04-17 23:17:03 UMIIterator Processed 14,000,000 records. Elapsed time: 00:05:36s. Time for last 1,000,000: 18s. Last read position: 3:152,463,136
INFO 2021-04-17 23:17:24 UMIIterator Processed 15,000,000 records. Elapsed time: 00:05:58s. Time for last 1,000,000: 21s. Last read position: 4:139,103,789
INFO 2021-04-17 23:17:46 UMIIterator Processed 16,000,000 records. Elapsed time: 00:06:19s. Time for last 1,000,000: 21s. Last read position: 5:113,793,822
INFO 2021-04-17 23:18:05 UMIIterator Processed 17,000,000 records. Elapsed time: 00:06:39s. Time for last 1,000,000: 19s. Last read position: 6:71,581,275
INFO 2021-04-17 23:18:29 UMIIterator Processed 18,000,000 records. Elapsed time: 00:07:03s. Time for last 1,000,000: 24s. Last read position: 7:30,186,942
[Sat Apr 17 23:18:32 PDT 2021] org.broadinstitute.dropseqrna.barnyard.DigitalExpression done. Elapsed time: 10.69 minutes.
Runtime.totalMemory()=3504865280
Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: Problem writing temporary file file:///home/liuyin/temp/dir/sortingcollection.1331153896778371679.tmp. Try setting TMP_DIR to a file system with lots of space.
at htsjdk.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:260)
at htsjdk.samtools.util.SortingCollection.add(SortingCollection.java:182)
at org.broadinstitute.dropseqrna.utils.SortingIteratorFactory.create(SortingIteratorFactory.java:67)
at org.broadinstitute.dropseqrna.utils.readiterators.SamRecordSortingIteratorFactory.create(SamRecordSortingIteratorFactory.java:57)
at org.broadinstitute.dropseqrna.utils.readiterators.UMIIterator.(UMIIterator.java:145)
at org.broadinstitute.dropseqrna.utils.readiterators.UMIIterator.(UMIIterator.java:75)
at org.broadinstitute.dropseqrna.barnyard.GatherMolecularBarcodeDistributionByGene.getNumTranscriptsPerCell(GatherMolecularBarcodeDistributionByGene.java:150)
at org.broadinstitute.dropseqrna.barnyard.BarcodeListRetrieval.getListCellBarcodesByTranscriptCount(BarcodeListRetrieval.java:160)
at org.broadinstitute.dropseqrna.barnyard.BarcodeListRetrieval.getCellBarcodes(BarcodeListRetrieval.java:85)
at org.broadinstitute.dropseqrna.barnyard.DigitalExpression.doWork(DigitalExpression.java:145)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:305)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:103)
at org.broadinstitute.dropseqrna.cmdline.DropSeqMain.main(DropSeqMain.java:42)
Caused by: htsjdk.samtools.util.RuntimeIOException: Write error; BinaryCodec in writemode; streamed file (filename not available)
at htsjdk.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:222)
at htsjdk.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:212)
at htsjdk.samtools.util.BinaryCodec.writeString(BinaryCodec.java:319)
at htsjdk.samtools.util.BinaryCodec.writeString(BinaryCodec.java:306)
at htsjdk.samtools.BinaryTagCodec.writeTag(BinaryTagCodec.java:193)
at htsjdk.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:197)
at htsjdk.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:40)
at htsjdk.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:254)
... 12 more
Suppressed: java.io.IOException: No space left on device
I'm doing this on a server, and I think I still have some space..
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 2.0M 358 2.0M 1% /dev
tmpfs 2.0M 1 2.0M 1% /dev/shm
tmpfs 2.0M 624 2.0M 1% /run
tmpfs 2.0M 17 2.0M 1% /sys/fs/cgroup
/dev/mapper/cl-root 25M 152K 25M 1% /
/dev/sda1 64K 328 64K 1% /boot
/dev/mapper/cl-home 3.5M 604K 2.9M 18% /home
tmpfs 2.0M 5 2.0M 1% /run/user/1037
tmpfs 2.0M 5 2.0M 1% /run/user/1027
tmpfs 2.0M 5 2.0M 1% /run/user/1022
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 8088600 0 8088600 0% /dev
tmpfs 8105756 0 8105756 0% /dev/shm
tmpfs 8105756 33456 8072300 1% /run
tmpfs 8105756 0 8105756 0% /sys/fs/cgroup
/dev/mapper/cl-root 52403200 25775324 26627876 50% /
/dev/sda1 999320 360396 570112 39% /boot
/dev/mapper/cl-home 147918292 146429428 1488864 99% /home
tmpfs 1621148 0 1621148 0% /run/user/1037
tmpfs 1621148 0 1621148 0% /run/user/1027
tmpfs 1621148 0 1621148 0% /run/user/1022
I don't know what went wrong..