Troubleshoot subprocess memory consumption
jjg opened this issue · comments
I first thought this was the issue that #83 was intended to resolve, however after analysis it turns out one of of the initialization subprocesses (storage_utilization_initializer) is consuming a lot of memory.
Found that the fs.readSync call used to determine the size of each physically stored block was the source of the memory consumption (not sure if it's truly leaking, but the outcome is the same).
Here's a log slice that demonstrates the problem:
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 11MB 0 SUI: processed inode 0143921d63fc5b5210f831e3e20c31c1ceb8c30e
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 60MB 0 SUI: processed inode 73feba8279c30cdf01910e561e07e0d05050c13c
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 97MB 0 SUI: processed inode ec4a1d91b2f122368e7aa5fe6b19878e64d265d1
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 135MB 0 SUI: processed inode 2289e04e827b2e24c2ce5aefd50a677c0f1724e2
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 172MB 0 SUI: processed inode a991106cd0db49dcee05fb88b085346b5134438c
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 209MB 0 SUI: processed inode 6a2ad5ee837093b357c79387e8b42deffca6c1df
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 246MB 0 SUI: processed inode 4b7bc1ad50dbd9c4c524fcfc47dc63a1b1a09376
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 267MB 0 SUI: processed inode 22789aaaa6f912e9a5d21cadfdd1632fe2aaad87
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 268MB 0 SUI: processed inode c1ac2079ff16d93a41bf77cd4c8caa0400216fac
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 268MB 0 SUI: processed inode ea3906312fe1387a51857a9c37bb6dcfee499e7f
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 268MB 0 SUI: processed inode 1b5f24a4c8de7c16a8ebf510e863532d486975dd
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 268MB 0 SUI: processed inode 76cf98e7f0c68c65247e1165161ccc433dd47a38
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 268MB 0 SUI: processed inode c4f22a815f741738f3122a2c5ca9673927fa9446
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 270MB 0 SUI: processed inode 147a0684310338ad65b1f857f86ec2801396511f
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 270MB 0 SUI: processed inode a6066d3fd4302650817b60a3f5b1eb5cf1a75497
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 270MB 0 SUI: processed inode 439e40b21b505ea0ee165e0deff0f20c69526a9e
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 270MB 0 SUI: processed inode cc46eeac134599a35923de5f637a393f1e58bcf6
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 270MB 0 SUI: processed inode 3694f3efc0851f626df2d60d8093d1a3f7070ea4
Wed Aug 19 2015 09:39:15 GMT-0500 (CDT) 13MB 0 storage_utilization_initializer exited
Switched to using fs.stat to get the stored block size, results are encouraging:
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 0143921d63fc5b5210f831e3e20c31c1ceb8c30e
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 UBI: processed inode 439e40b21b505ea0ee165e0deff0f20c69526a9e
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 UBI: processed inode cc46eeac134599a35923de5f637a393f1e58bcf6
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 73feba8279c30cdf01910e561e07e0d05050c13c
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 UBI: processed inode 3694f3efc0851f626df2d60d8093d1a3f7070ea4
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 13MB 1 unique_block_initializer: superblock processing complete
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 13MB 0 unique_block_initializer disconnected
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode ec4a1d91b2f122368e7aa5fe6b19878e64d265d1
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 2289e04e827b2e24c2ce5aefd50a677c0f1724e2
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode a991106cd0db49dcee05fb88b085346b5134438c
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 13MB 0 unique_block_initializer exited
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 6a2ad5ee837093b357c79387e8b42deffca6c1df
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 4b7bc1ad50dbd9c4c524fcfc47dc63a1b1a09376
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 22789aaaa6f912e9a5d21cadfdd1632fe2aaad87
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode c1ac2079ff16d93a41bf77cd4c8caa0400216fac
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode ea3906312fe1387a51857a9c37bb6dcfee499e7f
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 1b5f24a4c8de7c16a8ebf510e863532d486975dd
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 76cf98e7f0c68c65247e1165161ccc433dd47a38
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode c4f22a815f741738f3122a2c5ca9673927fa9446
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 147a0684310338ad65b1f857f86ec2801396511f
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode a6066d3fd4302650817b60a3f5b1eb5cf1a75497
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 439e40b21b505ea0ee165e0deff0f20c69526a9e
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode cc46eeac134599a35923de5f637a393f1e58bcf6
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 11MB 0 SUI: processed inode 3694f3efc0851f626df2d60d8093d1a3f7070ea4
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 13MB 1 0.6843090420588851 out of 100 gigabytes (1%) used on ./blocks/
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 13MB 0 storage_utilization_initializer disconnected
Wed Aug 19 2015 09:57:06 GMT-0500 (CDT) 13MB 0 storage_utilization_initializer exited
Testing to see if I can get memory use lower by removing the fs.existsSync call (seems unnecessary if we're calling fs.statSync immediately after).
Before:
Wed Aug 19 2015 11:28:30 GMT-0500 (CDT) 45MB 0 SUI: processed inode f8cfb39206cf8b9bcabdeaf67e5e8ecdb1f58eb1
Wed Aug 19 2015 11:28:30 GMT-0500 (CDT) 39MB 0 UBI: processed inode 4929fc6ec703986983d18a5df27e9d6c5afc5545
Wed Aug 19 2015 11:28:30 GMT-0500 (CDT) 39MB 0 UBI: processed inode aad8d757001d8309a51e8bf5811881b31c59ffce
Wed Aug 19 2015 11:28:30 GMT-0500 (CDT) 53MB 0 SUI: processed inode f5c95377dd1d71ba3adea5483c1ebc6b04f6ace6
Wed Aug 19 2015 11:28:30 GMT-0500 (CDT) 53MB 0 SUI: processed inode b106e9979fdab843eb33e2bc1d9fc0d6050049a9
Wed Aug 19 2015 11:28:30 GMT-0500 (CDT) 53MB 0 SUI: processed inode a66e2ee50ad440e381e6bbed6d22c91f9c8b1857
Wed Aug 19 2015 11:28:30 GMT-0500 (CDT) 53MB 0 SUI: processed inode 3e359a52207b63542ca93c4885fb62f18efabfd8
Wed Aug 19 2015 11:28:31 GMT-0500 (CDT) 55MB 0 SUI: processed inode 7d97c557380f3037b088ca0b49464193f6009059
Wed Aug 19 2015 11:28:31 GMT-0500 (CDT) 39MB 0 UBI: processed inode d670c6446a1662607f0772edae3028c7b43a711b
Wed Aug 19 2015 11:28:31 GMT-0500 (CDT) 55MB 0 SUI: processed inode e1ef3764ab1251986161a91bfbab89710fb49db8
Wed Aug 19 2015 11:28:31 GMT-0500 (CDT) 55MB 0 SUI: processed inode 6d2ef3e061e5351dbef1814ff117e1405a4dc92c
Wed Aug 19 2015 11:28:31 GMT-0500 (CDT) 89MB 1 45.67119275499135 out of 100 gigabytes (46%) used on ./blocks/
Wed Aug 19 2015 11:28:31 GMT-0500 (CDT) 89MB 0 storage_utilization_initializer disconnected
Wed Aug 19 2015 11:28:31 GMT-0500 (CDT) 89MB 0 storage_utilization_initializer exited
After:
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 38MB 0 SUI: processed inode 87d0f9da8a2d92f6023e8aa992718c3972633dc5
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 34MB 0 UBI: processed inode 7d7c01df7dbc527204d70d61b554463add7a84f3
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 38MB 0 SUI: processed inode f8cfb39206cf8b9bcabdeaf67e5e8ecdb1f58eb1
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 36MB 0 UBI: processed inode 76f6bda92a0b9d83f8b6c98efa4d875ae5b5f9c2
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 39MB 0 SUI: processed inode f5c95377dd1d71ba3adea5483c1ebc6b04f6ace6
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 39MB 0 SUI: processed inode b106e9979fdab843eb33e2bc1d9fc0d6050049a9
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 39MB 0 SUI: processed inode a66e2ee50ad440e381e6bbed6d22c91f9c8b1857
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 39MB 0 SUI: processed inode 3e359a52207b63542ca93c4885fb62f18efabfd8
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 37MB 0 UBI: processed inode 759d5fe880429e116f030be7361c0af3d6b1b11e
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 41MB 0 SUI: processed inode 7d97c557380f3037b088ca0b49464193f6009059
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 38MB 0 UBI: processed inode db0fbb6e343d53cccdd98747a8fa3a88770b4f36
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 45MB 0 SUI: processed inode e1ef3764ab1251986161a91bfbab89710fb49db8
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 46MB 0 SUI: processed inode 6d2ef3e061e5351dbef1814ff117e1405a4dc92c
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 88MB 1 45.67119275499135 out of 100 gigabytes (46%) used on ./blocks/
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 88MB 0 storage_utilization_initializer disconnected
Wed Aug 19 2015 11:31:39 GMT-0500 (CDT) 88MB 0 storage_utilization_initializer exited
After loading ~85GB of data it's clear that memory utilization by the initialize processes increases with pool size:
Wed Aug 19 2015 14:05:49 GMT-0500 (CDT) 11MB 1 JSFS starting up...
Wed Aug 19 2015 14:05:49 GMT-0500 (CDT) 42MB 1 superblock loaded from disk
Wed Aug 19 2015 14:05:49 GMT-0500 (CDT) 42MB 1 Starting unique_block_initializer
Wed Aug 19 2015 14:05:49 GMT-0500 (CDT) 111MB 1 Starting storage_utilization_initializer
Wed Aug 19 2015 14:05:49 GMT-0500 (CDT) 108MB 1 Ready to process requests
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 42MB 0 UBI: processed inode 0143921d63fc5b5210f831e3e20c31c1ceb8c30e
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 UBI: processed inode 73feba8279c30cdf01910e561e07e0d05050c13c
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 UBI: processed inode ec4a1d91b2f122368e7aa5fe6b19878e64d265d1
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 UBI: processed inode 2289e04e827b2e24c2ce5aefd50a677c0f1724e2
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 UBI: processed inode a991106cd0db49dcee05fb88b085346b5134438c
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 UBI: processed inode 6a2ad5ee837093b357c79387e8b42deffca6c1df
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 UBI: processed inode 4b7bc1ad50dbd9c4c524fcfc47dc63a1b1a09376
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 UBI: processed inode 22789aaaa6f912e9a5d21cadfdd1632fe2aaad87
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode c1ac2079ff16d93a41bf77cd4c8caa0400216fac
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode ea3906312fe1387a51857a9c37bb6dcfee499e7f
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode 1b5f24a4c8de7c16a8ebf510e863532d486975dd
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 42MB 0 SUI: processed inode 0143921d63fc5b5210f831e3e20c31c1ceb8c30e
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode 76cf98e7f0c68c65247e1165161ccc433dd47a38
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode c4f22a815f741738f3122a2c5ca9673927fa9446
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode 147a0684310338ad65b1f857f86ec2801396511f
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode a6066d3fd4302650817b60a3f5b1eb5cf1a75497
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode 439e40b21b505ea0ee165e0deff0f20c69526a9e
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode cc46eeac134599a35923de5f637a393f1e58bcf6
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode 3694f3efc0851f626df2d60d8093d1a3f7070ea4
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode 73feba8279c30cdf01910e561e07e0d05050c13c
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode ec4a1d91b2f122368e7aa5fe6b19878e64d265d1
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode 2289e04e827b2e24c2ce5aefd50a677c0f1724e2
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode a991106cd0db49dcee05fb88b085346b5134438c
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode 6a2ad5ee837093b357c79387e8b42deffca6c1df
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode 4b7bc1ad50dbd9c4c524fcfc47dc63a1b1a09376
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode 22789aaaa6f912e9a5d21cadfdd1632fe2aaad87
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode c1ac2079ff16d93a41bf77cd4c8caa0400216fac
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode ea3906312fe1387a51857a9c37bb6dcfee499e7f
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode 1b5f24a4c8de7c16a8ebf510e863532d486975dd
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode 76cf98e7f0c68c65247e1165161ccc433dd47a38
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode c4f22a815f741738f3122a2c5ca9673927fa9446
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode 147a0684310338ad65b1f857f86ec2801396511f
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode 9bc20916ad8d8b941ac43843263f8dbe903b072d
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode a6066d3fd4302650817b60a3f5b1eb5cf1a75497
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode 439e40b21b505ea0ee165e0deff0f20c69526a9e
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode cc46eeac134599a35923de5f637a393f1e58bcf6
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 45MB 0 SUI: processed inode 3694f3efc0851f626df2d60d8093d1a3f7070ea4
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 SUI: processed inode 9bc20916ad8d8b941ac43843263f8dbe903b072d
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 47MB 0 SUI: processed inode 1614862b45beba29401c868fe217c3d992f02818
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 46MB 0 UBI: processed inode 1614862b45beba29401c868fe217c3d992f02818
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 48MB 0 SUI: processed inode 7d7c01df7dbc527204d70d61b554463add7a84f3
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 50MB 0 SUI: processed inode 76f6bda92a0b9d83f8b6c98efa4d875ae5b5f9c2
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 51MB 0 SUI: processed inode 759d5fe880429e116f030be7361c0af3d6b1b11e
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 51MB 0 SUI: processed inode db0fbb6e343d53cccdd98747a8fa3a88770b4f36
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 56MB 0 SUI: processed inode 4929fc6ec703986983d18a5df27e9d6c5afc5545
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 48MB 0 UBI: processed inode 7d7c01df7dbc527204d70d61b554463add7a84f3
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 56MB 0 SUI: processed inode aad8d757001d8309a51e8bf5811881b31c59ffce
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 56MB 0 SUI: processed inode d670c6446a1662607f0772edae3028c7b43a711b
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 56MB 0 SUI: processed inode d1f72854dab0a4795547a0d29d08eaa87446cb2e
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 56MB 0 SUI: processed inode 87d0f9da8a2d92f6023e8aa992718c3972633dc5
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 50MB 0 UBI: processed inode 76f6bda92a0b9d83f8b6c98efa4d875ae5b5f9c2
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 64MB 0 SUI: processed inode f8cfb39206cf8b9bcabdeaf67e5e8ecdb1f58eb1
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 51MB 0 UBI: processed inode 759d5fe880429e116f030be7361c0af3d6b1b11e
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 51MB 0 UBI: processed inode db0fbb6e343d53cccdd98747a8fa3a88770b4f36
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 65MB 0 SUI: processed inode f5c95377dd1d71ba3adea5483c1ebc6b04f6ace6
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 65MB 0 SUI: processed inode b106e9979fdab843eb33e2bc1d9fc0d6050049a9
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 65MB 0 SUI: processed inode a66e2ee50ad440e381e6bbed6d22c91f9c8b1857
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 65MB 0 SUI: processed inode 3e359a52207b63542ca93c4885fb62f18efabfd8
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 65MB 0 SUI: processed inode 7d97c557380f3037b088ca0b49464193f6009059
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 53MB 0 UBI: processed inode 4929fc6ec703986983d18a5df27e9d6c5afc5545
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 65MB 0 SUI: processed inode e1ef3764ab1251986161a91bfbab89710fb49db8
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 65MB 0 SUI: processed inode 6d2ef3e061e5351dbef1814ff117e1405a4dc92c
Wed Aug 19 2015 14:05:50 GMT-0500 (CDT) 65MB 0 SUI: processed inode 3c6fa2ef4353602afa97a36af0900c8d4594bd43
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 59MB 0 UBI: processed inode aad8d757001d8309a51e8bf5811881b31c59ffce
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode 2c8d578398a56d5f9a9e21e5057d91e92bcf0c10
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode 0223d9c2af3775a46cf40c63f4583e5a09b5a3c1
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode ff3a610bfd1abbe2052d43a3103b825fae8fc56b
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode dfc2966fd802bdecda95fd536b851c2010dfc741
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 60MB 0 UBI: processed inode d670c6446a1662607f0772edae3028c7b43a711b
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode cfc88b8c5fa4ccb4a85fb23f3f37d02421829c52
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode 337cb086c4f5ab171fbfeb8245b43b0cc59459e6
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode f8f389de0423420304b49f98af3c625a42cacdcd
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode b6cc0e26186c7f06cd3b7784e362e376f2c9c318
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode 673e78ebe0cd59d18ebd14d439b31f25f3493125
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode 564f27556e220375ffab7e739e83757e9a69e741
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode 10bbb4cecd556e8ccf126d25e61da5000d5215ee
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode 3bf3a6170d0b59a539046674c83642292cb2dc09
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 60MB 0 UBI: processed inode d1f72854dab0a4795547a0d29d08eaa87446cb2e
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode 2ac61ebfde1ead452059dbf1a54d30a348a28e88
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode 9504a725b6575fb3b8d295bd4e7b17c572bd2d79
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode f7bad43e6a677f715082dac76decc15805bf02c0
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode c91350be20b3b73b2be80dfa64651d3500ff7d47
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode e934b10fbea1d7a42942fac130bf950061128c23
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode ac690d0a8b364b254a2d9a6d26d957360c2c3e28
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode e9f779026fb14e8f79e82724ffae57f29fc5a0eb
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 60MB 0 UBI: processed inode 87d0f9da8a2d92f6023e8aa992718c3972633dc5
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 65MB 0 SUI: processed inode 07409b8ed1ad865e93195ea5695fe6fae993e1c2
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 71MB 1 85.15804611053318 out of 100 gigabytes (85%) used on ./blocks/
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 71MB 0 storage_utilization_initializer disconnected
Wed Aug 19 2015 14:05:51 GMT-0500 (CDT) 71MB 0 storage_utilization_initializer exited
Wed Aug 19 2015 14:05:52 GMT-0500 (CDT) 60MB 0 UBI: processed inode f8cfb39206cf8b9bcabdeaf67e5e8ecdb1f58eb1
Wed Aug 19 2015 14:05:54 GMT-0500 (CDT) 60MB 0 UBI: processed inode f5c95377dd1d71ba3adea5483c1ebc6b04f6ace6
Wed Aug 19 2015 14:05:54 GMT-0500 (CDT) 60MB 0 UBI: processed inode b106e9979fdab843eb33e2bc1d9fc0d6050049a9
Wed Aug 19 2015 14:05:54 GMT-0500 (CDT) 61MB 0 UBI: processed inode a66e2ee50ad440e381e6bbed6d22c91f9c8b1857
Wed Aug 19 2015 14:05:54 GMT-0500 (CDT) 61MB 0 UBI: processed inode 3e359a52207b63542ca93c4885fb62f18efabfd8
Wed Aug 19 2015 14:05:55 GMT-0500 (CDT) 65MB 0 UBI: processed inode 7d97c557380f3037b088ca0b49464193f6009059
Wed Aug 19 2015 14:05:56 GMT-0500 (CDT) 69MB 0 UBI: processed inode e1ef3764ab1251986161a91bfbab89710fb49db8
Wed Aug 19 2015 14:05:56 GMT-0500 (CDT) 72MB 0 UBI: processed inode 6d2ef3e061e5351dbef1814ff117e1405a4dc92c
Wed Aug 19 2015 14:05:57 GMT-0500 (CDT) 72MB 0 UBI: processed inode 3c6fa2ef4353602afa97a36af0900c8d4594bd43
Wed Aug 19 2015 14:05:58 GMT-0500 (CDT) 74MB 0 UBI: processed inode 2c8d578398a56d5f9a9e21e5057d91e92bcf0c10
Wed Aug 19 2015 14:05:59 GMT-0500 (CDT) 74MB 0 UBI: processed inode 0223d9c2af3775a46cf40c63f4583e5a09b5a3c1
Wed Aug 19 2015 14:06:00 GMT-0500 (CDT) 75MB 0 UBI: processed inode ff3a610bfd1abbe2052d43a3103b825fae8fc56b
Wed Aug 19 2015 14:06:00 GMT-0500 (CDT) 76MB 0 UBI: processed inode dfc2966fd802bdecda95fd536b851c2010dfc741
Wed Aug 19 2015 14:06:02 GMT-0500 (CDT) 79MB 0 UBI: processed inode cfc88b8c5fa4ccb4a85fb23f3f37d02421829c52
Wed Aug 19 2015 14:06:02 GMT-0500 (CDT) 79MB 0 UBI: processed inode 337cb086c4f5ab171fbfeb8245b43b0cc59459e6
Wed Aug 19 2015 14:06:03 GMT-0500 (CDT) 79MB 0 UBI: processed inode f8f389de0423420304b49f98af3c625a42cacdcd
Wed Aug 19 2015 14:06:04 GMT-0500 (CDT) 80MB 0 UBI: processed inode b6cc0e26186c7f06cd3b7784e362e376f2c9c318
Wed Aug 19 2015 14:06:04 GMT-0500 (CDT) 80MB 0 UBI: processed inode 673e78ebe0cd59d18ebd14d439b31f25f3493125
Wed Aug 19 2015 14:06:05 GMT-0500 (CDT) 80MB 0 UBI: processed inode 564f27556e220375ffab7e739e83757e9a69e741
Wed Aug 19 2015 14:06:06 GMT-0500 (CDT) 81MB 0 UBI: processed inode 10bbb4cecd556e8ccf126d25e61da5000d5215ee
Wed Aug 19 2015 14:06:07 GMT-0500 (CDT) 82MB 0 UBI: processed inode 3bf3a6170d0b59a539046674c83642292cb2dc09
Wed Aug 19 2015 14:06:09 GMT-0500 (CDT) 83MB 0 UBI: processed inode 2ac61ebfde1ead452059dbf1a54d30a348a28e88
Wed Aug 19 2015 14:06:09 GMT-0500 (CDT) 84MB 0 UBI: processed inode 9504a725b6575fb3b8d295bd4e7b17c572bd2d79
Wed Aug 19 2015 14:06:10 GMT-0500 (CDT) 84MB 0 UBI: processed inode f7bad43e6a677f715082dac76decc15805bf02c0
Wed Aug 19 2015 14:06:13 GMT-0500 (CDT) 85MB 0 UBI: processed inode c91350be20b3b73b2be80dfa64651d3500ff7d47
Wed Aug 19 2015 14:06:14 GMT-0500 (CDT) 86MB 0 UBI: processed inode e934b10fbea1d7a42942fac130bf950061128c23
Wed Aug 19 2015 14:06:15 GMT-0500 (CDT) 86MB 0 UBI: processed inode ac690d0a8b364b254a2d9a6d26d957360c2c3e28
Wed Aug 19 2015 14:06:16 GMT-0500 (CDT) 87MB 0 UBI: processed inode e9f779026fb14e8f79e82724ffae57f29fc5a0eb
Wed Aug 19 2015 14:06:19 GMT-0500 (CDT) 89MB 0 UBI: processed inode 07409b8ed1ad865e93195ea5695fe6fae993e1c2
Wed Aug 19 2015 14:06:19 GMT-0500 (CDT) 71MB 1 unique_block_initializer: superblock processing complete
Wed Aug 19 2015 14:06:19 GMT-0500 (CDT) 71MB 0 unique_block_initializer disconnected
Wed Aug 19 2015 14:06:19 GMT-0500 (CDT) 71MB 0 unique_block_initializer exited
It's not clear to me if this is happening because GC isn't kicking in at these utilization levels or of there is something else going on. Next step is to review the code and see if there is anything that can be changed to keep this in check.
Pushing storage to a full 100GB, the memory used by storage_utilization_initializer stays around ~70MB, so I think that's stable, but unique_block_initializer keeps climbing. Looking closer at that for optimizations.