Regression: bloaty rejects compressed debug info (implausible uncompressed size) for real C++ shared library
hesiod opened this issue · comments
PR #273 added a check whether the uncompressed debug info size is greater than 12 times the compressed size. However this limit can be hit with real-world shared libraries.
The shared library I hit this issue with is a C++ library that uses certain Boost libraries (e.g. Boost.Geometry) that rely on an ungodly amount of template parameters, leading to extremely long symbols names.
Possible solutions:
- If the intention of #273 is to reject crafted ELF files with extremely large uncompressed debug info for security reasons, then it might make sense to add a fixed size limit (e.g. 128 MB) for the uncompressed debug info size
- Add a command line flag to override the limit
- Raise the compression ratio limit (but on the other hand: Why punish the compression algorithm if it does a good job?)
Ping @haberman
Compare libexample.so (compressed debug information):
FILE SIZE VM SIZE
-------------- --------------
38.3% 3.91Mi 0.0% 0 .debug_info
16.9% 1.72Mi 62.2% 1.72Mi .text
16.7% 1.71Mi 0.0% 0 .debug_str
6.6% 686Ki 0.0% 0 .strtab
5.0% 524Ki 18.5% 524Ki .dynstr
4.7% 489Ki 0.0% 0 .debug_loclists
2.8% 291Ki 0.0% 0 .debug_line
1.9% 199Ki 7.0% 199Ki .eh_frame
1.5% 156Ki 0.0% 0 .debug_loc
1.1% 114Ki 0.0% 0 .symtab
0.8% 81.5Ki 2.6% 74.8Ki [29 Others]
0.7% 74.1Ki 0.0% 0 .debug_rnglists
0.7% 69.6Ki 2.4% 69.5Ki .dynsym
0.5% 48.9Ki 1.7% 48.9Ki .gcc_except_table
0.4% 43.5Ki 1.5% 43.4Ki .rela.plt
0.3% 35.6Ki 1.3% 35.5Ki .rodata
0.3% 29.3Ki 0.0% 0 .debug_ranges
0.3% 29.0Ki 1.0% 29.0Ki .plt
0.3% 26.6Ki 0.9% 26.6Ki .eh_frame_hdr
0.2% 25.2Ki 0.0% 0 .debug_abbrev
0.2% 23.2Ki 0.8% 23.2Ki .hash
100.0% 10.2Mi 100.0% 2.77Mi TOTAL
With libexample.so (uncompressed debug information):
FILE SIZE VM SIZE
-------------- --------------
38.1% 20.2Mi 0.0% 0 .debug_info
33.7% 17.9Mi 0.0% 0 .debug_str
10.5% 5.59Mi 0.0% 0 .debug_loc
4.2% 2.26Mi 0.0% 0 .debug_line
3.2% 1.72Mi 62.1% 1.72Mi .text
3.1% 1.63Mi 0.0% 0 .debug_loclists
3.0% 1.59Mi 0.0% 0 .debug_ranges
1.3% 686Ki 0.0% 0 .strtab
1.0% 524Ki 18.5% 524Ki .dynstr
0.4% 217Ki 0.0% 0 .debug_rnglists
0.4% 199Ki 7.0% 199Ki .eh_frame
0.3% 176Ki 0.0% 0 .debug_abbrev
0.2% 114Ki 0.0% 0 .symtab
0.2% 92.9Ki 2.6% 74.8Ki [29 Others]
0.1% 69.6Ki 2.4% 69.5Ki .dynsym
0.1% 48.9Ki 1.7% 48.8Ki .gcc_except_table
0.1% 43.5Ki 1.5% 43.4Ki .rela.plt
0.1% 35.6Ki 1.3% 35.5Ki .rodata
0.1% 29.0Ki 1.0% 29.0Ki .plt
0.0% 26.7Ki 0.9% 26.6Ki .eh_frame_hdr
0.0% 23.2Ki 0.8% 23.2Ki .hash
100.0% 53.1Mi 100.0% 2.77Mi TOTAL
In comparison:
FILE SIZE VM SIZE
-------------- --------------
+417% +16.3Mi [ = ] 0 .debug_info
+947% +16.2Mi [ = ] 0 .debug_str
+36e2% +5.44Mi [ = ] 0 .debug_loc
+692% +1.97Mi [ = ] 0 .debug_line
+55e2% +1.57Mi [ = ] 0 .debug_ranges
+241% +1.15Mi [ = ] 0 .debug_loclists
+600% +151Ki [ = ] 0 .debug_abbrev
+194% +143Ki [ = ] 0 .debug_rnglists
+176% +8.20Ki [ = ] 0 .debug_line_str
+219% +3.19Ki [ = ] 0 .debug_aranges
+0.0% +80 [ = ] 0 .strtab
+0.0% +8 +0.0% +8 .eh_frame_hdr
-2.7% -4 -2.7% -4 [LOAD #1 [R]]
-0.0% -8 [ = ] 0 .eh_frame
-0.0% -24 [ = ] 0 .symtab
-0.1% -28 -0.1% -28 .gcc_except_table
-0.0% -864 -0.0% -864 .text
+420% +42.9Mi -0.0% -888 TOTAL
Bloaty rejects the compressed version with the following message:
warning: ignoring compressed debug data, implausible uncompressed size (compressed: 29886, uncompressed: 375184)
For the record, this is one of the extremely long symbols (11729 characters):
bool boost::geometry::detail::partition::partition_two_ranges<0, boost::geometry::model::box<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian> > >::apply<std::vector<__gnu_cxx::__normal_iterator<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> > const*, std::vector<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> >, std::allocator<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> > > > >, std::allocator<__gnu_cxx::__normal_iterator<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> > const*, std::vector<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> >, std::allocator<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> > > > > > >, std::vector<__gnu_cxx::__normal_iterator<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring const*, std::vector<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring, std::allocator<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring> > >, std::allocator<__gnu_cxx::__normal_iterator<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring const*, std::vector<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring, std::allocator<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring> > > > >, boost::geometry::detail::buffer::turn_in_original_visitor<std::vector<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> >, std::allocator<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> > > >, boost::geometry::strategies::relate::cartesian<void> >, boost::geometry::detail::buffer::turn_get_box<boost::geometry::strategies::relate::cartesian<void> >, boost::geometry::detail::buffer::turn_in_original_overlaps_box<boost::geometry::strategies::relate::cartesian<void> >, boost::geometry::detail::buffer::original_get_box<boost::geometry::strategies::relate::cartesian<void> >, boost::geometry::detail::buffer::original_overlaps_box<boost::geometry::strategies::relate::cartesian<void> >, boost::geometry::detail::partition::visit_no_policy>(boost::geometry::model::box<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian> > const&, std::vector<__gnu_cxx::__normal_iterator<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> > const*, std::vector<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> >, std::allocator<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> > > > >, std::allocator<__gnu_cxx::__normal_iterator<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> > const*, std::vector<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> >, std::allocator<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> > > > > > > const&, std::vector<__gnu_cxx::__normal_iterator<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring const*, std::vector<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring, std::allocator<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring> > >, std::allocator<__gnu_cxx::__normal_iterator<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring const*, std::vector<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring, std::allocator<boost::geometry::detail::buffer::buffered_piece_collection<boost::geometry::model::ring<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, true, true, std::vector, fast_pool_allocator_t>, boost::geometry::strategies::relate::cartesian<void>, boost::geometry::strategy::buffer::distance_symmetric<double>, boost::geometry::detail::robust_policy<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::model::point<long long, 2ul, boost::geometry::cs::cartesian>, double> >::original_ring> > > > > const&, unsigned long, unsigned long, boost::geometry::detail::buffer::turn_in_original_visitor<std::vector<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> >, std::allocator<boost::geometry::detail::buffer::buffer_turn_info<boost::geometry::model::d2::point_xy<double, boost::geometry::cs::cartesian>, boost::geometry::segment_ratio<long long> > > >, boost::geometry::strategies::relate::cartesian<void> >&, boost::geometry::detail::buffer::turn_get_box<boost::geometry::strategies::relate::cartesian<void> > const&, boost::geometry::detail::buffer::turn_in_original_overlaps_box<boost::geometry::strategies::relate::cartesian<void> > const&, boost::geometry::detail::buffer::original_get_box<boost::geometry::strategies::relate::cartesian<void> > const&, boost::geometry::detail::buffer::original_overlaps_box<boost::geometry::strategies::relate::cartesian<void> > const&, boost::geometry::detail::partition::visit_no_policy&)
Thanks for the report and sorry for the breakage.
Indeed this was to avoid security vulnerabilities leading to excessive memory allocation.
I'm a little worried about setting a fixed size limit, since this would be an effective cap on the size of translation unit Bloaty can handle. It seems annoying to have to override this just because you are profiling a large file.
It looks like your example comes in just above my arbitrarily-chosen "12." What if we just make the limit 100 instead (eg. roughly 10x what you observed)?
100 sounds like a reasonable limit, but on the other hand I'm wondering whether a fixed limit on compression ratio is future-proof (after all, perhaps in a few years another language or compiler joins the block and generates even longer symbol names for some reason).
All in all this is your choice, but my suggestion would be to raise the limit only a moderate amount (say, up to a ratio of 20x or 30x) and add a command line flag to override the check. I think an override flag would be a good solution since I'd guess around 95% of the files bloaty sees have a compression ratio below 5 or so, and in the remaining 5% it shouldn't be an issue to manually override the check.
I see your point, but I think I'd prefer to avoid an extra flag if possible. Extra configuration knobs multiply the testing matrix, and are hard to change/deprecate later. It also asks more of the user.
To decrease the chance of running into the limit, I made it 30x + 128MB. I think it will take some extreme input to trigger that check, but it also seems low enough to avoid triggering OOM in real systems or fuzzers.
To decrease the chance of running into the limit, I made it 30x + 128MB. I think it will take some extreme input to trigger that check, but it also seems low enough to avoid triggering OOM in real systems or fuzzers.
I didn't think of a combined limit, but that sounds like an excellent idea! Thanks for the fix.