ValueError when adding support for Libgcrypt 1.8.4

[Winston Yin]

So right now I'm trying to add support for identification of the library libgcrypt. This time, I created a new JSON configuration file from the non-inlined and compiled version of libgcrypt. When I run karta_matcher.py on a test case for this library, it is capable of identifying libgcrypt. However, when it reaches the point of Linking the binary functions to their respective tentative files, Karta throws this exception:

C:\Users\wyin\Documents\Karta\src\karta_matcher.py: None is not in list
Traceback (most recent call last):
  File "C:\Program Files\IDA Pro 7.3\python\ida_idaapi.py", line 572, in IDAPython_ExecScript
    execfile(script, g)
  File "C:/Users/wyin/Documents/Karta/src/karta_matcher.py", line 193, in <module>
    pluginMain()
  File "C:/Users/wyin/Documents/Karta/src/karta_matcher.py", line 183, in pluginMain
    matchLibraries()
  File "C:/Users/wyin/Documents/Karta/src/karta_matcher.py", line 144, in matchLibraries
    matchLibrary(lib_name, actual_version)
  File "C:/Users/wyin/Documents/Karta/src/karta_matcher.py", line 95, in matchLibrary
    startMatch(cur_config_path, lib_name)
  File "C:/Users/wyin/Documents/Karta/src/karta_matcher.py", line 60, in startMatch
    matching_engine.locateFileBoundaries()
  File "C:/Users/wyin/Documents/Karta/src\core\matching_engine.py", line 587, in locateFileBoundaries
    file_match = file_class(file_name, src_start_index, src_end_index, local_bins_ctx, local_bin_start_index, local_bin_end_index, src_end_index - src_start_index + 1, self)
  File "C:/Users/wyin/Documents/Karta/src\file_layer.py", line 29, in __init__
    super(FileMatcher, self).__init__(name, src_index_start, src_index_end, fuzzy_bin_functions_ctx, bin_limit_lower, bin_limit_upper, src_scope, engine)
  File "C:/Users/wyin/Documents/Karta/src\core\file_layer.py", line 142, in __init__
    bin_range = range(self._bin_functions_ctx.index(self._lower_match_ctx), self._bin_functions_ctx.index(self._upper_match_ctx) + 1)
ValueError: None is not in list

Again, the identification script I'm using for libgcrypt is just a slightly edited version of the identification script for zlib. I'm running Karta on IDA Pro 7.3
Do you know what could be causing this error and what I can do to fix it?

[Eyal Itkin]

The only option in which self._lower_match_ctx will be None and/or self._upper_match_ctx will be None is in the case that the binary file will be marked as located but with size of 0 function. It looks like the for loop preform no iterations and this is why the contexts won't get initialized.

I can better handle this error case on my side, however I still want to know why the binary file looks empty. Can you post the number of functions in each src file, and the len(self._bin_functions_ctx) at the beginning of this function? Maybe the compilation of your configuration file contained errors / wasn't without optimizations

[Winston Yin]

Thanks for the quick response, these are the function counts(extracted from the JSON config file) for each of the source files used:

• dumpsexp-dumpsexp.o is 16
• hmac256-hmac256.o is 10
• mpicalc-mpicalc.o is 21
• .libs\hwf-x86.o is 5
• .libs\libgcrypt_la-context.o is 4
• .libs\libgcrypt_la-fips.o is 23
• .libs\libgcrypt_la-global.o is 35
• .libs\libgcrypt_la-hmac256.o is 8
• .libs\libgcrypt_la-hwfeatures.o is 5
• .libs\libgcrypt_la-misc.o is 26
• .libs\libgcrypt_la-secmem.o is 24
• .libs\libgcrypt_la-sexp.o is 41
• .libs\libgcrypt_la-stdmem.o is 6
• .libs\libgcrypt_la-visibility.o is 217
  When I try to obtain the length of self._bin_functions_ctx, I still get ValueError: None is not in list. The command I used to run karta_analyze_src.py is python karta_analyze_src.py libpython 2.7.0 C:/Users/wyin/Documents/libpython C:/Users/wyin/Documents/libpython/libpython2.7.a. When I compiled the makefile for libgcrypt, I changed the CFLAGS from -g -O2 -fvisibility=hidden -fno-delete-null-pointer-checks -Wall to -g -O0 -fno-inline-functions -fno-builtin -fno-delete-null-pointer-checks -Wall and the CCASFLAGS from -g -O2 to -g -O2 fno-inline-functions -fno-builtin -fno-delete-null-pointer-checks. I've attached the makefile and the JSON configuration file below. Are there any optimizations or issues with the way I'm making the JSON config file?

Makefile.zip

libgcrypt_1.8.4.zip

[Eyal Itkin]

Added a commit to better handle this edge case, including clearer logs so the relevant file will be shown too.

We had similar errors in the past when IDA's analysis wasn't perfect / missed a few functions. My suggestion is to check again if you can help IDA better detect all of the functions - even functions without xrefs should be marked as functions so Karta will work as intended. I recommend using Thumbs Up: https://karta.readthedocs.io/en/latest/Thumbs%20Up.html

[Eyal Itkin]

Closing for now.
If the fix didn't solve the issue, feel free to re-open the ticket.