Improve error reporting of interpreter fallback

[Lukáš Petrovický]

jpyinterpreter is a very complex piece of code. It would be naive to think that it implements every little nuance of Python bytecode correctly. (Case in point: #146.)

The current behavior is that such issues throw an exception, and the user is effectively locked out of using OptaPlanner. I am proposing that we implement a different behavior instead - we log an error, and fall back to unoptimized version. Slower performance beats no performance 100 % of the time.

[Christopher Chianelli]

In fact, we do that, see

optapy/jpyinterpreter/tests/test_functions.py

Lines 266 to 286 in 3c35ef7

 def test_code_works_if_compilation_failed(): 
     def my_function(x: int) -> tuple: 
         def inner_function(y: int) -> type: 
             nonlocal x 
  
             class MyClass:  # TODO: Replace this with something else that fails when class creation is supported 
                 def __init__(self): 
                     self.outer_arg = x 
                     self.inner_arg = y 
  
                 def get_args(self): 
                     return self.outer_arg, self.inner_arg 
  
             return MyClass 
  
         return inner_function(2 * x)().get_args() 
  
     verifier = verifier_for(my_function) 
     verifier.verify(1, expected_result=(1, 2)) 
     verifier.verify(2, expected_result=(2, 4)) 
     verifier.verify(3, expected_result=(3, 6)) 

and

optapy/jpyinterpreter/src/main/python/python_to_java_bytecode_translator.py

Lines 227 to 300 in 3c35ef7

 elif inspect.iscode(value): 
     try: 
         from org.optaplanner.jpyinterpreter.types import PythonLikeFunction, PythonCode 
         java_class = translate_python_code_to_java_class(value, PythonLikeFunction) 
         out = PythonCode(java_class) 
         put_in_instance_map(instance_map, value, out) 
         return out 
     except: 
         from org.optaplanner.jpyinterpreter.types import PythonLikeFunction, PythonCode 
         java_class = translate_python_code_to_python_wrapper_class(value) 
         out = PythonCode(java_class) 
         put_in_instance_map(instance_map, value, out) 
         return out 
 elif type(value) is object: 
     java_type = type_to_compiled_java_class[type(value)] 
     out = CPythonBackedPythonLikeObject(java_type) 
     put_in_instance_map(instance_map, value, out) 
     CPythonBackedPythonInterpreter.updateJavaObjectFromPythonObject(out, 
                                                                     JProxy(OpaquePythonReference, inst=value, 
                                                                            convert=True), 
                                                                     instance_map) 
     return out 
 elif not inspect.isfunction(value) and type(value) in type_to_compiled_java_class: 
     if type_to_compiled_java_class[type(value)] is None: 
         return None 
     java_type = type_to_compiled_java_class[type(value)] 
     if isinstance(java_type, CPythonType): 
         return None 
     java_class = java_type.getJavaClass() 
     out = java_class.getConstructor(PythonLikeType).newInstance(java_type) 
     put_in_instance_map(instance_map, value, out) 
     CPythonBackedPythonInterpreter.updateJavaObjectFromPythonObject(out, 
                                                                     JProxy(OpaquePythonReference, inst=value, 
                                                                            convert=True), 
                                                                     instance_map) 
  
     if isinstance(out, AbstractPythonLikeObject): 
         for (key, value) in getattr(value, '__dict__', dict()).items(): 
             out.setAttribute(key, convert_to_java_python_like_object(value, instance_map)) 
  
     return out 
 elif inspect.isbuiltin(value) or is_c_native(value): 
     return None 
 elif inspect.isfunction(value): 
     try: 
         from org.optaplanner.jpyinterpreter.types import PythonLikeFunction 
         wrapped = PythonLikeFunctionWrapper() 
         put_in_instance_map(instance_map, value, wrapped) 
         out = translate_python_bytecode_to_java_bytecode(value, PythonLikeFunction) 
         wrapped.setWrapped(out) 
         put_in_instance_map(instance_map, value, out) 
         return out 
     except: 
         return None 
 else: 
     try: 
         java_type = translate_python_class_to_java_class(type(value)) 
         if isinstance(java_type, CPythonType): 
             return None 
         java_class = java_type.getJavaClass() 
         out = java_class.getConstructor(PythonLikeType).newInstance(java_type) 
         put_in_instance_map(instance_map, value, out) 
         CPythonBackedPythonInterpreter.updateJavaObjectFromPythonObject(out, 
                                                                         JProxy(OpaquePythonReference, inst=value, 
                                                                                convert=True), 
                                                                         instance_map) 
  
         if isinstance(out, AbstractPythonLikeObject): 
             for (key, value) in getattr(value, '__dict__', dict()).items(): 
                 out.setAttribute(key, convert_to_java_python_like_object(value, instance_map)) 
  
         return out 
     except: 
         return None 

. I suspect it falls through because of

optapy/jpyinterpreter/src/main/python/python_to_java_bytecode_translator.py

Lines 1220 to 1224 in 3c35ef7

 out = PythonClassTranslator.translatePythonClass(python_compiled_class) 
 type_to_compiled_java_class[python_class] = out 
 PythonClassTranslator.setSelfStaticInstances(python_compiled_class, out.getJavaClass(), out, 
                                              CPythonBackedPythonInterpreter.pythonObjectIdToConvertedObjectMap) 
 return out 

, which is not wrapped in a try...except (and hence why it is caused by optapy annotations which directly call translate_python_class_to_java_class to get the superclass needed)

[Lukáš Petrovický]

Thanks for the pointers. They don't show me the error messages that we output if/when this happens, though. (We want people to tell us of these errors, don't we? Even if we handle them properly by falling back.)

[Christopher Chianelli]

The error situation can definitely improve. I am planning in the future to return a CompliationResult instead of the raw Java class. The CompliationResult would have fields for errors and warnings (ex: potential None reference, type mismatch, missing method/field, etc). The translator can then report not just mistranslation errors (our errors) but also user errors.

[Lukáš Petrovický]

Ok, I updated the title of this issue.