alex-sherman / deco

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

IndentationError: unexpected indent

dataoverflow opened this issue · comments

Python 3.5.1 / deco from pip installed from github

Test case:

from deco import concurrent, synchronized
import time


class DecoClass:

    @concurrent
    def work(self):
        time.sleep(0.1)

    @synchronized
    def run(self):
        for _ in range(100):
            self.work()

if __name__ == "__main__":
    start = time.time()
    x = DecoClass()
    x.run()
    print("Executing in serial should take 10 seconds")
    print("Executing in parallel took:", time.time() - start, "seconds")

The error message:

  File "<unknown>", line 1
    @synchronized
    ^
IndentationError: unexpected indent

Thanks for finding this! The problem is that the @synchronization decorator doesn't unindent the source of the function its attached to before trying to compile it into an AST. I wasn't able to find any built in functions to unindent the source lines, so the two manual ways I can think of are:

  1. Regex match indentation from the first line and remove it from all subsequent lines
  2. Call string.lstrip() on the lines up to a line matching a function definition but no further

Both seem kind of derpy, but I'll pick one and implement it some time today. If anyone has some input or alternative I'm happy to hear it.

I've committed b48e575 to address this issue and fix the bug with indentation handling.

However, I also ran across another issue with the example. Because the synchronized class doesn't provide a __get__ method, it won't function correctly when used on class methods. To address this in the immediate future I added a __get__ method that throws an exception explaining what's happening. Perhaps in the future we can support this functionality but I'm not really sure if deco would become that much more useful for how much more complicated it would need to be.

@dataoverflow, if you had a specific use case in mind for this and could provide it I would be really interested to see how you would make use of this functionality.

@alex-sherman my use case is a class which follows interfaces from sklearn (fit/predict etc) so it can be used along with existing sklearn implementation. My fit() method in my class has a loop which would benefit from concurrent execution. It'd be great if I could annotate methods of my class with deco. Refactoring this code into a module purely so I can use deco seem not to be a great option in my opinion.

Thanks for the reply! I don't have much familiarity with sklearn so it's cool to see someone using deco with it. I tried just briefly to get the decorators to work with class methods, but ran into some complications. I unfortunately don't have enough free time at the moment to work through this, but will definitely consider it at a later date. If you're interested here are the main complications:

Descriptor API and implementing get
Both concurrent and synchronized decorators need to implement __get__(self, obj, type = None) methods in order to be used as class methods. That problem lies in being able to return basically a copy of the wrapper object that has a modified __call__ attribute which will contain the appropriate self in it's *args.

More complicated AST parsing
This is really the hardest problem to solve. The AST parsing has remained really simple so far because deco can expect concurrent functions to be in the global namespace. Once that assumption is gone, detecting which function calls are actually concurrent functions becomes more complicated.

TL;DR:
Working with class methods is maybe possible, but definitely hard. I hope to revisit this at a later date, but hopefully you'll still find deco useful in the mean time.

Thanks again for mentioning this!

Alex, thank you for looking into this, even if at a later date! I found deco very useful already. I am using it in a different part of my project and it works great.