code review?

[anarcat]

would you be interested in a code review? i have a few suggestions of improvements that could be done to the code...

[Ricardo F.]

Yes, please report whatever you consider opportune.

Thanks,

[anarcat]

alright... so here they come:

• take a look at use_scm_version in setuptools_scm or versionner to manage the version string instead of hardcoding it
• as mentionned in #14, latest/ is a rather uncommon directory structure. i had trouble finding the actual source when i first looked at the git repo. i recommend removing the directory altogether and moving all files directly to the top-level, or at least tuptime itself to the toplevel, renaming latest to etc. as a last resort, rename to src

try:
    import sys, os, optparse, locale, platform, subprocess, time, sqlite3
    from datetime import datetime
except Exception as e:
    sys.exit('ERROR modules are not imported correctly: ' + str(e))

do not hide exceptions from imports. this would hide what the problem actually is: it could be there is an error in one of the imported modules and the exception handling here would make it needlessly hard for the user to figure out what is wrong. those modules are all part of the standard library so there shouldn't be any such exception anyways. and if some or not, they should be on separate lines.

in general, follow the style guides around: the google style guide is pretty amazing, but big. the canonical source is of course pep8 and there are great commandline tools like pep8 and flake to check files for style issues. i recommend you use those, which will identify more issues than even i could find here. :)

def vprint(*args):
    """Print verbose information"""
    if opt.verbose:
        for arg in args:
            sys.stdout.write(str(arg))
        print('')

i wrote all this stuff very often. :) first off, you can pass all the args directly to sys.stdout.write() with sys.stdout.write(*args) to avoid that clunky loop. then i recommend you take a look at the logging module which supports writing to syslog and a bunch of similar output devices, which could be very useful for this software, instead of writing your own verbosity handler. i often map -v to the log level, see bup-cron on how i setup the logging module there.

def time_conv(secs):
[...]

you went through a lot of trouble there! :) i believe that the datetime module can do such conversions automatically, try this out: https://github.com/anarcat/bup-cron/blob/master/bup_cron/__init__.py#L860

also, just quickly: https://github.com/rfrail3/tuptime/blob/master/latest/tuptime#L91

you should split up main() some more... make functions based on different ... well, functions: argument parsing, writing the database, calculating time could all be split up, and it would make code review easier.

more to come...

[anarcat]

rows_to_dict seems strange: i am pretty sure you can extract a named tuple or straight out dict directly from the database.

            if platform_env == 'FreeBSD':
                sysctl_bin = '/sbin/sysctl'
            if platform_env == 'Darwin':
                sysctl_bin = '/usr/sbin/sysctl'
            sysctl = subprocess.check_output([sysctl_bin, 'kern.boottime']);

you should rely on the $PATH variable here (which you can expand yourself by editing os.environ) instead of hardcoding paths and trying to detect environment. same with detecting /proc/uptime: look for it first, and if you fail, fallback to sysctl. you could even enclose each of those checks in separate functions to make it easier to distinguish them and implement new platforms.

    try:  # - Reading last btime and uptime registered in db
        conn.execute('select btime, uptime from tuptime where oid = (select max(oid) from tuptime)')
        last_btime, last_uptime = conn.fetchone()
        vprint('Last btime from db = ', last_btime)
        vprint('Last uptime from db = ', str(last_uptime))
        lasts = last_uptime + last_btime
    except Exception as e:
        sys.exit('ERROR reading from db: '+ str(e))

in this case, and many similar others, you are checking exceptions too widely. in the above code, for example, you should put the try: expect: only around the statement that will trigger a database error, and only check the exception type that is documented in the sqlite3 connection documentation. otherwise you may hide other unrelated coding errors or bugs.

    except Exception as e:
        if type(e).__name__ == 'OperationalError':

this is another example of this problem: if you had exception handling more restricted to less statements, you could have said:

except OperationalError:
 #...

...instead of this unconventional error checking.

        db_rows = conn.fetchall()
        db_rows[-1] = list(db_rows[-1])
        db_rows[-1][2] = uptime
        db_rows[-1][4] = opt.endst
        db_rows[-1][6] = kernel
        db_rows[-1] = tuple(db_rows[-1])

this is just weird. what are you trying to do here? why not simply db_rows.append([something])?? and why append to db_rows in the first place? is this the new entry you're adding to the DB? then why not just INSERT it right now?

that's about it for now. i got confused from here on - it would be much easier to review the code once you refactor it in separate functions.

[Ricardo F.]

• Good to know use_scm_versions. But I don't understand, what is the problem with hardcoding the version number? Its not easier to change the number in a few files and avoid relaying on other stuff to do the same in a small proyect? Please, don't be afraid, maybe I don't have the experience for view the problem. (TODO?)
• I moved the source to src, its a best name for their contents. I prefer separate them from the the toplevel between general files like readmes and other general things not exactly related with the working needings of the program.
• Modules are imported without try/except, you are right.
• I ran pylint before but I only fix the worst problems. Now it have less recomendations in the report and I keep them as low as possible. (TODO)
• I can't put sys.stdout.write(*args) for the verbose mode, it report 'takes exactly 1 argument (2 given)'. The solutions that I found in google did't help me in replace this two lines of code in other more simple, sorry.
• Syslog was excluded because I rely on systemd journal for logging. It catch stout/stderr by default. In a future I plan to replace the cron execution with a time unit for this reason.
• If datetime.timedelta report years, it would be the solution, but it don't do.
• I wrote functions only for avoid duplicate code, making more functions is a code improvement that I will do in future relases. (TODO)
• rows_to_dict ... someting to investigate. (TODO)
• I didn't rely in os.environ. It was a pull request and I did't modify, thanks. (TODO)
• Exceptions, well I did it for cover general cases but as you said, is not the best option. If I make a particular exception for OperationalError, later, how can I report the other prints for any other exception, its only allow one Exception. (TODO)
• I can't rely on know that the values are inserted on the db because if tuptime is executed with a unprivileged user that can't write into db, the last values will be wrong. For that reason, the first execution of the tuptime after a reboot need to rely on a user who can write into db. This was a bug in one of the previous releases.

Apart, I need to refeactor global variables and I continue fixing this issues in dev branch, master have a lot of references over internet for install tuptime and its a requirement to keep it stable.

Thanks!

[anarcat]

On 2015-12-04 12:31:40, Ricardo F. wrote:

• Good to know use_scm_versions. But I don't understand, what is the
  problem with hardcoding the version number?

It is more error prone to manually change the version number: you are
likely to forget. Plus, it introduces needless changes to the version
control history.

But yes, it does mean an extra dependency.

[...]

• I can't put sys.stdout.write(*args) for the verbose mode, it report
  'takes exactly 1 argument (2 given)'. The solutions that I found in
  google did't help me in replace this two lines of code in other more
  simple, sorry.

You're right, of course: i was thinking of logging.info(*args).

• Syslog was excluded because I rely on systemd journal for
  logging. It catch stout/stderr by default. In a future I plan to
  replace the cron execution with a time unit for this reason.

Well, you assume systemd is used, but then you provide hooks for
init.d-style setups, including FreeBSD.

You should really support syslog, i think. But even if you don't want
to, the logging library could help, and it's part of the standard
library.

• If datetime.timedelta report years, it would be the solution, but it don't do.

I see... never got into those date ranges! But i recommend using an
existing library here as well: dateutil performs this (and other things)
pretty well!

http://stackoverflow.com/a/8971809/1174784
http://labix.org/python-dateutil

• Exceptions, well I did it for cover general cases but as you said,
  is not the best option. If I make a particular exception for
  OperationalError, later, how can I report the other prints for any
  other exception, its only allow one Exception. (TODO)

Not sure i understand you here. You can do multiple except statements in
one try block, that's for sure...

• I can't rely on know that the values are inserted on the db because if tuptime is executed with a unprivileged user that can't write into db, the last values will be wrong. For that reason, the first execution of the tuptime after a reboot need to rely on a user who can write into db. This was a bug in one of the previous releases.

Not sure i follow... what does this refer to?

Apart, I need to refeactor global variables and I continue fixing this issues in dev branch, master have a lot of references over internet for install tuptime and its a requirement to keep it stable.

Of course! Thanks for your work.

A.

That's the kind of society I want to build. I want a guarantee - with
physics and mathematics, not with laws - that we can give ourselves
real privacy of personal communications.
- John Gilmore

[Ricardo F.]

Hi Anarcat,

• Pylint report a note over 9.5, it's much better than before.
• Verbose now use logging and not the function with sys.stdout.write(*args). I don't implement logging levels as you did in your bup-cron, with basic functionallity is great. It works nice with it.
• python-dateutil is better than datetime.timedelta but as it is an external module and the actual function works well, I prefer continue with it.
• A great refactoring was did in the code. Getting values have been splitted in multiple functions, core login and db is still in main function and print section have been splitted too in other functions. rows_to_dict function was replaced with row_factory property. os.environ works great and os detection is better defined... Please, check the changes in dev branch.

[Ricardo F.]

Hi Anarcat,

Do you have any other reply or recomendation? I'm planning merge dev branch into master along the week.

Thanks,

[anarcat]

On 2015-12-16 06:41:42, Ricardo F. wrote:

Hi Anarcat,

Do you have any other reply or recomendation? I'm planning merge dev branch into master along the week.

I won't have time to do another review until next year (january, so not
so far ;). Please do merge and release, do not wait for me...

Choose a job you love and you will never have to work a day in your
life.
- Confucius

[Ricardo F.]

Ok Anarcat, release done.

Thanks,