argparse doesn't play nice with skedulord

[Enrico Alemani]

I have a question about argument parsing.
Looking at the getting started guide (schedule section), I noted that you pass some custom arguments into the python script (i.e. --output and --repo). I double checked the source code, and my understanding output and repo are custom argument rather than default argument in skedulord like retry and wait - let me know if this is not correct.

I tried to do the same with my own project but I'm currently getting an unrecognized arguments error.
Here a sample script to replicate the error:

# test_lord.py
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--myarg1', help='arg1 name')
parser.add_argument('--myarg2', help='arg2 name')

args = parser.parse_args()

print(args.myarg1)
print(args.myarg2)

# schedule.yml
user: user
schedule:
    - name: scraping-job
      command: python full/path/to/test_lord.py --myarg1 hello --myarg2 world
      retry: 1
      wait: 0
      cron: "*/1 * * * *"

Is the unrecognized arguments error expected or is there some other issues going on?

Thank you for your time. Appreciated.
And thank you very much for skedulord. I find it a really nice project.

[vincent d warmerdam]

I haven't used the repo in months but the design is that when you run directly you need to "stringify" your own command. Like so;

python -m skedulord run jobname1 "python script.py" --retry 3 --wait 60

You won't need to do this from the schedule.yml file because skedulord will do the "stringification" for you. That said, could you share the actual traceback? That makes it easier to debug.

[Enrico Alemani]

Thanks for the prompt reply.

I'm running the script above using:

python -m skedulord schedule schedule.yml

I do not get an error in the terminal. However, skedulord log shows the following:

run_id=2290b4e1                                                                                                         
name=scraping-job                                                                                                       
command=/Users/user/miniforge3/envs/my-project/bin/python                                                         
/Users/user/Documents/Projects/my-project/scripts/main/test_lord.py --myarg1 hello --myarg2 world                 
attempt=1                                                                                                               
datetime=2021-06-16                                                                                                     
usage: test_lord.py [-h] [--myarg1 MYARG1] [--myarg2 MYARG2]                                                            
test_lord.py: error: unrecognized arguments:

I'm running skedulord in a conda environment. I made sure it is activated.
I'm using python 3.8 on a macbook pro M1 2020.
Let me know if you need more information.

I have also tested the script by running it from the terminal python scripts/main/test_lord.py --myarg1 hello --myarg2 world and it seems to work properly.

[Edit]
I have now modify the test_lord.py script to use sys.argv rather than argparse and it works running skedulord through the .yml file.

# test_lord.py
import sys

print('Number of arguments: {}'.format(len(sys.argv)))
print('Argument(s) passed: {}'.format(str(sys.argv)))

I'll do more tests and let you know.

[vincent d warmerdam]

What happens when you run

/Users/user/miniforge3/envs/my-project/bin/python                                                         
/Users/user/Documents/Projects/my-project/scripts/main/test_lord.py --myarg1 hello --myarg2 world

[Enrico Alemani]

When I run it in the terminal, hello and world are successfully printed in the terminal.

[vincent d warmerdam]

Then I don't know what's happening here. It might be something related to argparse. I mainly use typer and I've never used it together with argparse before.

[Enrico Alemani]

Thanks for checking. I'll test a bit more and if I find a solution for argparse, I'll post it here.
I tried sys.argv and it seems to work ok.

We can close the issue for now.
(Again) thank you for your help. Appreciated 👍

[vincent d warmerdam]

I'm fine with keeping the issue open. I don't have time to work on this project for a while, but the idea of the project is that it should work for general scripts. Argparse included.

[vincent d warmerdam]

I just fixed a bug where commands would get put into crontab like:

"python full/path/to/test_lord.py --myarg1 hello --myarg2 world "

As opposed to

"python full/path/to/test_lord.py --myarg1 hello --myarg2 world"

Notice the extra space there? That might've been the issue. You can install from GitHub to try out the fix.

[vincent d warmerdam]

Closing due to radio silence.