Bug & Fix notice: ValueError: No closing quotation

[mormoassaf]

Example error stack

    .batch(self._batch(node))
           ^^^^^^^^^^^^^^^^^
  File "/Users/moassaf/anaconda3/lib/python3.11/site-packages/metaflow/plugins/aws/step_functions/step_functions.py", line 678, in _batch
    .create_job(
     ^^^^^^^^^^^
  File "/Users/moassaf/anaconda3/lib/python3.11/site-packages/metaflow/plugins/aws/batch/batch.py", line 208, in create_job
    self._command(
  File "/Users/moassaf/anaconda3/lib/python3.11/site-packages/metaflow/plugins/aws/batch/batch.py", line 97, in _command
    return shlex.split('bash -c "%s"' % cmd_str)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/moassaf/anaconda3/lib/python3.11/shlex.py", line 315, in split
    return list(lex)
           ^^^^^^^^^
  File "/Users/moassaf/anaconda3/lib/python3.11/shlex.py", line 300, in __next__
    token = self.get_token()
            ^^^^^^^^^^^^^^^^
  File "/Users/moassaf/anaconda3/lib/python3.11/shlex.py", line 109, in get_token
    raw = self.read_token()
          ^^^^^^^^^^^^^^^^^
  File "/Users/moassaf/anaconda3/lib/python3.11/shlex.py", line 191, in read_token
    raise ValueError("No closing quotation")
ValueError: No closing quotation

My fix was:

 def _command(self, environment, code_package_url, step_name, step_cmds, task_spec):
        mflog_expr = export_mflog_env_vars(
            datastore_type="s3",
            stdout_path=STDOUT_PATH,
            stderr_path=STDERR_PATH,
            **task_spec
        )
        init_cmds = environment.get_package_commands(code_package_url, "s3")
        init_expr = " && ".join(init_cmds)
        step_expr = bash_capture_logs(
            " && ".join(environment.bootstrap_commands(step_name, "s3") + step_cmds)
        )

        # construct an entry point that
        # 1) initializes the mflog environment (mflog_expr)
        # 2) bootstraps a metaflow environment (init_expr)
        # 3) executes a task (step_expr)

        # the `true` command is to make sure that the generated command
        # plays well with docker containers which have entrypoint set as
        # eval $@
        cmd_str = "true && mkdir -p %s && %s && %s && %s; " % (
            LOGS_DIR,
            mflog_expr,
            init_expr,
            step_expr,
        )
        # after the task has finished, we save its exit code (fail/success)
        # and persist the final logs. The whole entrypoint should exit
        # with the exit code (c) of the task.
        #
        # Note that if step_expr OOMs, this tail expression is never executed.
        # We lose the last logs in this scenario (although they are visible
        # still through AWS CloudWatch console).
        cmd_str += "c=$?; %s; exit $c" % BASH_SAVE_LOGS
        new_cmd_str = 'bash -c "%s"' % cmd_str
        new_cmd_str = shlex.quote(new_cmd_str)
        return shlex.split(new_cmd_str)
    

Last four lines ^ from /plugins/aws/batch/batch.py

Happy fixing

[mormoassaf]

My apologies there was extra quote in my config