This Action does not work with extracted providers

Question

This Action does not work with extracted providers

solvaholic opened this issue 3 years ago · comments

Description

Providers are being extracted from octodns/octodns to their own repositories in the octodns org: octodns/octodns#622

Currently, this Action does not work with providers that have already been extracted. That means, for example, this Action cannot be used with octodns/octodns at the default branch.

Expected Behavior

This Action should Just Work with octodns: If an extracted provider is used in the config, then install it.

Actual Behavior

If the provider has been installed by other means, this Action continues to function. If not, then an error probably similar to:

2022-01-11T20:48:20  [140712345147200] WARNING Route53 octodns_route53 shimmed. Update your provider class to octodns_route53.Route53Provider. Shim will be removed in 1.0
2022-01-11T20:48:20  [140712345147200] ERROR Route53 Route53Provider has been moved into a seperate module, octodns_route53 is now required. Provider class should be updated to octodns_route53.Route53Provider

Possible Fix

Could detect providers used in config, and install them.
Or install all of the extracted providers.
Or stick with v0.9.14 forever.

Steps to Reproduce

Configure octodns to use an extracted provider
Configure workflow to run this action with octodns_ref set to "master"
Run that workflow

Context

Can't use the latest octodns/octodns with this Action.

This workflow definition failed because I didn't install octodns_route53:

      - name: Run `octodns-sync` with ${{ needs.meta.outputs.config }}
        id: octodns-sync
        uses: solvaholic/octodns-sync@main
        with:
          config_path: ${{ needs.meta.outputs.config }}
          octodns_ref: master

This workflow definition ran successfully:

      - name: Run `octodns-sync` with ${{ needs.meta.outputs.config }}
        id: octodns-sync
        uses: solvaholic/octodns-sync@main
        with:
          config_path: ${{ needs.meta.outputs.config }}
          octodns_ref: v0.9.14

Your Environment

Version used: solvaholic/octonds-sync@main, octodns/octodns at master
Link to your project: https://github.com/solvahol/dns

Ingo Struck · Answer 1 · Tue Jan 18 2022 06:41:02 GMT+0800 (China Standard Time)

I took a very superficial look at one repo that uses this action (https://github.com/solvahol/dns). Up to commit 536c0bb550e3c527c59340751d5f0cf0ed33a946 you had a requirements.txt in place. I could not determine offhand what mechanism replaced it. My expectation would be that you (as well as the vast majority of users) have some python package management in place that handles the requirements and installation step, given that you don't change providers very often in workflows like that.

imho this would be the proper place to address your issue (in a "package install" or "dependency solution" step) rather than expecting that some package auto-installs some other packages just due to input in the payload (the configuration file).
For me, requirements.txt would be the perfect place to put the newly created OctoDNS Provider requirements alongside the probably pinned version of octodns itself, rather than to bury it deep down in the workflow scripts.

Personally, I would always refrain from any "auto-install-something-based-on-input-data" pattern for the sake of security, safety and readability. ymmv.

(edit / just to clarify: I came here from the link at octodns/octodns#622 and was curious where OctoDNS users are affected by the split, and if so -- how and why)

Roger D. Winans · Answer 2 · Sun Jan 23 2022 09:35:25 GMT+0800 (China Standard Time)

Thank you @istr 🙇

For me, requirements.txt would be the perfect place to put the newly created OctoDNS Provider requirements alongside the probably pinned version of octodns itself, rather than to bury it deep down in the workflow scripts.

I appreciate you taking the time to look into this and share your recommendation. I've been thinking of installing OctoDNS and installing provider dependencies as part of this Action, and now I see they needn't be.

Removing those to a step in the user's workflow would simplify their control of the environment, and their dependency and vulnerability management. And it'd remove complexity from this project. While eliminating this issue. 👍

Martin Simon · Answer 3 · Mon Jun 13 2022 15:09:08 GMT+0800 (China Standard Time)

The proposed solution of pinning octodns to v0.9.14 doesn't seem to work anymore:

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

jobs:
  octodns-sync:
    name: Run octodns-sync plan
    runs-on: ubuntu-latest
    outputs:
      plan: ${{ steps.generate-plan.outputs.plan }}
    steps:
      - uses: actions/checkout@v3
      - uses: solvaholic/octodns-sync@v2.3.0
        id: generate-plan
        with:
          config_path: config.yaml
          octodns_ref: v0.9.14

fails with the following error:

INFO: Cleaning up plan and log files if they already exist
INFO: _config_path: config.yaml
Script started, file is /home/runner/work/zonefiles/zonefiles/octodns-sync.log
2022-06-13T07:02:26  [139820823603008] INFO  Manager __init__: config_file=config.yaml
2022-06-13T07:02:26  [139820823603008] INFO  Manager __init__:   max_workers=1
2022-06-13T07:02:26  [139820823603008] INFO  Manager __init__:   include_meta=False
2022-06-13T07:02:26  [139820823603008] ERROR Manager _get_{}_class: Unable to import module octodns_route53.Route53Provider
Traceback (most recent call last):
  File "/home/runner/.local/lib/python3.8/site-packages/octodns/manager.py", line 190, in _get_named_class
    module = import_module(module_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'octodns_route53'
Traceback (most recent call last):
  File "/home/runner/.local/lib/python3.8/site-packages/octodns/manager.py", line 190, in _get_named_class
    module = import_module(module_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'octodns_route53'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/runner/.local/bin/octodns-sync", line 8, in <module>
    sys.exit(main())
  File "/home/runner/.local/lib/python3.8/site-packages/octodns/cmds/sync.py", line 37, in main
    manager = Manager(args.config_file)
  File "/home/runner/.local/lib/python3.8/site-packages/octodns/manager.py", line 115, in __init__
    _class = self._get_named_class('provider', _class)
  File "/home/runner/.local/lib/python3.8/site-packages/octodns/manager.py", line 194, in _get_named_class
    raise ManagerException(f'Unknown {_type} class: {_class}')
octodns.manager.ManagerException: Unknown provider class: octodns_route53.Route53Provider
Script done, file is /home/runner/work/zonefiles/zonefiles/octodns-sync.log
FAIL: octodns-sync exited with an error.
Error: Process completed with exit code 1.

Any suggestions on how this could be fixed, at least on a temporary basis?

EDIT: this has been temporarily fixed by adding the following to a requirements.txt file:

octodns==0.9.17
octodns_route53==0.0.4

and the following step to the job:

      - uses: actions/setup-python@v2
        with:
          python-version: '3.10'
      - run: pip install -r requirements.txt

Roger D. Winans · Answer 4 · Wed Jun 15 2022 20:37:25 GMT+0800 (China Standard Time)

@barnumbirr Thank you for writing this up and describing workaround you used 🙇

I'll make time this coming weekend (18-19 Jun) to fix this in the octodns-sync Action and its documentation.

Do you think these steps would suffice?:

Have the Action error out when differing octodns versions are indicated, for example in workflow definition, environment variables, and requirements.txt
When the has been moved into a seperate module message occurs, log a link to documented solution
Instruct octodns-sync Action users, in documentation, to specify octodns and provider versions in requirements.txt

Martin Simon · Answer 5 · Wed Jun 15 2022 21:44:36 GMT+0800 (China Standard Time)

Have the Action error out when differing octodns versions are indicated, for example in workflow definition, environment variables, and requirements.txt

I'd prefer a solution in which package requirements don't have to be redefined multiple times. With the temporary solution I've described above, solvaholic/octodns-sync@v2.3.0 tries to install octodns v0.9.12, which does nothing as octodns v0.9.17 was already installed through requirements.txt.
A single "source of truth" would make more sense.

When the has been moved into a seperate module message occurs, log a link to documented solution

That would be fine.

Instruct octodns-sync Action users, in documentation, to specify octodns and provider versions in requirements.txt

Perfect!

Roger D. Winans · Answer 6 · Sun Jun 19 2022 04:58:12 GMT+0800 (China Standard Time)

👋 I pushed changes to the issue86 branch for this. They're up for review in #89

If you'd like to test the issue86 branch, configure your workflow job step to use it:

        uses: solvaholic/octodns-sync@issue86

To see a test already set up, check out the validate and deploy workflows in solvahol/dns and solvahol/dns#35.

I plan to get these changes, or improved versions, in to main on 19 Jun.

Roger D. Winans · Answer 7 · Mon Jun 20 2022 05:14:46 GMT+0800 (China Standard Time)

👋 Merging #89 closed this issue. It removed octodns installation to a workflow step.

The user's workflow can install octodns and required providers any way the user likes. Example workflows in documentation suggest installing from requirements.txt, as @barnumbirr described in #86 (comment).

If you have questions about this, or continue to see problems, please re-open this issue or create a new one 🙇

Martin Simon · Answer 8 · Mon Jun 20 2022 16:06:59 GMT+0800 (China Standard Time)

@solvaholic thanks a bunch. Limitied testing on my end seems to confirm this works as expected 🎉 .
Is a new release of this action planned at some point? I'd prefer to be able to used a defined version in my workflows rather than have to use @main.

Roger D. Winans · Answer 9 · Mon Jun 20 2022 22:43:54 GMT+0800 (China Standard Time)

Thanks for following up and letting me know @barnumbirr!

Is a new release of this action planned at some point? I'd prefer to be able to used a defined version in my workflows rather than have to use @main.

Yes! I've messed up releases in this repo, and I think there are some feedback to integrate, so I want to properly get my head into it.

I think this'll become 3.0 on 21 Jun. I feel silly going up to 3.x but the changes for this will break things for users who specify octodns_ref for this Action, either in the Action input or through environment variables.