seporaitis / yum-s3-iam

Yum package manager plugin for private S3 repositories. Uses Amazon IAM & EC2 Roles.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Important Update From Author - Jun 5th, 2020

seporaitis opened this issue · comments

Hello everyone, author of the plugin would like to make an announcement.

This project has not been updated in a while. I have stopped maintaining it in 2014/15 (Issue 38). After that a few excellent open source maintainers took over and kept this project up-to-date until late 2017. I think mostly thanks to them is why this project surprisingly is alive, still in use and even finds new users. I don't know the reasons for stopping maintenance after 2017, but I am sure there is a valid one and I appreciate their time maintaining project that long So what is this about?

Lately, I had a few inquiries about this project and since there had not been any releases, I intend to come back "from retirement" and bring this project up-to-date. I haven't worked with CentOS flavor Linux for a while (6 years), but in the coming weeks I plan to:

  1. Refamiliarize myself with the plugin code and it's usage.
  2. Go through issues open and closed and understand the use-cases this project was part of.
  3. Go through pull requests, evaluate and merge them.

Overall my goal is to:

  1. Bring the code up-to-date (e.g. switch to boto3 if possible; here - #78 is very helpful contribution).
  2. Setup a visible CI pipeline so that it is easier to contribute and test changes.
  3. Update the documentation with concrete examples.

Understandably, I don't want to accidentally break someone's use-case unannounced - I will be deliberately slow at first. If anyone wants to help and steer me and point out potential issues - I would really appreciate that in a form of a comment or code review. Hopefully, together we all can continue keeping this plugin enjoyable to work with.

Welcome back @seporaitis,

One of the major reason of the slowdown on my part is that I lost access to a proper test platform. Several PRs unfortunately were not backward compatible (e.g. #69 inverses the priority of configuration vs environment variables, #78 adds the dependency to boto3, etc..).

A few notes:

  • Regarding the use of boto3, one of the advantages of the yum-s3-iam module for its users is that it does not have dependencies that are not typically part of the standard distributions. At the time RHEL6/CentOS6 did not have a boto3 package (and making one was not easy because of the dependencies). It's certainly an option, but it would limit the target to distributions that have a boto3 package (or put a significant burden on users of those that don't).
  • Trying to support too many syntaxes for the URLs has made some parts of the code too complicated. It's not an easy problem either (hostname validation, user-friendliness, etc.).
  • As you have probably seen, we relied on a blog entry as a tutorial, and that blog was deleted.

Thank you for the detailed answer @mbrossard it is very helpful.

Re: testing environment. I have set-up a basic GitHub Workflow that builds the RPM. It is my first GW, and just runs make rpm on CentOS 6/7. Probably it can be improved further (I saw a mention of rpmlint while reacquainting myself with rpm/yum). Next week/end I'll look for setting up a testing environment that could run the tests and maybe even could do integration testing. My thoughts are towards setting up a separate AWS account under my personal organization, with some limited credentials and have secrets set-up in this repo, so that various cases could be tested using GitHub Workflow. S3 buckets are free and the price for a few megabytes - negligible. How did the previous testing platform look/work?

Re: boto3 - very good point. My thoughts now are somewhere between:

  • vendoring full or part of boto, responsible for credentials;
  • re-implement a simple version of credential retrieval, so that it matches what people expect from boto3;
  • provide an option to plug-in boto3 as needed;
  • keep it as is, but document the limitations and/or workarounds for various use-cases.

All carry trade-offs, so before I change anything I will try to establish a somewhat good enough CI setup.