Giters

myoung34 / docker-github-actions-runner

This will run the new self-hosted github actions runners with docker-in-docker

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Runner stopped working after downgrade to 2.285.3

joeyparrish opened this issue · comments

commented

The latest tag on Docker Hub was changed from 2.301.1 to 2.285.3, resulting in all my runner services failing with the message "Runner version v2.285.3 is deprecated and cannot receive messages."

It looks like the automated "bump" in b1535c3 was actually a downgrade.

I switched my services to use myoung34/github-runner:2.301.1 instead of myoung34/github-runner:latest to get my services working again, but I would like to switch back to latest when this is fixed.

Can your automation be fixed to only increase version numbers?

commented

I don't have a way off hand to tell if semver is higher or lower but that would be an ideal fix. Id also add to https://github.com/orgs/community/discussions/12225 because they seem to do this often

commented

I hard reset to 2.301.1 and this will fix itself. Its frustrating for sure on both sides of this

commented

I don't have a way off hand to tell if semver is higher or lower but that would be an ideal fix.

May I offer to help with that? Show me the code that needs to judge that, and I'll offer a PR. It's not difficult to parse and compare semvers. I've written it in several languages before.

commented

Here's something in Python that takes a string semver and produces a sortable tuple, based on this code from one of my projects:

def semver_to_tuple(semver):
  main_version, _, suffix = semver.partition('-')
  if not suffix:
    # this puts main releases (no tag) after prerelease versions (any tag)
    suffix = '}}}'
  version_tuple = [int(x) for x in main_version.split('.')]
  return version_tuple + [suffix]

print(semver_to_tuple('1.2.3') > semver_to_tuple('9.8.7'))  # False

It's pretty easy to do the same in many other languages. If you're writing in JavaScript, there's a module called semver:

const semver = require('semver');
semver.gt('1.2.3', '9.8.7'); // false

If your automation is private, just tell me the language, and I'll provide you a working snippet. (Unless it's PHP. You're SOL with PHP. 😁)

commented

Automation is zapier, not private but not shared
If its python it works but their bundling of python doesnt have a lot of extras

commented

This? https://zapier.com/

Their service commits directly to your repo with your PAT?

commented

Correct
The JS sandbox does not have semver fwiw Cannot find module 'semver'

commented

Would you accept a PR to replace that with an Actions workflow on a schedule? Output to PR or directly commit with PAT?

commented

If there's a way to automate based on upstream releases sure. At the time that wasnt a thing

The zap basically responds to actions/runner releases, parses the tag off the API and does a bump via python

that code:

import requests
import re
import json
import base64
from datetime import datetime

tag = re.sub(
    r'^v',
    '',
    input_data['tag'],
)
token = input.get('github_token')
user = input.get('github_user')

req = requests.get(
    'https://api.github.com/repos/myoung34/docker-github-actions-runner/contents/Dockerfile?ref=master',
    headers={ 'Accept': 'application/vnd.github.v3+json'},
    auth=(user, token)
)
req.raise_for_status()
new_contents = re.sub(
    r'GH_RUNNER_VERSION="[0-9\.]+"',
    'GH_RUNNER_VERSION="{}"'.format(tag),
    base64.b64decode(req.json()['content']).decode("utf-8")
)

payload_body = {
    "message": "[Automated :robot: ] Bump to version {}".format(tag),
    "committer": {
      "name": "Marcus Young",
      "email": "myoung34@my.apsu.edu"
    },
    "sha": req.json()['sha'],
    "content": base64.b64encode(new_contents.encode('ascii')).decode('utf-8')
}

_req = requests.put(
    'https://api.github.com/repos/myoung34/docker-github-actions-runner/contents/Dockerfile',
    headers={ 'Accept': 'application/vnd.github.v3+json'},
    auth=(user, token),
    data=json.dumps(payload_body).encode("utf-8")
)
_req.raise_for_status()

new_sha = _req.json()['commit']['sha']

tag_body = {
  "tag": tag,
  "message": "[Automated :robot: ] Bump to version {}".format(tag),
  "object": new_sha,
  "type": "commit",
  "tagger": {
    "name": "Marcus Young",
    "email": "myoung34@my.apsu.edu",
  }
}
_tag_body = {
  "ref": "refs/tags/{}".format(tag),
  "sha": new_sha
}

_tag_req = requests.post(
    'https://api.github.com/repos/myoung34/docker-github-actions-runner/git/refs',
    headers={ 'Accept': 'application/vnd.github.v3+json'},
    auth=(user, token),
    data=json.dumps(_tag_body).encode("utf-8")
)

_tag_req.raise_for_status()
commented

I think it can be done easily enough with the GitHub API. I'll see what I can do in a fork, and I'll let you know.

commented

I updated above with the code that does the bump itself if its helpful

commented

Id like to not do a schedule if possible, i like to know that it responds directly to a release upstream but not sure if its possible with only GHA

commented

What is the trigger in Zapier? Are they not polling at some interval?

commented

No, but I can't say because the magic sauce is NDA (im an ex-employee)

commented

Okay, then if you want to keep using Zapier, I would suggest where you have this:

new_contents = re.sub(
    r'GH_RUNNER_VERSION="[0-9\.]+"',
    'GH_RUNNER_VERSION="{}"'.format(tag),
    base64.b64decode(req.json()['content']).decode("utf-8")
)

You first extract the old GH_RUNNER_VERSION, parse both that and tag with the parser I offered, and compare the resulting tuples. If tag <= old_version, abort the process. Would that work?

commented

I can do that and store the previous but semver gets a bit complicated when it comes to comparisons

That said they always bump above, so i could remove the dots (2.302.1 making it 23021) and do a raw int compare unless you can think of a side effect?

ie: 2.285.1 to 2.301.2 = 22851 < 23012 -> bump
2.301.2 to 2.293.1 = 23012 < 22931 (fail) -> no bump

commented

semver gets a bit complicated when it comes to comparisons

In what way? There are three integers separated by dots, followed optionally by a hyphen and a tag. In python, convert that to a list or tuple like [2, 285, 1], then compare those. [2, 285, 1] < [2, 301, 2]. Since lists and tuples are comparable in python, that's an easy, safe choice.

Removing the dots and parsing the whole string as a single int will go wrong when they change the number of digits in any one part. For example, 2.301.10 vs 2.302.0 will give you the wrong answer if you use this method.

The only possible complexity is dealing with tags. For example, we have had versions of our software with names like 3.0.0-beta and 3.0.0-beta2. Both of those should sort before 3.0.0, which is why my code snippet above filled in a "default" tag of "}}}", which sorts after any alphanumeric ASCII.

If you don't expect GitHub runner versions to have betas or other tags, you don't need that. Just split, map to ints, and return a list:

  def parse_semver(semver):
    return [int(x) for x in semver.split('.')]

Those are directly comparable and will work no matter how many digits are in each part of the version number.

commented

I had assumed that but I don't think I've seen it. I'll get something in tonight to try and prevent this going forward until they have a definite fix

commented

I hard reset to 2.301.1 and this will fix itself. Its frustrating for sure on both sides of this

Hi, it seems this was still pointing to 2.285.3 30 min ago or so. At least from my experience.

I just re-ran a docker pull and now docker history shows this was reset 30 min ago or so: https://gist.github.com/chantra/3e609ceb7ac013bde5167c20e7ba4305

So people may have experience the issue between your first comment mentioning you had reset the latest tag and now.

Also https://github.com/myoung34/docker-github-actions-runner/releases still show 2.285.3 as latest, not sure if it really matters though.

commented

I hard reset to 2.301.1 and this will fix itself. Its frustrating for sure on both sides of this

Hi, it seems this was still pointing to 2.285.3 30 min ago or so. At least from my experience.

I just re-ran a docker pull and now docker history shows this was reset 30 min ago or so: https://gist.github.com/chantra/3e609ceb7ac013bde5167c20e7ba4305

So people may have experience the issue between your first comment mentioning you had reset the latest tag and now.

Also https://github.com/myoung34/docker-github-actions-runner/releases still show 2.285.3 as latest, not sure if it really matters though.

Ill take a look, it should have updated to 2.301.1

commented

@chantra mustve been a race condition on your pull. the force reset yesterday worked:

docker pull myoung34/github-runner:latest
docker run \
  -e REPO_URL="https://github.com/octokode/test1" \
  -e RUNNER_TOKEN="redact" \
  -v /var/run/docker.sock:/var/run/docker.sock \
  myoung34/github-runner:latest

√ Connected to GitHub

Current runner version: '2.301.1'
2023-01-31 13:40:24Z: Listening for Jobs
commented

OK all: anyone that comes and sees this thread it should be a problem of the past.

@joeyparrish I added a filter in zapier:

import requests
import re

new_tag = re.sub(
    r'^v|\.',
    '', 
    input_data['tag'],
)

req = requests.get(
    'https://api.github.com/repos/myoung34/docker-github-actions-runner/releases/latest',
    headers={ 'Accept': 'application/vnd.github.v3+json'},
)
req.raise_for_status()

my_latest_tag = re.sub(
    r'^v|\.',
    '', 
    req.json()['tag_name'],
)

print(f'actions tag: {new_tag}')
print(f'my latest tag: {my_latest_tag}')

return new_tag > my_latest_tag

Theres now a filter that depends on that return being True early on so Ill still get internal notifications when actions releases a "new" version but unless its > than what my latest tag is via the /releases/latest tag_name property it wont do anything

commented

That's great, thanks! But since you are comparing strings, "2.301.2" > "2.301.10", which is not what we want.

If you parse the string into a list of ints, and compare lists, you can make your function immune to changes in the number of digits. If there are no suffixes (like -beta), and the versions are purely numeric, this is exceedingly simple:

def parse_semver(semver):
  return list(semver.split('.').map(int))

Or:

def parse_semver(semver):
  return [int(part) for part in semver.split('.')]

Or for a full semver with suffixes, if you want to be more thorough:

def parse_semver(semver):
  main_version, _, suffix = semver.partition('-')
  if not suffix:
    # this puts main releases (no suffix) after prerelease versions (any suffix)
    suffix = '}}}'
  version_tuple = list(main_version.split('.').map(int))
  return version_tuple + [suffix]

The returned lists can be directly compared and are immune to the string comparison problem.

commented

Good catch, will fix. Thanks!

commented

Thanks for quickly fixing this and following up with action items so it does not re-occur BTW.

@chantra mustve been a race condition on your pull. the force reset yesterday worked:

Probably. When this workflow ran (around 4:30 PST):
https://github.com/kernel-patches/runner/actions/runs/4049450958/jobs/6965829616#step:9:154

latest was:

#5 [1/2] FROM docker.io/myoung34/github-runner:2.301.1@sha256:0887abe24ad091c9aea703a5b82eb71790f0c3d0b6b48cb3d95ce04b065a68a8

When latest was tagged at 5:06 PST: https://hub.docker.com/layers/myoung34/github-runner/latest/images/sha256-bdbfbc12b908417ece377959968d716639d157dda351ebac92f60c29fae4dc29?context=explore .

My comment was that in #277 (comment) (2:43 PST), you said you hard reset to 2.301.1 but as far as I can tell, this was not fixed till 5:06 PST (or maybe it flip-flopped during that time), so people may have seen this issue as closed and fixed, but yet may still have experienced the problem during that time window.

docker pull myoung34/github-runner:latest
docker run \
  -e REPO_URL="https://github.com/octokode/test1" \
  -e RUNNER_TOKEN="redact" \
  -v /var/run/docker.sock:/var/run/docker.sock \
  myoung34/github-runner:latest

√ Connected to GitHub

Current runner version: '2.301.1'
2023-01-31 13:40:24Z: Listening for Jobs
commented

@chantra are you good to go with latest at this point ? want to make sure everything is working as expected

commented

From a docker pull and docker history, it seems everything is fine. I can report back after I remove the hardcoded 2.301.1 pinning. Thanks

commented

From a docker pull and docker history, it seems everything is fine. I can report back after I remove the hardcoded 2.301.1 pinning. Thanks

ok, confirmed that the newly generated containers based of latest are running 2.301.1. Thanks again.