coin-or / coinbrew

running:

./coinbrew fetch Ipopt:releases/3.14.12 --no-prompt

on RHEL7 clones Ipopt successfully but on getting dependencies gives:

awk: cmd. line:1: /^stable\\/2.0{print;}
awk: cmd. line:1:               ^ syntax error

It works fine on RHEL8.

RHEL7 bash version: 4.2.46
RHEL7 awk version: 4.0.2
RHEL8 bash version: 4.4.20
RHEL8 awk version: 4.2.1

Is this with the most recent coinbrew? I can't really tell where in coinbrew this would be arising or easily replicate the failure. It doesn't look like you are using the most recent version of coinbrew. Can you run coinbrew with --debug and attach the output as a file?

Yes, this is the most recent version of coinbrew. The error comes from

coinbrew/coinbrew

Line 1235 in 55cda7f

cut -d '/' -f 4 | awk "/^${version_num/\//\\\/}/{print;}" | \

Here's the log: log.txt

Another test which might be useful:

version_num=stable/2.0
echo "/^${version_num/\//\\\/}/{print;}"

RHEL8 /^stable\/2.0/{print;}
RHEL7 /^stable\\/2.0/{print;}

So I think the issue is differences in the shell.

In discussing with others it has suggested it should be:

"/^${version_num/\//\\/}/{print;}"`

rather than

"/^${version_num/\//\\\/}/{print;}"

For the test above, this gives consistent results with the original RHEL8 result on both RHEL7 and RHEL8. However, I'm not sure it handles the possible values of version_num. As an aside, few people also said that is not the "proper" way to use variables with awk, but this is out of my wheelhouse.

Ah, OK, thanks for running this down. I'm honestly not an expert shell scripter, so I'm not surprised if I did something that is not "proper" :). What that chunk of code is doing is finding the latest release corresponding to a given stable version. So in principle, version_num always be of the form stable/x.y or just x.y. It's getting just the x.y part and looking for a release tag releases/x.y.z with max value of z. So I guess your fix should be fine.

@svigerske, you are more of an expert than me, what do you think?

Maybe it's a problem with escaping the slash in the regular expression for awk (which is enclosed in slashes). I sometimes have the impression that this works different between different awk variants.

What about changing to grep?

grep "^${version_num}"

I might be missing something, but I'm not sure how grep would work. If you're confident it's always stable/x.y or x.y then how about:

   version_postfix=${version_num: -3}
   tmp=$(echo "$latest_release" | fgrep releases | \
                  fgrep -v -e "^{}" | \
                  cut -d '/' -f 4 | awk "/^${version_postfix}/{print;}" | \
                  sort -nr -t. -k1,1 -k2,2 -k3,3 | head -1)

I understood awk "/^${version_num/\//\\\/}/{print;}" as "print every line starting with $version_num". The /\//\\/ is to escape forward slashes in $version_num. I would think you get the same via grep, with the advantage that it doesn't need forward slashes to be escaped.

A ${version_num: -3} gives the last three characters from ${version_num}. I'm not sure that he wants the stable/ to be removed. Maybe, but then there would also be a problem if x > 9 or y > 9.

That's a good point about x,y > 9. Putting that issue aside, I was going off @tkralphs comment "It's getting just the x.y part..." - which I think makes sense as the input to awk is e.g.:

1.4.3
1.4.4
2.0.0
2.0.1

so you then want to match the 2.0 in stable/2.0 which gives you:

2.0.0
2.0.1

and then finally 2.0.1 which seems to match the intent given by @tkralphs .

This perhaps indicates that it's not just a syntax error on RHEL7, but that the script is broken on RHEL8 (and others) because for ThirdParty/ASL on RHEL8 you get No release found, using version stable/2.0...but ThirdParty/ASL DOES have a tag releases/2.0.0

But yeah - maybe we can nail down the intent and THEN propose solutions :)

maybe it's a bug in version_num...the name implies it's just the number

OK, sorry for this. I was a bit too hasty in my earlier answer. Now that I've taken a closer look, yes, it was a bug in version_num. Indeed, it should be a number. I was not properly stripping off stable/ when reading config.yml. It's never come up in any of the projects I manage because a while back, I switched the version number string in most of the YAML files to be either x.y or x.y.z rather than stable/x.y or releases/x.y.z but at least in this Ipopt release, it is still in the old form.

Actually, now that I dug in further, it turns out this was already reported previously in #68! But the fix that we came up with then was exactly what caused the syntax error here and that fix didn't address the root cause. After the previous fix (and on @rconde01's RHEL8 machine apparently), it seems that the stable branches of all dependencies would be checked out instead of getting the latest release, which was the actual intent. So the error was fixed, but the script was still not working as intended. At that time, I think I didn't notice this because it just so happened that ASL 2.0 had no releases and so getting the stable version was proper behavior and there was a fix that actually ensured this would happen (see 112519). So all of this combined to make this bug slip under the radar.

In any case, I was a bit incorrect about what the code snippet that actually errored was doing. Indeed, version_num was supposed to be a number, so all it was actually doing was taking the list of tags and filtering it down to ones of the form releases/x.y.*, where x.y is the version_num, then sorting to find the latest.

So with all that said, I don't actually think we do need this

/^${version_num/\//\\\/}/{print;}

It seems that

/^${version_num}/{print;}

should be fine and this is what was originally there, actually. @svigerske suggested adding /\//\\\ as a fix for #68, but I take the full blame for not putting my finger on exactly what was happening at that time.

OK, sorry for this. I was a bit too hasty in my earlier answer. Now that I've taken a closer look, yes, it was a bug in version_num. Indeed, it should be a number. I was not properly stripping off stable/ when reading config.yml. It's never come up in any of the projects I manage because a while back, I switched the version number string in most of the YAML files to be either x.y or x.y.z rather than stable/x.y or releases/x.y.z but at least in this Ipopt release, it is still in the old form.

So I should remove the stable/ in https://github.com/coin-or/Ipopt/blob/stable/3.14/.coin-or/config.yml ?

@svigerske suggested adding /\//\\\ as a fix for #68, but I take the full blame for not putting my finger on exactly what was happening at that time.

It seems to be just a year ago, but I have no memory of this... :(

So I should remove the stable/ in https://github.com/coin-or/Ipopt/blob/stable/3.14/.coin-or/config.yml ?

Yes, the stable/ is not needed and I personally think it's better to just use the actual version number there. If it's two digits, it's automatically interpreted as a stable branch. The script can handle either case, though.

It seems to be just a year ago, but I have no memory of this... :(

I know how you feel :(. I looked back to see how that regex got added in the first place, since it doesn't look like something I would concoct, and was really surprised to see that we had already tried to fix this bug once :).

By the way @rconde01 , the syntax you are using for coinbrew is deprecated and you don't need --no-prompt anymore if you specify a project and version. So you should now just do

./coinbrew fetch Ipopt@3.14.12

awk syntax error on RHEL7