Giters

FreshPorts / freshports

The website part of FreshPorts

Home Page:http://www.freshports.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

automatically know the list of valid ABI

dlangille opened this issue · comments

commented 
          > The goal: automatically know the list of valid ABI. I like to do as little as possible.

Compiling that list from a script (/bin/sh, python, or perl) would be helpful.

That script would make a great standalone project.

This just in, via: https://lists.freebsd.org/archives/freebsd-hackers/2023-December/002733.html

# fetch https://pkg.freebsd.org/index.html
index.html                                            3606  B  103 MBps    00s

# grep FreeBSD: index.html | sed -e 's@.*\(FreeBSD:[^ <]*\).*@\1@' | sort
FreeBSD:12:aarch64
FreeBSD:12:amd64
FreeBSD:12:armv6
FreeBSD:12:armv7
FreeBSD:12:i386
FreeBSD:13:aarch64
FreeBSD:13:amd64
FreeBSD:13:armv6
FreeBSD:13:armv7
FreeBSD:13:i386
FreeBSD:13:powerpc
FreeBSD:13:powerpc64
FreeBSD:13:powerpc64le
FreeBSD:14:aarch64
FreeBSD:14:amd64
FreeBSD:14:armv6
FreeBSD:14:armv7
FreeBSD:14:i386
FreeBSD:14:powerpc
FreeBSD:14:powerpc64
FreeBSD:14:powerpc64le
FreeBSD:15:aarch64
FreeBSD:15:amd64
FreeBSD:15:armv6
FreeBSD:15:armv7
FreeBSD:15:i386
FreeBSD:15:powerpc
FreeBSD:15:powerpc64
FreeBSD:15:powerpc64le

Originally posted by @dlangille in #505 (comment)

commented

On 2023-12-31, FreeBSD 12 was removed from the list. Now we have:

[15:42 empty dan ~/tmp] % fetch https://pkg.freebsd.org/index.html
index.html                                            3068  B   37 MBps    00s
[15:42 empty dan ~/tmp] % grep FreeBSD: index.html | sed -e 's@.*\(FreeBSD:[^ <]*\).*@\1@' | sort
FreeBSD:13:aarch64
FreeBSD:13:amd64
FreeBSD:13:armv6
FreeBSD:13:armv7
FreeBSD:13:i386
FreeBSD:13:powerpc
FreeBSD:13:powerpc64
FreeBSD:13:powerpc64le
FreeBSD:14:aarch64
FreeBSD:14:amd64
FreeBSD:14:armv6
FreeBSD:14:armv7
FreeBSD:14:i386
FreeBSD:14:powerpc
FreeBSD:14:powerpc64
FreeBSD:14:powerpc64le
FreeBSD:15:aarch64
FreeBSD:15:amd64
FreeBSD:15:armv7
FreeBSD:15:powerpc
FreeBSD:15:powerpc64
FreeBSD:15:powerpc64le
commented

Hint for future me:

construct queries from scripts. Dump it to a file. use psql -f

re: https://www.postgresql.org/docs/current/app-psql.html

commented 
[19:31 dev-ingress01 dan ~/scripts] % cat add-new-abi.sh
#!/bin/sh

ABI=$(./current_list_of_valid_abi.sh)

in_list=''
#echo $ABI
for abi in ${ABI}
do
#  echo $abi
  if [ "$in_list" == "" ]
  then
    in_list="('${abi}')"
  else
    in_list="${in_list}, ('${abi}')"
  fi
done

#echo $in_list

query="
with new_and_old as (
WITH current_abi_list as (
select name 
  from (VALUES $in_list)
   v(name))
select current_abi_list.name as name, abi.name as missing from current_abi_list left outer join abi on current_abi_list.name = abi.name) 
insert into abi (name)
select name from new_and_old where missing is null;"

echo $query

[19:31 dev-ingress01 dan ~/scripts] % ./add-new-abi.sh 
with new_and_old as ( WITH current_abi_list as ( select name from (VALUES ('FreeBSD:13:aarch64'), 
('FreeBSD:13:amd64'), ('FreeBSD:13:armv6'), ('FreeBSD:13:armv7'), ('FreeBSD:13:i386'), ('FreeBSD:13:powerpc'),
 ('FreeBSD:13:powerpc64'), ('FreeBSD:13:powerpc64le'), ('FreeBSD:14:aarch64'), ('FreeBSD:14:amd64'),
 ('FreeBSD:14:armv6'), ('FreeBSD:14:armv7'), ('FreeBSD:14:i386'), ('FreeBSD:14:powerpc'), ('FreeBSD:14:powerpc64'),
 ('FreeBSD:14:powerpc64le'), ('FreeBSD:15:aarch64'), ('FreeBSD:15:amd64'), ('FreeBSD:15:armv7'),
 ('FreeBSD:15:powerpc'), ('FreeBSD:15:powerpc64'), ('FreeBSD:15:powerpc64le')) v(name)) 
select current_abi_list.name as name, abi.name as missing
from current_abi_list left outer join abi on current_abi_list.name = abi.name) 
insert into abi (name) 
select name from new_and_old where missing is null;
commented

Running that:

freshports.dev=# begin;
BEGIN
freshports.dev=*# with new_and_old as ( WITH current_abi_list as ( select name from (VALUES ('FreeBSD:13:aarch64'), 
('FreeBSD:13:amd64'), ('FreeBSD:13:armv6'), ('FreeBSD:13:armv7'), ('FreeBSD:13:i386'), ('FreeBSD:13:powerpc'),
 ('FreeBSD:13:powerpc64'), ('FreeBSD:13:powerpc64le'), ('FreeBSD:14:aarch64'), ('FreeBSD:14:amd64'),
 ('FreeBSD:14:armv6'), ('FreeBSD:14:armv7'), ('FreeBSD:14:i386'), ('FreeBSD:14:powerpc'), ('FreeBSD:14:powerpc64'),
 ('FreeBSD:14:powerpc64le'), ('FreeBSD:15:aarch64'), ('FreeBSD:15:amd64'), ('FreeBSD:15:armv7'),
 ('FreeBSD:15:powerpc'), ('FreeBSD:15:powerpc64'), ('FreeBSD:15:powerpc64le')) v(name)) select current_abi_list.name
 as name, abi.name as missing from current_abi_list left outer join abi on current_abi_list.name = abi.name) insert into abi
 (name) select name from new_and_old where missing is null;
INSERT 0 0
freshports.dev=*# rollback;
ROLLBACK
freshports.dev=#
commented 
[19:32 dev-ingress01 dan ~/scripts] % cat ./delete-depcreated-abi.sh
#!/bin/sh

ABI=$(./current_list_of_valid_abi.sh)

in_list=''
#echo $ABI
for abi in ${ABI}
do
#  echo $abi
  if [ "$in_list" == "" ]
  then
    in_list="'${abi}'"
  else
    in_list="${in_list}, '${abi}'"
  fi
done

#echo $in_list

query="DELETE FROM abi where name not in (${in_list})"

echo $query
[19:35 dev-ingress01 dan ~/scripts] % ./delete-depcreated-abi.sh 
DELETE FROM abi where name not in ('FreeBSD:13:aarch64', 'FreeBSD:13:amd64', 'FreeBSD:13:armv6', 
'FreeBSD:13:armv7', 'FreeBSD:13:i386', 'FreeBSD:13:powerpc', 'FreeBSD:13:powerpc64', 'FreeBSD:13:powerpc64le', 
'FreeBSD:14:aarch64', 'FreeBSD:14:amd64', 'FreeBSD:14:armv6', 'FreeBSD:14:armv7', 'FreeBSD:14:i386', 
'FreeBSD:14:powerpc', 'FreeBSD:14:powerpc64', 'FreeBSD:14:powerpc64le', 'FreeBSD:15:aarch64', 'FreeBSD:15:amd64', 
'FreeBSD:15:armv7', 'FreeBSD:15:powerpc', 'FreeBSD:15:powerpc64', 'FreeBSD:15:powerpc64le')
[19:35 dev-ingress01 dan ~/scripts] %
commented

Running that, we get:

freshports.dev=*# rollback;
ROLLBACK
freshports.dev=# select * from abi order by name;
 id |          name          | active 
----+------------------------+--------
  2 | FreeBSD:13:aarch64     | t
 11 | FreeBSD:13:amd64       | t
 20 | FreeBSD:13:armv6       | t
 21 | FreeBSD:13:armv7       | t
 10 | FreeBSD:13:i386        | t
 23 | FreeBSD:13:mips64      | t
 58 | FreeBSD:13:powerpc     | t
 24 | FreeBSD:13:powerpc64   | t
 57 | FreeBSD:13:powerpc64le | t
 34 | FreeBSD:13:riscv64     | t
 26 | FreeBSD:14:aarch64     | t
 27 | FreeBSD:14:amd64       | t
 28 | FreeBSD:14:armv6       | t
 29 | FreeBSD:14:armv7       | t
 30 | FreeBSD:14:i386        | t
 32 | FreeBSD:14:mips64      | t
 59 | FreeBSD:14:powerpc     | t
 33 | FreeBSD:14:powerpc64   | t
 56 | FreeBSD:14:powerpc64le | t
 35 | FreeBSD:14:riscv64     | t
 54 | FreeBSD:15:aarch64     | t
 47 | FreeBSD:15:amd64       | t
 48 | FreeBSD:15:armv6       | t
 49 | FreeBSD:15:armv7       | t
 50 | FreeBSD:15:i386        | t
 51 | FreeBSD:15:mips64      | t
 60 | FreeBSD:15:powerpc     | t
 52 | FreeBSD:15:powerpc64   | t
 55 | FreeBSD:15:powerpc64le | t
 53 | FreeBSD:15:riscv64     | t
(30 rows)

freshports.dev=# begin;
BEGIN
freshports.dev=*# DELETE FROM abi where name not in ('FreeBSD:13:aarch64', 'FreeBSD:13:amd64', 'FreeBSD:13:armv6', 
'FreeBSD:13:armv7', 'FreeBSD:13:i386', 'FreeBSD:13:powerpc', 'FreeBSD:13:powerpc64', 'FreeBSD:13:powerpc64le', 
'FreeBSD:14:aarch64', 'FreeBSD:14:amd64', 'FreeBSD:14:armv6', 'FreeBSD:14:armv7', 'FreeBSD:14:i386', 
'FreeBSD:14:powerpc', 'FreeBSD:14:powerpc64', 'FreeBSD:14:powerpc64le', 'FreeBSD:15:aarch64', 'FreeBSD:15:amd64', 
'FreeBSD:15:armv7', 'FreeBSD:15:powerpc', 'FreeBSD:15:powerpc64', 'FreeBSD:15:powerpc64le')
freshports.dev-*# ;
DELETE 8
freshports.dev=*# select * from abi order by name;
 id |          name          | active 
----+------------------------+--------
  2 | FreeBSD:13:aarch64     | t
 11 | FreeBSD:13:amd64       | t
 20 | FreeBSD:13:armv6       | t
 21 | FreeBSD:13:armv7       | t
 10 | FreeBSD:13:i386        | t
 58 | FreeBSD:13:powerpc     | t
 24 | FreeBSD:13:powerpc64   | t
 57 | FreeBSD:13:powerpc64le | t
 26 | FreeBSD:14:aarch64     | t
 27 | FreeBSD:14:amd64       | t
 28 | FreeBSD:14:armv6       | t
 29 | FreeBSD:14:armv7       | t
 30 | FreeBSD:14:i386        | t
 59 | FreeBSD:14:powerpc     | t
 33 | FreeBSD:14:powerpc64   | t
 56 | FreeBSD:14:powerpc64le | t
 54 | FreeBSD:15:aarch64     | t
 47 | FreeBSD:15:amd64       | t
 49 | FreeBSD:15:armv7       | t
 60 | FreeBSD:15:powerpc     | t
 52 | FreeBSD:15:powerpc64   | t
 55 | FreeBSD:15:powerpc64le | t
(22 rows)

freshports.dev=*# rollback;
ROLLBACK
freshports.dev=#
commented

Here we go. I amended the scripts above to take the list of ABI from a file. Then output full SQL.

The result is this script, which does the refresh:

[15:48 dev-ingress01 dan ~/scripts] % cat refresh-abi.sh  
#!/bin/sh

# Ensure the ABI table contains the right list of values.

if [ ! -f config.sh ]
then
	echo "config.sh not found by missing-port-categories.sh..."
	exit 1
fi

. config.sh

export PGDATABASE=$DB
export PGHOST=$HOST
export PGUSER=$DBUSER_ABI

# see also PGSSLMODE and PGSSLROOTCERT in config.sh


valid=$(mktemp  ${SPOOLINGDIR}/abi-valid.XXXXXX)
sql=$(mktemp ${SPOOLINGDIR}/abi-sel.XXXXXX)

# fetch and extract the list of valid ABI
./current-list-of-valid-abi.sh > $valid

# build the SQL commands
echo 'BEGIN;'                           >  $sql

# ABI to be deleted
./delete-depcreated-abi.sh     < $valid >> $sql

# ABI to be added
echo                                    >> $sql
./add-new-abi.sh               < $valid >> $sql

# and we end
echo                                    >> $sql
echo 'ROLLBACK;'                          >>  $sql

# run the SQL
psql -f $sql 


# if all good, remove the files. If not, leave the files
# the output of this script will be saved and an error flagged
if [ $? == 0 ]
then
  # commit taken
  rm $valid $sql
else
  # raise an error, somewhere, somehow
fi
[15:48 dev-ingress01 dan ~/scripts] %
commented

This is now ready to run overnight on dev.

commented

From /var/log/freshports/freshports.log when there is no change in ABI:

Jan  7 13:58:00 dev-ingress01 FreshPorts[7047]: into /usr/local/etc/periodic/daily/330.refresh-abi
Jan  7 13:58:00 dev-ingress01 dan[7056]: ./refresh-abi.sh starts
Jan  7 13:58:00 dev-ingress01 dan[7082]: ./refresh-abi.sh nothing to change
Jan  7 13:58:00 dev-ingress01 dan[7088]: ./refresh-abi.sh finishes

When an ABI goes away:

Jan  7 13:58:18 dev-ingress01 FreshPorts[7669]: into /usr/local/etc/periodic/daily/330.refresh-abi
Jan  7 13:58:18 dev-ingress01 dan[7679]: ./refresh-abi.sh starts
Jan  7 13:58:18 dev-ingress01 dan[7699]: Sun Jan 7 13:58:18 UTC 2024
Jan  7 13:58:18 dev-ingress01 dan[7704]: BEGIN DELETE 1 INSERT 0 0 COMMIT
Jan  7 13:58:18 dev-ingress01 dan[7711]: ./refresh-abi.sh finishes

You would also find this entry in /var/log/freshports/abi.log which is rotated very slowly based on size.

Sun Jan 7 13:58:18 UTC 2024
BEGIN
DELETE 1
INSERT 0 0
COMMIT
commented

now packaged.

Installed on test, dev, and stage.