Cannot run older Trino versions with chart version trino-0.14.0
david-streamlio opened this issue · comments
I am a committer on the Apache Pulsar project, and we have created a connector based on the 368 version of Trino. Now we are unable to deploy it using the latest version of the Trino helm chart.
The Trino Helm Version
helm list -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
trino-cluster trino 1 2023-12-12 09:09:13.0732119 -0800 PST deployed trino-0.14.0 432
Steps to Reproduce
helm install -f 368.yaml $RELEASE_NAME trino/trino --namespace $TRINO_K8S_NS --create-namespace
Contents of the 368.yaml file:
image:
tag: "368"
pullPolicy: "Always"
Observed Error Behavior in the Worker Pods
Step1: Confirm that image tagged 368 is downloaded
kubectl describe <worker pod>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23s default-scheduler Successfully assigned trino/trino-cluster-worker-c4d5bbcc8-xflxh to k8s-node00
Normal Pulled 19s kubelet Successfully pulled image "trinodb/trino:368" in 1.254727643s (2.728144379s including waiting)
Normal Pulling 18s (x2 over 22s) kubelet Pulling image "trinodb/trino:368"
Normal Pulled 16s kubelet Successfully pulled image "trinodb/trino:368" in 1.299575889s (2.112766319s including waiting)
Normal Created 16s (x2 over 19s) kubelet Created container trino-worker
Normal Started 16s (x2 over 19s) kubelet Started container trino-worker
Warning BackOff 12s (x4 over 15s) kubelet Back-off restarting failed container trino-worker in pod trino-cluster-worker-c4d5bbcc8-xflxh_trino(ecb3e37d-67b2-4199-b6b7-94c0da17b5a7)
Step 2: Check the pod logs for the error
kubectl logs <worker pod>
+ set +e
+ grep -s -q node.id /etc/trino/node.properties
+ NODE_ID_EXISTS=1
+ set -e
+ NODE_ID=
+ [[ 1 != 0 ]]
+ NODE_ID=-Dnode.id=trino-cluster-worker-59bff658d4-94mf8
+ exec /usr/lib/trino/bin/launcher run --etc-dir /etc/trino -Dnode.id=trino-cluster-worker-59bff658d4-94mf8
Error occurred during initialization of VM
Could not find agent library /usr/lib/trino/bin/libjvmkill.so in absolute path, with error: /usr/lib/trino/bin/libjvmkill.so: cannot open shared object file: No such file or directory
FWIW, it appears that starting the chart version 412, there is a step to explicitly copy this file into the image at the /usr/lib/trino/bin folder.
Maybe we should document this as a restriction .. the chart should mostly be used with a recent release of Trino .. if you need older Trino release you probably need to use an older version of the chart.
However .. maybe we also should not copy jvmkill into the container either ..
And lastly .. the recent switch to Java 21 might have an effect as well .. and the so might or might not work depending on what processor architecture the container ends up using .. since its a static .so file...
Also @david-streamlio .. it would be great to work collaborate and get the connector into Trino itself .. that would reduce your maintenance and headaches for users.
Thanks for the quick response @mosabua . For now, do you happen to know which version of the chart would work with the older, "368" release, which runs on JDK 11?
Longer term, we should work together to update the Apache Pulsar Trino connector to the latest version of Trino, and maybe even get it added as a supported connector that is included in the deployment by default.
I will open an issue in our repository to track the upgrade effort.
If you get it merged into the Trino repo it will be part of every release, part of the Trino docs, and part of the default docker container. And therefore also flow into all the ecosystems and vendors where the main project is used.
In terms of which chart version .. honestly no idea .. I would go back in the git history in terms of date for starters. Probably a very old version of the chart. 368 is VERY old (2 years... https://trino.io/docs/current/release/release-368.html).
FYI, I found that helm search provides some useful information on the charts and the supported app version.
helm search repo trino -l
NAME CHART VERSION APP VERSION DESCRIPTION
trino/trino 0.14.0 432 Fast distributed SQL query engine for big data ...
trino/trino 0.13.0 423 Fast distributed SQL query engine for big data ...
trino/trino 0.12.0 423 Fast distributed SQL query engine for big data ...
trino/trino 0.11.0 375 Fast distributed SQL query engine for big data ...
trino/trino 0.10.2 375 Fast distributed SQL query engine for big data ...
trino/trino 0.10.1 375 Fast distributed SQL query engine for big data ...
trino/trino 0.10.0 375 Fast distributed SQL query engine for big data ...
trino/trino 0.9.0 375 Fast distributed SQL query engine for big data ...
trino/trino 0.8.0 375 Fast distributed SQL query engine for big data ...
trino/trino 0.7.0 375 Fast distributed SQL query engine for big data ...
trino/trino 0.6.0 375 Fast distributed SQL query engine for big data ...
trino/trino 0.5.0 372 Fast distributed SQL query engine for big data ...
trino/trino 0.4.0 370 Fast distributed SQL query engine for big data ...
trino/trino 0.3.0 355 Fast distributed SQL query engine for big data ...
trino/trino 0.2.0 355 Fast distributed SQL query engine for big data ...
Using version 0.11.0 of the chart resolved the issue.