support docker-aio
pdurbin opened this issue · comments
In pull request #7 I started adding dependencies for docker-aio such as a yum install of Docker itself. 😄 Thank you to @donsizemore and @poikilotherm for helping me! I don't think I've ever installed Docker on anything but a Mac or a Linux laptop before.
I'm spinning off this new issue from #1 because my current goal is parity with what I have at https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-apitest-develop/ and I don't use Docker for that. So I don't see Docker as an absolute dependency for parity but it should would be nice to get it working. Easier on the AWS bill for sure.
Parity is the first goal I tried to explain in my "thoughts on automated testing" post at https://groups.google.com/d/msg/dataverse-dev/ISot5k4VjZQ/t-hzPk8tAwAJ which I expressed as " Get https://jenkins.dataverse.org to do the same job that the older Jenkins server and phoenix do which is to catch regressions before a release."
Because I'm having so much trouble getting docker-aio working on a newly spun up Jenkins server I decided to back up and run in on my Mac again. I got a working Dataverse. I'm on IQSS/dataverse@3ce173c9d
Mac
Setting up your glassfish4 to support Dataverse
Glassfish directory: /usr/local/glassfish4
Domain directory: /usr/local/glassfish4/glassfish/domains/domain1
/usr/local/glassfish4/bin /opt/dv/testdata/scripts/installer
domain running
remote failure: Application dataverse-4.0 is not deployed on this target [server]
Command undeploy failed.
Deleted 1 option(s)
Command delete-jvm-options executed successfully.
Created 1 option(s)
Command create-jvm-options executed successfully.
Created 1 option(s)
On my Jenkins server (a t2.large), I'm getting the following error.
Jenkins server
Setting up your glassfish4 to support Dataverse
Glassfish directory: /usr/local/glassfish4
Domain directory: /usr/local/glassfish4/glassfish/domains/domain1
./glassfish-setup.sh: line 285: pushd: /usr/local/glassfish4/bin: No such file or directory
./glassfish-setup.sh: line 289: ./asadmin: No such file or directory
domain running
./glassfish-setup.sh: line 49: ./asadmin: No such file or directory
./glassfish-setup.sh: line 53: ./asadmin: No such file or directory
...
Help me @pameyer you're my only hope!
@pdurbin I'm partly speculating here, but my suspicion is that the base system doesn't have unzip installed; so that script that downloads and patches glassfish can't unpack the distribution archive. If this is on the right track, conf/docker-aio/dv/deps/glassfish4dv.tgz should be a lot smaller than usual, and not have a functioning glassfish inside it.
If that's not the case, it might be image caching (aka - docker images show a different hash for the centos:7 image being used.
@pameyer THANK YOU! In my previous commit I had added yum install zip but in 8ea4fc3 I just added yum install unzip. I just spun up a new EC2 instance and ran conf/docker-aio/prep_it.bash which ended with this output:
docker-aio ready to run integration tests
{"status":"OK","data":{"version":"4.14","build":"develop-3c697b1"}}
Fingers crossed!
Woo-hoo! Running conf/docker-aio/run-test-suite.sh not only worked, all the tests passed!
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.266 s - in SearchIT
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 92, Failures: 0, Errors: 0, Skipped: 4
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4:05.998s
[INFO] Finished at: Fri Jun 07 10:50:59 UTC 2019
[INFO] Final Memory: 65M/755M
[INFO] ------------------------------------------------------------------------
I'm on 8ea4fc3 of this dataverse-jenkins repo (on a newly spun up t2.large EC2 instance) and on IQSS/dataverse@3c697b1 on the dataverse side.
I created a new job in Jenkins based on the above but my latest blocker is some sort of permission problem with /tmp. Here's a screenshot from http://ec2-3-81-58-115.compute-1.amazonaws.com:8080/job/IQSS-dataverse-api-test-suite-docker-aio-develop/1/testReport/junit/edu.harvard.iq.dataverse.export/SchemaDotOrgExporterTest/testExportDataset/
Over at IQSS/dataverse#5885 @donsizemore reported exactly this problem:
Similarly, in SchemaDotOrgExporterTest.java:
try (PrintWriter printWriter = new PrintWriter("/tmp/dvjsonld.json"))
breaks the SchemaDotOrgExporter test with a permission denied.
For now I'm just going to delete the file in /tmp that was created when I did the run as "centos" rather than "jenkins":
[root@ip-172-31-34-143 dataverse-jenkins]# ls -l /tmp/dvjsonld.json
-rw-rw-r--. 1 centos centos 2458 Jun 7 10:35 /tmp/dvjsonld.json
[root@ip-172-31-34-143 dataverse-jenkins]# rm /tmp/dvjsonld.json
rm: remove regular file ‘/tmp/dvjsonld.json’? y
[root@ip-172-31-34-143 dataverse-jenkins]#
Bah!
dvinstall/solrconfig.xml dvinstall/install
adding: dvinstall/glassfish-setup.sh (deflated 68%)
adding: dvinstall/pgdriver/ (stored 0%)
adding: dvinstall/pgdriver/postgresql-42.2.2.jar (deflated 7%)
adding: dvinstall/dataverse.war
zip I/O error: No space left on device
A t2.large isn't big enough.
[root@ip-172-31-34-143 dataverse-jenkins]# df -hl
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 8.0G 7.9G 176M 98% /
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 17M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
tmpfs 783M 0 783M 0% /run/user/1000
overlay 8.0G 7.9G 176M 98% /var/lib/docker/overlay2/90125d0480750e69da4bb229aa3f1b68e9e0bd4ff9d771429926be3460e559f1/merged
shm 64M 4.0K 64M 1% /var/lib/docker/containers/bbccc5cc756edeb7dbd5391d83f06cceb3607bdf6af598ddef90fea34e780316/mounts/shm
tmpfs 783M 0 783M 0% /run/user/0
[root@ip-172-31-34-143 dataverse-jenkins]#
@donsizemore is the boat at https://jenkins.dataverse.org big enough for you to play around with pull request #9? 😄 I just requested a review from you. I'd be happy for @pameyer and others to take a look as well.