Test error: invalid remote context

Question

Test error: invalid remote context

bencomp opened this issue 6 years ago · comments

I think this is different from #9 and possibly related to the move of making all of iiif.io available via HTTPS by default.

When I run mvn package, two tests error with a Jena RiotException:

...
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ simpleAnnotationStore ---

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running uk.org.llgc.annotation.store.test.TestBOR
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.095 sec - in uk.org.llgc.annotation.store.test.TestBOR
Running uk.org.llgc.annotation.store.test.TestMirador2
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.327 sec - in uk.org.llgc.annotation.store.test.TestMirador2
Running uk.org.llgc.annotation.store.test.TestMirador214
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.228 sec - in uk.org.llgc.annotation.store.test.TestMirador214
Running uk.org.llgc.annotation.store.test.TestPublish
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.021 sec - in uk.org.llgc.annotation.store.test.TestPublish
Running uk.org.llgc.annotation.store.test.TestSearch
16:47:17,514 ERROR [org.apache.jena.riot] invalid remote context: http://iiif.io/api/presentation/2/context.json
Test testEndToEnd(uk.org.llgc.annotation.store.test.TestSearch) failed. Jena data is aviliable in: /var/folders/mw/cyflttp13qb25jvyqqhn74qwljlstg/T/27-47-2018_16-47-05-614/data
16:47:18,387 ERROR [org.apache.jena.riot] invalid remote context: http://iiif.io/api/presentation/2/context.json
Test loadManifest(uk.org.llgc.annotation.store.test.TestSearch) failed. Jena data is aviliable in: /var/folders/mw/cyflttp13qb25jvyqqhn74qwljlstg/T/27-47-2018_16-47-18-330/data
short: 6cffa0b6558ab0f3e40d61f5ab3be1bf
Tests run: 8, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 13.344 sec <<< FAILURE! - in uk.org.llgc.annotation.store.test.TestSearch
testEndToEnd(uk.org.llgc.annotation.store.test.TestSearch)  Time elapsed: 11.91 sec  <<< ERROR!
org.apache.jena.riot.RiotException: invalid remote context: http://iiif.io/api/presentation/2/context.json
	at uk.org.llgc.annotation.store.test.TestSearch.testEndToEnd(TestSearch.java:273)

loadManifest(uk.org.llgc.annotation.store.test.TestSearch)  Time elapsed: 0.063 sec  <<< ERROR!
org.apache.jena.riot.RiotException: invalid remote context: http://iiif.io/api/presentation/2/context.json
	at uk.org.llgc.annotation.store.test.TestSearch.loadManifest(TestSearch.java:124)

Running uk.org.llgc.annotation.store.test.TestSetup
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in uk.org.llgc.annotation.store.test.TestSetup

Results :

Tests in error: 
  TestSearch.loadManifest:124 » Riot invalid remote context: http://iiif.io/api/...
  TestSearch.testEndToEnd:273 » Riot invalid remote context: http://iiif.io/api/...

Tests run: 23, Failures: 0, Errors: 2, Skipped: 0
...

I wonder if it has to do with the redirect of http://iiif.io/api/presentation/2/context.json to https://iiif.io/api/presentation/2/context.json. Could it?

Ben Companjen · Answer 1 · Fri Sep 28 2018 18:44:23 GMT+0800 (China Standard Time)

I tried changing the context URLs in some of the test files, but that didn't help.

Searching for "Jena riot invalid remote context" brought me to this issue: https://issues.apache.org/jira/browse/JENA-1178. Old versions of Jena use outdated versions of HTTPClient that cannot handle HTTPS (well). Perhaps it is time to update some dependencies?

Ben Companjen · Answer 2 · Fri Sep 28 2018 19:10:56 GMT+0800 (China Standard Time)

On a first try on a new branch, I found:

I was able to use mvn versions:use-latest-releases to update all dependencies to their latest releases
I had to replace all instances of the old com.hp.hpl.jena package with org.apache.jena
I had to update the SolrStore to use the new way of creating an HttpSolrClient (with a Builder)

Now the code compiles again, but tests fail because transactions are not being used (correctly):

Tests in error: 
  TestBOR.testAberAnno:125 » TDBTransaction Not in a transaction
  TestBOR.testAnnotation:65 » TDBTransaction Not in a transaction
  TestBOR.testRetrieveAnnotation:105 » TDBTransaction Not in a transaction
  TestMirador2.TestMirador2:72 » TDBTransaction Not in a transaction
  TestMirador2.testPopulate:87 » TDBTransaction Not in a transaction
  TestMirador214.testMirador214:72 » TDBTransaction Not in a transaction
  TestPublish.testCreate:96 » TDBTransaction Not in a transaction
  TestPublish.testDates:180 » TDBTransaction Not in a transaction
  TestPublish.testDelete:107 » TDBTransaction Not in a transaction
  TestPublish.testDuplicate:165 » TDBTransaction Not in a transaction
  TestPublish.testPage:135 » TDBTransaction Not in a transaction
  TestPublish.testPublish:69 » TDBTransaction Not in a transaction
  TestPublish.testUTF8:153 » TDBTransaction Not in a transaction
  TestPublish.testUpdate:119 » TDBTransaction Not in a transaction
  TestSearch.getAllAnnotations:247 » TDBTransaction Not in a transaction
  TestSearch.loadManifest:114 » TDBTransaction Not in a transaction
  TestSearch.testEndToEnd:269 » TDBTransaction Not in a transaction
  TestSearch.testMirador:194 » TDBTransaction Not in a transaction
  TestSearch.testPagination:215 » TDBTransaction Not in a transaction
  TestSearch.testPassedWithin:100 » TDBTransaction Not in a transaction
  TestSearch.testSearching:154 » TDBTransaction Not in a transaction

Tests run: 23, Failures: 0, Errors: 21, Skipped: 0

Glen Robson · Answer 3 · Fri Sep 28 2018 19:40:20 GMT+0800 (China Standard Time)

Thanks Ben for looking into this. I'm afraid I didn't think to test SAS when I switched iiif.io to https. Thanks also for your investigation, and that mvn command looks very useful!

I can have a look at the transactions (unless your already looking into this). I take it, its this I should be following: https://jena.apache.org/documentation/txn/transactions_api.html

Thanks.

Ben Companjen · Answer 4 · Fri Sep 28 2018 19:51:55 GMT+0800 (China Standard Time)

@glenrobson I'm looking into this, but I had missed the txn documentation – I was looking at the tdb and tdb2 documentation, wondering which to use…

Ben Companjen · Answer 5 · Sat Sep 29 2018 00:20:02 GMT+0800 (China Standard Time)

Some hours of staring at code, trial and error later… I'm beginning to think SAS isn't as simple as its name suggests 🤓

And looking at some constructs in the code, I think this hasn't been the first time someone ran into the issue of starting and stopping transactions unless one is already in progress. From my humble perspective transactions are managed at too many layers, which allows that they get entangled.

Ben Companjen · Answer 6 · Sat Sep 29 2018 01:02:13 GMT+0800 (China Standard Time)

I pushed some work in progress to https://github.com/bencomp/SimpleAnnotationServer/tree/fix-transactions, if you're interested.

Glen Robson · Answer 7 · Sat Sep 29 2018 05:58:15 GMT+0800 (China Standard Time)

Thanks Ill take a look. The transaction stuff for Jena should only be in the JenaStore.java...

Glen Robson · Answer 8 · Sat Sep 29 2018 08:01:11 GMT+0800 (China Standard Time)

This should be fixed in the master branch now. I agree the transaction process is messy at the moment. At the start of this project I decided to use jena models as the package to pass around annotations but its becoming clear that this isn't a good idea. I've started some work in the bor_merged branch to move away from RDF models to json-ld wrapped by a java class. I think its time to push the RDF model down to the Store level and to keep the jena transactions as low as possible.

As it is at the moment you have to handle transactions to read a model that has been retrieved from the TBD dataset and as the models are passed around it means the transaction code is bubbling up higher than needed.

Thanks for all your help identifying the issues and fixing the https issue.

Ben Companjen · Answer 9 · Mon Oct 01 2018 14:18:38 GMT+0800 (China Standard Time)

You're welcome and thanks for continuing the work! I can confirm it now builds from the master branch.