XAPI-00254 The Activity Object must contain all available information..
spsheni opened this issue · comments
In this test case, it does POST on 2 statements and does a GET on Activities. My LRS is designed such that the POST request on Statements returns immediately and writing of the statements to back-end DB happens slightly at later point of time.
Due to this, GET on Activities does not find requested activity in the DB. But to meet the 'SHOULD' requirement of the spec, I return 200 OK with a dummy Activity object with the requested IRI as its id. But at the end, the test case evaluates to 'failed'. How do we handle it? Can the GET on Activity Resource return 404-NotFound and the test case to retry until a 200-OK response?
As per the specifications, 'The LRS MAY respond before Statements that have been stored are available for retrieval'. And on Activities Resource, 'If an LRS does not have a canonical definition of the Activity to return, the LRS SHOULD* still return an Activity Object when queried'.
In the event that the LRS does not yet have any definition of the activity (let alone a canonical one), i would expect a 404 rather than fake, false data being returned.
What sort of delay are you talking about and is that represented in your Consistent Through header you are returning?
I started my LRS where the controllers will write the statements to DB synchronously and return back the status. In this model all the tests ran fine.
But in my design I need to have the statements be processed asynchronously. I have dedicated writer tasks for writing the statements to DB.
Now with this model I am seeing several of the test cases are failing, which were passed in the previous model.
With regard to delay, on the average I notice 30 to 60 seconds of delay between the controller request and the DB writes (not considering load, only test suite runs).
I have not understood clearly on how Consistent Through header will help here. In my LRS, when it returns, this header is set to the current time. The DB writer tasks are written such that they write the current time when the statements are really written into DB. This way the data returned must be consistent with respect to returned time information.
On the other hand, I tried to go through the sources in lrs-test-suite. In case XAPI-00254, first it POSTs 2 statements and expects 200 return code. Then immediately it sends a GET on Activities and expects 200 code. In my LRS, as the activity information is not yet updated in DB it returns 404. And the test suite reports 'failed'. So what I suspect is, instead of 'expect 200' these cases should atleast 'wait for 200 if 404', may be with a reasonable retry+delay.
Also to note, between Statement POST and Activity GET, there are no Consistent Through header info returned from LRS. So I guess this header will not really help.
For reference, here is the log from test suite run,
{
"title": "Activities Resource Requirements (Communication 2.5)",
"name": "Activities Resource Requirements",
"requirement": [
"Communication.md#2.5"
],
"log": "",
"status": "failed",
"tests": [
{
"title": "The Activity Object must contain all available information about an activity from any statements who target the same \"activityId\". For example, LRS accepts two statements each with a different language description of an activity using the exact same \"activityId\". The LRS must return both language descriptions when a GET request is made to the Activities endpoint for that \"activityId\" (multiplicity, Communication.md#2.5.s1.table1.row1, XAPI-00254)",
"name": "The Activity Object must contain all available information about an activity from any statements who target the same \"activityId\". For example, LRS accepts two statements each with a different language description of an activity using the exact same \"activityId\". The LRS must return both language descriptions when a GET request is made to the Activities endpoint for that \"activityId\"",
"requirement": [
"Communication.md#2.5.s1.table1.row1"
],
"log": "REQUEST SUPERTEST\n_______________________________________\nPOST /xapi/statements HTTP/1.1\r\nHost: localhost:8080\r\nAccept-Encoding: gzip, deflate\r\nUser-Agent: node-superagent/1.8.5\r\nContent-Type: application/json\r\nX-Experience-API-Version: 1.0.3\r\nAuthorization: Basic dTpw\r\nContent-Length: 1788\r\nConnection: close\r\n\r\n\n[{\"actor\":{\"objectType\":\"Agent\",\"name\":\"xAPI account\",\"mbox\":\"mailto:xapi@adlnet.gov\"},\"verb\":{\"id\":\"http://adlnet.gov/expapi/verbs/attended\",\"display\":{\"en-GB\":\"attended\",\"en-US\":\"attended\"}},\"object\":{\"objectType\":\"Activity\",\"id\":\"http://www.example.com/verify/complete/34534100123\",\"definition\":{\"type\":\"http://adlnet.gov/expapi/activities/meeting\",\"name\":{\"en-GB\":\"example meeting\",\"en-US\":\"example meeting\"},\"description\":{\"en-GB\":\"An example meeting that happened on a specific occasion with certain people present.\",\"en-US\":\"An example meeting that happened on a specific occasion with certain people present.\"},\"moreInfo\":\"http://virtualmeeting.example.com/345256\",\"extensions\":{\"http://example.com/profiles/meetings/extension/location\":\"X:\\\\meetings\\\\minutes\\\\examplemeeting.one\",\"http://example.com/profiles/meetings/extension/reporter\":{\"name\":\"Thomas\",\"id\":\"http://openid.com/342\"}}}}},{\"actor\":{\"objectType\":\"Agent\",\"name\":\"xAPI account\",\"mbox\":\"mailto:xapi@adlnet.gov\"},\"verb\":{\"id\":\"http://adlnet.gov/expapi/verbs/attended\",\"display\":{\"en-GB\":\"attended\",\"en-US\":\"attended\"}},\"object\":{\"objectType\":\"Activity\",\"id\":\"http://www.example.com/verify/complete/34534100123\",\"definition\":{\"type\":\"http://adlnet.gov/expapi/activities/meeting\",\"name\":{\"en-GB\":\"example meeting\",\"fr-FR\":\"réunion\"},\"description\":{\"en-GB\":\"An example meeting that happened on a specific occasion with certain people present.\",\"en-US\":\"An example meeting that happened on a specific occasion with certain people present.\"},\"moreInfo\":\"http://virtualmeeting.example.com/345256\",\"extensions\":{\"http://example.com/profiles/meetings/extension/location\":\"X:\\\\meetings\\\\minutes\\\\examplemeeting.one\",\"http://example.com/profiles/meetings/extension/reporter\":{\"name\":\"Thomas\",\"id\":\"http://openid.com/342\"}}}}}]\n\nRESPONSE SUPERTEST\n_______________________________________\nHTTP/1.1 200 OK\nserver: Apache-Coyote/1.1\nx-experience-api-version: 1.0.3\ncontent-type: application/json;charset=UTF-8\ncontent-length: 79\ndate: Sat, 07 Apr 2018 17:34:58 GMT\nconnection: close\n\n[\"11d43106-e52f-4687-84fe-2a13651d321d\",\"790b2a76-f1bf-4f1b-bf78-7a644e3741e8\"]\n=======================================\nError: Not Found\nREQUEST SUPERTEST\n_______________________________________\nGET /xapi/activities?activityId=http://www.example.com/verify/complete/34534100123 HTTP/1.1\r\nHost: localhost:8080\r\nAccept-Encoding: gzip, deflate\r\nUser-Agent: node-superagent/1.8.5\r\nX-Experience-API-Version: 1.0.3\r\nAuthorization: Basic dTpw\r\nConnection: close\r\n\r\n\nRESPONSE SUPERTEST\n_______________________________________\nHTTP/1.1 404 Not Found\nserver: Apache-Coyote/1.1\nx-experience-api-version: 1.0.3\nx-experience-api-consistent-through: 2018-04-07T17:34:58.936Z\ncontent-length: 0\ndate: Sat, 07 Apr 2018 17:34:58 GMT\nconnection: close\n=======================================\n",
"status": "failed",
"error": "RejectionError: expected 200 \"OK\", got 404 \"Not Found\"",
"tests": []
}
]
}
Other cases that are failing,
XAPI-00018
should return a voided statement when using GET "voidedStatementId"
XAPI-00015
XAPI-00016
should not void a voiding statement
XAPI-00096
XAPI-00172
XAPI-00254
XAPI-00330
There is no defined way to inform back to the LRP that the asynchronous process at LRS end has completed the processing and stored the statements in to DB.
One possible approach I think of is, the test-suite gets the statement ids back from the LRS after a POST. With these ids send a GET request for Statement, until server returns 'not a 404 code'. Then the test suite can assume the statement is completely processed at the LRS end.
I am working on a interim solution from LRS end. The Controller will wait (with a timeout say 60seconds) for the writer tasks to complete statements processing.
Hi,
I have the same issue.
From the spec
The LRS MAY respond before Statements that have been stored are available for retrieval and The LRS MUST include the header "X-Experience-API-Consistent-Through".
So the xAPI spec accepts an asynchronous behavior on the Statement API, which is great because the database may be updated by a worker, and not directly by the web request.
My implementation
My LRS supports both a synchronous recording mode and an asynchronous recording mode.
- The synchronous mode passes all the tests.
- The asynchronous mode passes the tests on the Statement API, but fails on XAPI-00254 (Activity API)
In my case, the asynchronous mode needs a short delay (<1 second) to update the statements AND activities in the database.
The X-Experience-API-Consistent-Through header is managed by the LRS.
What's wrong with XAPI-00254?
The XAPI-00254 test assumes that the activities will be immediately available and updated after a statements POST.
It does not take into account that statements recording may be asynchronous, and so related activities recording/update may also be asynchronous.
Suggested change
XAPI-00254 should first try to get the recorded statements (waiting them to be available, like in all the other tests), and then get the related activity definition.
Note: I speak about XAPI-00254 because it's the first test to fail for this reason, but there may be other tests with the same issue.