Need volunteers for Pilot test - May 27-June 7
zcorpan opened this issue · comments
We need 6 to 12 volunteers to run the pilot test May 27 (starting after the CG call) to June 7 (time limit was extended from June 2 to June 7).
The goal is to run through 3 test plans and produce test results, so we know that it works before we start the first complete test cycle.
If you are able to run tests (i.e., you know how to use the relevant screen reader), it would be great if you could contribute time that week for this. Please edit this comment to add your github username, or comment below. :-)
See #162 (comment) regarding assignments.
| Test plan | JAWS/ Chrome | JAWS/ Firefox | NVDA/ Chrome | NVDA/ Firefox | macOS VO/ Chrome | macOS VO/ Safari |
|---|---|---|---|---|---|---|
| checkbox | @jscholes | @jscholes | @jscholes @a11ydoer | @jscholes | @robfentress @a11ydoer | @robfentress |
| menubar | @jha11y | @jha11y | @shirsha @a11ydoer | @shirsha | @robfentress @a11ydoer | @JasonAment |
(Removed combobox, since it's not ready.)
From what I understand, @jongund @mcking65 and @mfairchild365 wanted to test everything. The runner only allows testers to assign themselves to a test plan if it currently has 0 or 1 assignees, but the admin interface allows assigning additional testers.
How to volunteer
Comment below with what you can test. (We need your GitHub username, so commenting here makes this easy.)
Instructions for running pilot tests
- You will be invited to the relevant GitHub team.
- Accept the invitation.
- Go to http://aria-at.w3.org/ and click "Sign in with GitHub" in the menu. Authorize the app with GitHub.
- Go to Settings and select the Assistive Technology that you will be using, then Save (there’s currently a known issue where the “Save” button doesn’t notify the selection has been saved, but nevertheless it works).
- Navigate to the Test Queue Page
- Open the Test Pilot test cycle
- Assign yourself to the appropriate test plans per the table above. (If there are already 2 assignees, the Assign me button is not present; comment below and an admin can add you.)
- Click the link for the test plan to start running the tests (in the left-most cell)
- Follow the test instructions and record all the results (all fields are required)
- Once all the results have been recorded, click Submit Results (in the iframe)
- An alert preceding the iframe should appear saying "This test is complete.". Click Next Test.
On the test page, there is an iframe that contains the test-specific instructions, a toolbar to navigate between tests within the test plan, and buttons:
- Raise an issue: if you find problems with a test.
- Re-do a test: remove any partial progress for the test and start over.
- Save and close: get back to the list of test plans.
What will happen with the pilot test results?
This is a pilot test to make sure the test runner app, test reports and the tests themselves work as anticipated. Any feedback you have is very welcome, please file an issue! The test results for this will be public but will not be published as being "official", since the tests themselves haven't yet been reviewed.
Later, we plan to do a full test cycle of all tests that are reviewed, and the results of that will be published as the first set of official test results.
@zcorpan Happy to test with NVDA/Chrome and NVDA/Firefox.
@zcorpan I am happy to test any combinations
I'm happy to help with any of those test combinations, with the qualification that I'm still resolving some issues with our license of JAWS 2019, so I may be stuck with JAWS 2018. Which versions of the AT and browsers are we using? For macOS, we should also specify the OS version, since the VoiceOver is integrated into the OS. I currently have access to machines with High Sierra and Catalina.
@robfentress good question! Let's discuss the issue of versions in #116
I cannot edit the initial comment. My guess is because I'm not a member of this GitHub repo. Please let me know which combinations I should test
@robfentress I've added you to macOS VoiceOver/Chrome and macOS VoiceOver/Safari.
@jha11y I've added you to JAWS/Chrome and JAWS/Firefox.
Thanks!
@zcorpan I'm also available to test with JAWS 2020 and Firefox/Chrome (in addition to NVDA as already indicated) if it would help.
@robfentress I've added you to macOS VoiceOver/Chrome and macOS VoiceOver/Safari.
So that means macOS 10.15.4, Chrome 81.0.4044.138, and Safari 13.1 assuming no changes before May 11?
Scheduling update: the new runner isn't yet ready for this pilot test, so we need to move the date. Apologies for the late rescheduling.
The new plan is to have the pilot test start on Wednesday May 20 and end on Tuesday May 26.
I’d be more than happy to do NVDA or Voiceover. Please put me down for the set that has the greatest need.
I can help with NVDA/Firefox and NVDA/Chrome
I've updated assignees and added instructions in the first post above.
hello: i can test with NVDA and jaws in both Firefox and chrome and i can test all the types thanks lucy
Scheduling update: All functionality needed for pilot test isn't ready yet. We need to move the pilot test date again, unfortunately.
The new plan is Wednesday May 27 to Tuesday June 2.
Thanks for your patience, and apologies for rescheduling.
Can you please let me know if the invite is already sent to join the github team?
I couldn't find it it in my mail.
I went ahead and tried to getting into staging environment via http://ara.w3.org/ .
Selected the "Authorize bocoup" button and got message saying unable to load
@shirsha the team we use (for now) is here: https://github.com/orgs/bocoup/teams/aria-at-report-testers . You have been invited to this team. You should be able to see your invitation here:
https://github.com/orgs/bocoup/invitation
After this is accepted, "Sign in with GitHub" in http://aria-at.w3.org/ should work.
@zcorpan Thankyou!
I am able to navigate to http://ara.w3.org/test-queue page. I will start working once the Pilot Test test cycle is initiated.
Note: A new NVDA version came out last week or so, please make sure to upgrade. See #116 (comment)
Note: new URL to test runner that should be used for the pilot test is http://aria-at.w3.org/
If you've previously signed in, the sign in should still work with the new URL.
@zcorpan I can test mac OS VO/ Chrome, NVDA/Chrome
I have performed the following steps:
- Made sure the version for NVDA is 2020.1, Chrome is 81.0.4 and Firefox is 76.0.
- Selected the AT from settings.
- Assigned my name to the appropriate test plans in both "Test" and "Test Staging”.
- Entered all the test results in iframe
But when I select the "Next Test" button, I got a message saying "Test 1 has not been completed in full and your progress on this test won't be saved." I have entered all required fields. Can you please let me know what am I missing?
This is happening in both "Test" and "Test Staging" test cycle.
Note: Same test cases are repeating after Test 5.
@a11ydoer I've invited you now and assigned you in the table above. Thanks!
@shirsha Are you using the the "run as" button when you run tests?
Can we test yet? I haven't tried because the info because the emails say the tests would be ready starting tomorrow (May 27)
@jha11y The pilot test has not officially started as there are still changes that need to be deployed to the app. Tomorrow is the official start of the test.
I have few questions:
- Which Test Cycle (Test or Test Staging) should I select?
- I don't see run as option. Where can I find it?
In the CG call tomorrow we plan to walk through how to run tests. The first post in this thread also has instructions, we'll make sure they are correct tomorrow.
Here are instructions for how to join the CG call.
I believe @mcking65 wanted another test to land today before we start the pilot test. The plan is to start after the CG call today.
@JasonAment it looks like you still need to accept the invitation. You can do so here: https://github.com/orgs/bocoup/invitation
Which Test Cycle (Test or Test Staging) should I select?
"Test Pilot"
I don't see run as option. Where can I find it?
After you've assigned yourself to a test plan, the test plan name in the left-most cell becomes a link. That link will start running the tests for that test plan.
See "Instructions for running pilot tests" above for more details; feel free to ask if something is still unclear.
In the CG call we talked briefly about redistributing the assignments, after combobox was removed. I haven't made any changes though. @mcking65 and @mfairchild365 can you follow up with this? I don't know what you wanted to test.
This could be done now in the app interface directly, and ignore the table above, if that is easier.
@JasonAment, hmm that's weird. I've invited you again now.
In the CG call we talked briefly about redistributing the assignments, after combobox was removed. I haven't made any changes though. @mcking65 and @mfairchild365 can you follow up with this? I don't know what you wanted to test.
This could be done now in the app interface directly, and ignore the table above, if that is easier.
Let's let people self-assign in the app interface. If you were assigned to the combobox test plan, or if you were not previously assigned to a test plan, or if you just have some extra time on your hands, feel free to self-assign to either the checkbox or menubar plans. We should have at least two testers for each combination of test plan + AT/browser.
If you are not sure how or where to assign yourself, comment here and we can help.
Thank you to everyone who contributed in the pilot test!
I look forward to hearing about this in the CG call today. What worked well, what could be improved. I see a lot of issues filed, many from @robfentress - thank you! I haven't yet reviewed them in detail, but it looks like we have some work to do.
Some tests are not completed. We can discuss whether to take the partial results or extend the pilot test.
We discussed learnings from the pilot test in the CG call, minutes here: https://www.w3.org/2020/06/03-aria-at-minutes.html
The feedback is fantastic. We will go through everything and make a summary report.
We want to reach complete test results by at least one tester for every test plan for the pilot test. To that end, the last day of the pilot test is moved to Sunday June 7.
Please comment if you're assigned and have tests left but are not able to complete further tests this week.
Thanks again!
@zcorpan Have completed all checkbox tests for JAWS/FF, JAWS/Chrome, NVDA/FF and NVDA/Chrome. Let me know if there are any problems with something or I can provide further feedback about the UX as a whole.
Hi @accessaces! Thank you, I have invited you to the relevant GitHub team. Please see the top of this issue for instructions, and coordinate with @mfairchild365 and @mcking65 with assignments.
I'll finish the menubar testing with VoiceOver using Chrome by the end of day on June 7th (probably sooner).
I have labeled all issues today that were reported during the pilot test.
- 52 issues with
pilot-test-2020, of which
We are also working on a document with more details on challenges and learnings.