Generate Implementation Dashboard

[Laurent Senta]

Base Bricks

• #125
• #145 | 1 day
• #146
  • Update workflows to expose their JSON outputs & retrieve them from gateway-conformance | 1 day
  • Try in experiment repositories
  • Implement the artifact-retrieval script
  • Update client repositories
    • ipfs/kubo#10092
    • ipfs/boxo#442
    • ipfs/bifrost-gateway#195

Phase 1: Example output

Generate Dashboard - Base Structure

We'll generate a dashboard in gh-pages, under a sub-path like project.io/dashboard.

• Define the Dashboard sitemap | 1 day
• #150
  • Output to gh-pages and setup a single URL | 1 day
  • Generate the dashboard daily using outputs from other repos | 1 day
• #152 | 3 days
  • /(index)
  • /dashboard
    • render gateway version
    • render implementation
    • render results
• #156 to be used in gateway checker | 1 day

Generate Dashboard - Drill Down

We'll let the user drill down test results, and link to test outputs / github action results

• Create pages to drill-down tests results | 2 day
  • per version
  • per implementation
  • per test / sub test / etc
• Render the execution logs in tests | 1 day
• #166 (comment) | 1 day

Generate Dashboard - Specs Relationships

• #159
• #163
  • Add https:// to the specs URL + add Specs field for multiple specs
  • Create spec -> tests & tests -> spec indexes | 2 days

Implementation Dashboard - Going Public | 5 days

Generate Dashboard - 💅

• Export Hints in the test outputs as well
• If possible, add links to the test definition sources (it should be possible to extract the test line number and file, etc and create permalink to github's conformance source)
• Nicer Design | 1 day
• Edit Content | 2 day

Follow Ups

• fix the redirects for /tests and /results (not critical if we host at the root)
• Deal with matrix in client jobs (like https://github.com/ipfs/bifrost-gateway/actions/runs/5925256845)
• Validate specs (check the page and anchors exists)
• Generate a spec coverage table (detect how many specs pages + anchors we have and how much we test)

Related Discussions

#116
#116 (review)

[Steve Loeppky]

Awesome project breakdown @laurentsenta !

[Laurent Senta]

Updated with notes from @BigLep review's and discussion with @lidel + reorganized to support ipfs/public-gateway-checker#442 more quickly.