Testing, verification and test automation
monperrus opened this issue · comments
Wikipedia references:
- https://en.wikipedia.org/wiki/Software_testing
- https://en.wikipedia.org/wiki/Unit_testing
- https://en.wikipedia.org/wiki/Regression_testing
- https://en.wikipedia.org/wiki/Code_coverage
"Automated testing is the act of conducting specific tests via automation (e.g. a set of regression tests) as opposed to conducting them manually, while test automation refers to automating the process of tracking and managing the different tests." source
Interesting tool.
Detailed failure reports and hassle free assertions
Scott test reporter
https://github.com/dodie/scott
From @AG00:
There is a relationship between desired properties (requirements), tests (verification) and test results. Those may be present in a hierarchy of components/subsystems/libraries. In addition, there is a relationship between planning activities on the system (product) being developed and its desired properties. Further, with time there will be more than one system/product configuration to consider and more than one version in use with different installations/deployments/customers. These relationships quickly becomes complex to understand and keep track of. As changes are being made (code or documents) and automatic tests are run, it would be good to have the information model changes generated rather than manually written. The information model can also be used to convince a customer (or the devops team itself) that a complex/high-level system property is achieved.
Testcontainers to test different environments on the developer's machine
https://www.testcontainers.org/
A post about the challenges to get global metrics for code coverage
https://massol.myxwiki.org/xwiki/bin/view/Blog/GlobalLocalCoverage
Security-based testing
https://www.inspec.io/
DeFlaker: Automatically Detecting Flaky Tests
https://www.jonbell.net/icse18-deflaker.pdf
Property-based testing in Java
https://github.com/quicktheories/QuickTheories
Cross Browser Testing, Selenium Testing, and Mobile Testing
https://saucelabs.com/
Perfecto - Cloud-based Devops Testing
https://www.perfecto.io/
Automated Tool for Prioritizing JUnit Test Suites
https://github.com/kanonizo/kanonizo
mutation testing
https://github.com/hcoles/pitest
automatic unit test amplification
https://github.com/STAMP-project/dspot
https://arxiv.org/pdf/1811.08330.pdf
automatic detection of pseudo-tested methods
https://github.com/STAMP-project/pitest-descartes
https://arxiv.org/pdf/1807.05030.pdf
mutation testing in JS
https://stryker-mutator.io/
Property-based testing in OCaml
https://github.com/c-cube/qcheck
@robertfeldt you may want to subscribe to this issue.
Chris Parnin's Lecture on software bots
https://docs.google.com/presentation/d/1JClXhqNX8ufzmxoDpVPCLRDlqu4tUkvbJgwYICw70kM/edit?usp=sharing
Testing of Microservices
https://labs.spotify.com/2018/01/11/testing-of-microservices/
Amazon codeguru
It’s like having a distinguished engineer on call, 24x7
https://aws.amazon.com/codeguru/
Postman/Newman https://www.getpostman.com/
New testing course from TU Delft
https://sttp.site/
Rultor is a DevOps team assistant. It helps your programmers and release managers automate routine operations (merge, deploy and release), with an easy-to-use intuitive chatbot interface.
https://github.com/yegor256/rultor
A series of recent posts about test and automation
http://blog.testingcurator.com/2020/04/12/testing-bits-april-5th-april-11th-2020/
ScalaTest: https://github.com/scalatest/scalatest
Jasmine is a behavior-driven development framework for testing JavaScript code.
https://jasmine.github.io/
Catch2: modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++
https://github.com/catchorg/Catch2
Commonly used test data
https://github.com/search?p=2&q=dQw4w9WgXcQ&type=Code
Commercial product to accelerate tests with an automated dependency graph
https://yourbase.io/
Automated Cucumber-JVM test pack for DECS built on the Serenity BDD framework
https://github.com/UKHomeOffice/hocs-serenity-automation
Testing at Electronic Arts
SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning
https://arxiv.org/pdf/2203.05566.pdf
Property-based testing in language X
For example, in javascript
https://marmelab.com/blog/2019/04/18/property-based-testing-js.html
An interesting essay / demo / tutorial topic: Automated migration of Junit4 to Junit5
Suggestion: Testing for GraphQL APIs and Engines.
An interesting and very original topic for a presentation / demo in software testing:
code coverage in production
Topic suggestion: in depth analysis of a mocking library in language X (eg Mockito)
fuzz testing is an essential topic for testing and security
Robot Framework is a generic open source automation framework. It can be used for test automation and robotic process automation (RPA). https://robotframework.org/
Predictive Test Selection cuts testing time up to 70% by using machine learning to select and run only tests that are likely to provide useful feedback during test runs
https://gradle.com/gradle-enterprise-solutions/predictive-test-selection/
Load-Test-As-Code. https://gatling.io/
Instancio is a Java library that automatically creates and populates objects for your unit tests.
https://github.com/instancio/instancio
Mutagen and cargo-mutants are two relevant mutation testing tools for Rust.
Credits @VIGGEEN
cc/ @ErikNatanael
following up on Jepsen
JepREST: Functional tests for distributed REST applications.
A command-line benchmarking tool for rust
https://github.com/sharkdp/hyperfine
BDD with Rust https://github.com/cucumber-rs/cucumber
A library for generating fake data in Rust.
https://github.com/cksac/fake-rs
The art and craft of test-driven development (Increment, 2019)
Integrity Test Framework - a sophisticated integration test automation framework for Java
https://github.com/integrity-tf/integrity
Newman is a command-line collection runner for Postman
https://github.com/postmanlabs/newman
Framework for web testing, and capture&replay
https://github.com/microsoft/playwright
Kurtosis is a composable build system for multi-container test environments.
https://docs.kurtosis.com/