-
May 26: A makeup class movie file for concolic testing is uploaded
-
May 24: Please run only 1 instance of AFL++ per person, since no one can run AFL++ if all CPU cores of a server are occupied by AFL++ instances.
-
May 16: No class on May 21 due to graduate applicant interview
-
May 9: We will have 10 more minutes in each of the remaining classes, instead of having makeup classes.
-
May 9: Offline class on May 14 (Tues) and online class on May 16 (Thurs) as usual
-
Apr 11: The due dates of HW#3 and HW#4 are postponed as follows for you to focus on midterm exams ;-)
- HW#3 due date: April 20
- HW#4 due date: April 30
-
Mar 11: Online class on Mar 19 (Tues), Offline class on Mar 21 (Thurs)
-
Feb 15: Midterm exam on April 23 (1 week later from the midterm exam period) due to the conflict w/ ICSE 2024 (April 14-20)
-
Feb 22: CS458 class room is changed to E3 RM#1101
-
Instructor: Prof. Moonzoo Kim
- Office: 2434 (located at the east wing of E3-1)
- Phone: 042-350-3543
- E-mail: moonzoo.kim @ gmail.com
- Office hour: Tues 4:00-5:00 PM (reservation e-mail is preferred)
-
Teaching assistants: Youngseok Choi (youngseok.choi@kaist.ac.kr)
-
Lecture hours: Tue/Thur 5:30 - 6:45 PM
-
Lecture room: E3 Rm#1101 (and through Zoom)
- a face-to-face lecture on Tues and an online lecture on Thurs (this plan may change later)
- Zoom link for Thurs class: https://kaist.zoom.us/j/5258253316
-
Recommended prerequisite: CS230 System Programming, proficiency in C/C++ programming and Linux utilities
- Due to the high load of TA, TA will not provide help for basic C/C++/Linux questions.
- This class can be difficult to follow for exchange students.
-
Grading: attendance/class participation/quiz: 20%, HW: 50%, mid/final exams:30%
- You should turn on your web cam when you participate an online class; class attendance will not be counted otherwise.
- More than 7 absences of the class will get F grade
- Late attendance shall be counted as 1/3 absence. If a student is not able to attend a class due to an important event (e.g., attending conf., etc.), he/she should submit 1 week prior notice to the professor.
- The official language in the class is English. All students should submit all written documents such as homework, project reports, exam, etc. in English; 20% penalty of the max score otherwise.
-
Homework:
- Homework should be submitted through KLMS https://klms.kaist.ac.kr/course/view.php?id=156169
- Hint: many questions of exams are from the homeworks.
- Late HW will be accepted with 10% penalty of the max score in 1 day, 30% penalty of the max score in 3 days. HW will not be accepted after then.
- All programming HWs you submit must be able to be replayed by executing a single script file on a TA's server account (i.e., submitted HW should not have a dependency on your home directory, environment, etc.). Also, the replayed execution must demonstrate the same output to the submitted hw. You will get 10% penalty of the max score otherwise.
- For your programming HWs, you should not use external libraries which are not available on the server machines. If you really need an external library, you have to ask TAs to install it on the server machines.
- Please, include your student ID in the name of your submiited file(s), so that TAs can easily identify which files are yours.
- Homework should be submitted through KLMS https://klms.kaist.ac.kr/course/view.php?id=156169
-
Questions on the course materials and homeworks should be posted as github issues
- https://github.com/swtv-kaist/cs458-spring24/issues
- You can see the Q&A of the previous year's CS458 class
-
Excerpts from 강의평가 (4.93 out of 5 in Spring 23)
- "이번 학기에 졸업하는데, 지금까지 들은 과목 중 가장 재미있고 실용적이었던 것 같아요! 과제도 이전 평에 비해 너무 어렵지 않고 딱 적당했던 것 같습니다."
- "테스팅에 대해 많이 배워갈 수 있었습니다. 과제가 많긴 하지만 다 도움이 되는 것 같습니다. 감사합니다."
- "먼저, 좋은 강의 해주셔서 진심으로 감사합니다. 본 강의를 통해 전반적인 테스팅 기법들에 대해 알 수 있었던 것 같습니다. 특히, fuzzing, concolic testing, SMT solver 등 실질적으로 써먹을 수 있는 기법들을 실습과 병행해서 잘 배울 수 있었던 것 같습니다... 이번 학기 가장 많은 것을 얻은 과목이라고 자부합니다. 교수님께서 학생들에게 진심을 다해 강의해주시는 모습을 보며 큰 동기부여가 되었고, 많은 지식을 습득할 수 있었던 것 같습니다. 항상 감사합니다."
This class covers automated SW testing/verification techniques to detect SW bugs by analyzing SW source code and its diverse runtime behaviors. In particular, this class focuses to teach techniques that automatically generate test cases to achieve high code coverage and, thus, to detect many bugs.
The class teaches practical applications of testing/verification techniques by applying open-source testing/verification tools to complete homework tasks. Also, it guides students to learn the underlying fundamental algorithms of such techniques/tools to improve the performance of automated testing/verification.
“Software testers do not make software; they only make them better.” – Anonymous
- Feb 27 : Introduction [pdf]
-
Feb 29: Necessity for systematic & automated testing techniques [pdf]
- "Variability and Reproducibility in Software Engineering: A Study of Four Companies that Developed the Same System" by Anda et al. IEEE Trans. on Software Engineering vol. 35, no. 3, pp. 407-429, May-June 2009.
-
Mar 5: Overview of testing techniques (including the input partitioning technique) [pdf]
“It’s hard enough to find an error in your code when you’re looking for it; it’s even harder when you’ve assumed your code is error-free.”— Steve McConnell
-
Mar 7: Graph coverage [pdf]
- Intro to Software Testing web site
- "An Experimental Comparison of Four Unit Test Criteria: Mutation, Edge-Pair, All-uses and Prime Path Coverage" by Nan Li et al. '09 Intl. Conf. on Software Testing, Verification, and Validation Workshops
-
Mar 12, 14: Graph coverage for source code [pdf], Generating CFG using GCC and Graphviz [pdf]
- http://www.webgraphviz.com/
- Cyclometic complexity (또다른 한글문서) https://www.guru99.com/cyclomatic-complexity.html
- Useful commercial tools to manually analyze SW:
- Understand C++ (https://scitools.com) (free academic version available)
- Source Insight (https://www.sourceinsight.com/)
-
Homework 1: Graph coverage (Due Mar 21)
-
Mar 19: gcov tutorial [pdf]
-
Mar 21, 26: Logic coverage [pdf]
-
Homework 2: Manual testing of
grep
utility for branch coverage (graph coverage) -
Mar 28: Logic coverage from source code [pdf]
-
Apr 2, 4: Mutation testing [pdf]
-
Homework 3: Manual testing of
tcas
utility for MC/DC coverage (logic coverage) (Due April 20 midnight)
-
Apr 9: Clang tutorial 2/2: a program analysis tool by using Clang [pdf]
-
Apr 11: Q&A for midterm exam
-
Homework 4: kcov coverage tool (Clang) (Due April 30 midnight)
-
Apr 23: Midterm exam (closed book, NOT including Clang part) 5:30-7:00 pm
-
Apr 30, May 2: LLVM pass [pdf], Clang vs LLVM IR [pdf]
-
Homework 5: kcov coverage tool (LLVM IR) (Due May 14 midnight)
-
May 7: Automated Bug Detection through Static Analysis [pdf]
-
May 14: Case study of Fuzzing Application: POWERUP fuzzer [pdf]
-
Homework 6: Automated testing of grep by applying AFL++ (Due May 26 midnight)
-
May 16: Automated SW analysis for high reliability: a Concolic testing approach [pdf]
-
May 21: No class
-
May 23: CROWN tutorial [pdf]
-
May 28: CROWN Examples [pdf], Example: Concolic testing a circular queue [pdf]
-
May 30, June 4: System-level concolic testing: Busybox application examples through CROWN [pdf]
-
Homework 7: Automated testing of TCAS by applying CROWN (Due June 9 midnight)
-
June 4: Automated unit testing [pdf], Unit-level concolic testing: Busybox ls example [pdf]
- June 13: Final exam (closed book) 5:30-7:00 pm