Artifact evaluation for Dogfood
Overview
Dogfood is a prototype tool to generate workloads and test file systems. In this artifact, we introduce its basic usage, how to use it to reproduce the experiment results in the paper, and how to reuse it for further file system testing.
Obtain the artifact
We have upload a docker image containing all necessary files and subjects. HERE
Dogfood should be used under a Linux operating system.
Ubuntu-16.04 and Ubuntu-18.04 are recommendations as we have tested Dogfood under them.
Note that a virtual machine such as virtual-box or vmware may not work because Dogfood leverages the KVM,
which may be unavailable in a virtual environment (see INSTALL.md
).
Please follow the instructions in INSTALL.md
to complete the installation.
Reproduce results in the paper
Since there are three testing scenarios are presented in the paper, we introduce steps to reproduce the results, respectively.
The sequential scenario
Reproduce the detected bugs
The tool tmux
is preferred to open multiple terminals:
tmux -2 # Start tmux
Ctrl+b c # Start another terminal window
Ctrl+b w # Switch between the windows; learn more from the tmux documentations
As shown in the INSTALL.md
, start QEMU first:
$qemu: ./ctrl start
Create a virtual machine snapshot:
$test: ./ctrl init
Run bug case scripts:
$test: ./bugcase/<subdirectory>/trigger.sh
The status of each bug is listed in the file workspace/bugcase/buglist.md
Compare with Syzkaller for long-time running
Run dogfood is easy:
$qemu: ./ctrl start # Start QEMU
$test: ./ctrl init
$test: ./food --fs=<file system> --test
After running for 12 hours,
use the workspace/analyze-log.py
to extract the number of crashes.
The <file system>
should be one of
(ext4, btrfs, f2fs, reiserfs, jfs, gfs2, ocfs2).
Run Syzkaller need some extra steps:
- Install Syzkaller, follow the instructions
- Create a disk formatted by a file system (we provide scripts to make it easy, run):
cd workspace/objs/debian-image
./create-image.sh
./create-fs-image.sh <file system>
Which means:
- Create a Debian Linux Image, instructions
- Create a disk formatted by a file system containing the Debian Linux tools
- Run Syzkaller
We have provided a Syzkaller configuration and a script.
- Specify
workspace/syzkaller/my.cfg
file with two parameters:$WORKSPACE_DIR
---the absolute path ofworkspace/
, and the absolute path where Syzkaller installed.
If Syzkaller is installed according to the above-mentioned instructions successfully, just run:
./run.sh
After running for 12 hours, check the results.
The crash consistency scenario
Crashmonkey works on specific kernels. To ease the burden of setting, we provide a VM image, download here, which is based on Ubuntu-16.04 but with the specific kernel and containing crashmonkey.
tar -xzvf ubuntu-16.04-clone.qcow2.tar.gz
sudo apt-get install virt-manager
sudo virt-manager
Import the VM image (in virt-manager, click):
File -> New Virtual Machine -> Importing existing disk image -> Forward -> Browse -> Browse Local -> Choose the path of qcow2 file, Choose OS type: Linux, Version: Ubuntu 16.04 -> Forward -> Forward -> Finish
Start up the VM and login via ssh:
ssh icse20@192.168.122.106
# password: icse
If the IP address of the VM is not sure, use the following commands to check it:
sudo virsh net-list
That should display like this:
Name State Autostart Persistent
---------------------------------------
default active yes yes
Then, use the following command to check the IP address.
sudo virsh net-dhcp-leases default # `default` is the name from the last result
Now, we work in the VM under directory /home/icse20
.
All workloads generate by ACE is located at directory /home/icse20/crashmonkey/code/tests/generated_workloads
.
All workloads generated by Dogfood is located at directory /home/icse20/subjects/cmp-cases
and /home/icse20/subjects/large-cases
.
The former is used to compared with ACE and the latter consists of a great number of workloads (presented in the paper).
Run workloads by ACE:
cd crashmonkey
sudo python xfsMonkey.py -f /dev/vda -d /dev/cow_ram0 -t <fs> -e 102400 -u build/tests/generated_workloads/ > outfile
<fs>
should be one of (f2fs, ext4, btrfs).
Run workloads by Dogfood:
cp subjects/cmp-cases/yy-*.cpp dogfood/ # comparison cases
# or
cp subjects/large-cases/zz-*.cpp dogfood/ # large cases
cd crashmonkey
make
sudo python xfsMonkey.py -f /dev/vda -d /dev/cow_ram0 -t <fs> -e 102400 -u build/tests/dogfood/ > outfile
Run the command to count #detected issues
./diff_count.py
Note that before each run, clean redundant files by
./clean.sh
You can also use script dogfood/py-code/b3Food.py
to generate more subjects (generated in direcotry output/
) and copy these subjects to the directory dogfood
in the VM.
The concurrency scenario
We have generate 500 workloads for concurrency testing,
which is located at workspace/con-subjects
after being extracted.
tar -xzvf con-subjects.tar.gz
Configure FS
in run-con-subjects.sh
and execute it:
./run-con-subjects
Reuse Dogfood: automatic testing
Reuse dogfood to test other file systems.
We can create a fresh workspace to test multiple file systems in parallel.
The script workspace/bundle
is used.
TEST_DIR=<A testing directory>
FS=<File system to test>
./bundle $TEST_DIR
Now, we can use dogfood as mentioned before. Start the QEMU and testing.
cd $TEST_DIR
$qemu: ./ctrl start # start QEMU
$test: cd $TEST_DIR
$test: ./ctrl init
$test: ./food --fs=$FS --test
If you do not want to start qemu manually, using
$test: ./food --fs=$FS --test --qemu
This will start QEMU as a daemon thread automatically,
but the logs/outputs are also not displayed in the terminal.
We must check the log vm.log
to analyze the results.
Notes for multiple testing in parallel
When testing multiple file systems in parallel,
you should use bundle
to create testing directories for each file system and edit file $TEST_DIR/mngr-tools/config.sh
.
You MUST set TELNET_PORT
, TCP_PORT
, and SSH_PORT
differently for each file system testing.
Analyze the results
$test: ./analyze-log.py # Count #crashes
$test: ./analyze-log.py | sort | uniq # Count unique bugs
Reuse Dogfood: how to reproduce a new bug
When finding a bug logged in the log file workspace/vm.log
,
you can make a test case to reproduce it.
The script workspace/bug-extract
is used to extract bug cases automatically.
When finding a bug in the log, you can make a test case to reproduce it.
The following section introduces manual steps to reproduce a bug:
1. Locate the bug, like
[ 34.786048] kernel BUG at fs/f2fs/data.c:317!
2. Check the log from the bug location backward to find the test case name, like
TEST CASE[ ../workspace/C-output/2019_07_31-20:06:15-cGKYX/2.c with f2fs-0.img ]
This is the C program file to manifest the bug and its disk image.
MOUNT OPT [*]
between the location of test case name and bug location, like
3. Collect all mount options in MOUNT OPT [ background_gc=off,nouser_xattr,noacl,active_logs=2,inline_data,inline_dentry,extent_cache,noextent_cache,noinline_data,data_flush,mode=lfs,io_bits=3,usrquota,grpquota,quota,noquota,fsync_mode=nobarrier ]
template
4. Copy the C program file and disk image found in step 2 to the directory template/trigger.sh
5. Change the script DISK_NAME
is the name of disk image with out extension, likef2fs-0
in our example.CASE_NAME
is the name of C program file, like2.c
in our example.FS
is the file system to test
template/mountfood
, like
6. Copy the mount options line by line (the order should be the same as the log file) in the file options=(
"..." # Other options
"background_gc=off,nouser_xattr,noacl,active_logs=2,inline_data,inline_dentry,extent_cache,noextent_cache,noinline_data,data_flush,mode=lfs,io_bits=3,usrquota,grpquota,quota,noquota,fsync_mode=nobarrier"
"..." # Other options
)
7. Run it as the demo in section Test QEMU
$test: ./template/trigger.sh