Giters

RichardHoOoOo / wTestPlus

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

We publish the wTest+ tool and the evaluation results in this page.

Subjects

The following table lists the 38 open-source Android apps used in our evaluation.

Index Package Index Package
1 app.simple.inure 20 de.taz.android.app.free
2 ch.rmy.android.http_shortcuts.debug 21 fr.free.nrw.commons
3 com.activitymanager 22 im.fdx.v2ex.debug
4 com.boardgamegeek 23 io.github.wulkanowy
5 com.dar.nclientv2.debug 24 ml.docilealligator.infinityforreddit
6 com.ds.avare 25 name.boyle.chris.sgtpuzzles
7 com.flxrs.dankchat 26 net.bible.android.activity
8 com.gh4a 27 net.osmand.plus
9 com.github.k1rakishou.chan 28 nl.asymmetrics.droidshows
10 com.ichi2.anki 29 org.andstatus.app
11 com.kidozh.discuzhub 30 org.catrobat.paintroid
12 com.manimarank.spell4wiki 31 org.koitharu.kotatsu
13 com.nextgis.mobile 32 org.openhab.habdroid
14 com.nutomic.syncthingandroid 33 org.quantumbadger.redreader
15 de.blau.android 34 org.runnerup.free
16 de.danoeh.antennapod 35 org.wikipedia.alpha
17 de.freehamburger.debug 36 org.woheller69.omweather
18 de.herrmann_engel.rbv 37 org.woheller69.weather
19 de.tap.easy_xkcd 38 xyz.zedler.patrick.grocy

Coverage results

  • WebView-specific property coverage, WebView API call site coverage, and code coverage for open-source apps (link)

Detected bugs

Tool

Our tool is composed of 3 parts. The first part staticly builds the GUI component transition graph. The second part instruments an app. The third part generates tests for the instrumented app. But before testing an app, a customized Android OS needs to be loaded by Android emulators or real devices.

Customized Android OS

The instrumented app needs to run on a customized Android system because we add one more ID field in java.lang.Object in order to ease the variable tracking. It is very simple to use our customized Android system image, you just need to follow the following 2 steps

Usually, these files are placed under <Android_SDK_Root>/system-images/android-29/google_apis/x86

Once you have finished the above two steps, you can try to start the Android emulator as usual.

Static analysis (download)

Requirements

  • Java 1.8 or above

Steps to use

  • Create a folder for an app under test. The folder name should be the app's package name (We have created a folder for org.woheller69.weather under <path_to_GraphAndInstrumentation>/apps).
  • Run sh graph.sh APP_FOLDER APP_PACKAGE_NAME NON_LIB_PKGS to statically build the GUI component transition graph for an app. NON_LIB_PKGS is an array of package names separated by "," that are used to filter out library code in an app (e.g., if an app's application code is in a.b.c and a.b.d, NON_LIB_PKGS should be a.b.c,a.b.d). An example command can be sh graph.sh apps/org.woheller69.weather org.woheller69.weather org.woheller69.weather).

Instrumentation (packaged together with static analysis)

Requirements

  • Java 1.8 or above
  • Python 2 or 3 (We have tested on Python 2.7.16 and Python 3.6.8)
  • Node.js (We have tested on v16.13.2, v12.16.2, and v10.16.0. You may find these versions here. If you are using a Linux-based OS, you may install node by following the steps here)

Steps to use

  • Run sh instrument.sh APP_FOLDER APP_PACKAGE_NAME PORT_NUM NON_LIB_PKGS to instrument an app. PORT_NUM is a port used by the instrumented app to communicate with a nodejs server to instrument dynamically-loaded JavaScript code. It is recommended to be set as 3016, 3018, 3020, ..., etc. An example command can be sh instrument.sh apps/org.woheller69.weather org.woheller69.weather 3016 org.woheller69.weather).
  • When instrumentation finishes, you can see an output folder under the APP_FOLDER. The apk file whose name ends with -aligned-debugSigned.apk is the instrumented apk.

Test generation (wTest+) (download)

requirements

  • export JAVA_HOME=YOUR_JAVA_HOME (example: export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/)
  • export ANDROID_HOME=YOUR_ANDROID_HOME (example: export ANDROID_HOME=/Library/Android/sdk)
  • export PATH=$PATH:${ANDROID_HOME}
  • export PATH=$PATH:<path_to_wTestPlus>/node_modules/.bin (make sure you have downloaded wTestPlus)
  • export PATH=$PATH:${ANDROID_HOME}/emulator
  • export PATH=$PATH:${ANDROID_HOME}/platform-tools
  • build-tools 27.0.3 (for ComboDroid) and 29.0.3 are required (they are usually at ${ANDROID_HOME}/build-tools)
  • python3
  • droidbot (link)

Steps to use

  • Enter <path_to_GraphAndInstrumentation>/js and run sh launchServer.sh PORT_NUM. This nodejs server is registered on a PORT_NUM in order to recevice JavaScript code that is dynamically constructed from the app under test. The server is responsible for instrumenting the js code and sending the instrumented code back to the app. PORT_NUM should be the same as the one used when instrumenting the app. An example command can be sh launchServer.sh 3016
  • Launch Appium by appium -p APPIUM_PORT (example: appium -p 4723)
  • Launch an Android emulator
  • Enter <path_to_wTestPlus> and run sh test.sh STRATEGY PATH_TO_APP_FOLDER TIME_LIMIT ANDROID_SDK_PATH EMULATOR_ID APPIUM_PORT AVD_NAME BUILD_TOOLS_VERSION SYSTEM_PORT PYTHON3 DROIDBOT. STRATEGY can be wVar_sa (stands for wTest+), or wVar (stands for wTest), or wDroid, or api (stands for wTest-API), or code (stands for wTest-Code), or QTesting, or ComboDroid, or Fastbot (stands for Fastbot2), or AdaT. AVD_NAME is the name of the android emulator. BUILD_TOOLS_VERSION must be set as 29.0.3. SYSTEM_PORT is a port used by appium if you test multiple apps in parallel. It is recommended to be set as 8200, 8201, 8202, etc. PYTHON3 is the path to your python3 command (you can check by which python3). DROIDBOT is the path to your droidbot command (you can check by which droidbot). An example command can be sh test.sh wVar_sa <path_to_GraphAndInstrumentation>/apps/org.woheller69.weather 3600 ${ANDROID_HOME} emulator-5554 4723 emulator0 29.0.3 8200 /usr/bin/python3 <root>/.local/bin/droidbot.
  • When reaching the time limit, you can see the output under <path_to_wTestPlus>/output/<STRATEGY>/<APP_PACKAGE_NAME> (e.g., <path_to_wTestPlus>/output/wVar_sa/org.woheller69.weather). Under this folder, coverage.txt contains the covered WebView-specific properties and WebView API call sites over 60 minutes (time0, time1, ..., time59 means 1 minute, 2 minute, ..., 60 minute). Code coverage is also reported.

About