steve1316 / android-cv-bot-template

Template Project for an Android CV Bot using OpenCV for template matching, MediaProjection for taking screenshots, and AccessiblityService to execute gestures on screen.

Android CV Bot Template

You can visit my Android bot of Granblue Fantasy for more usage details and examples of how to use this framework: https://github.com/steve1316/granblue-automation-android

Some additional usage examples at:

• Uma Android Training Helper
• Uma Android Automation
• Master Duel Decklist Importer
• Girls' Frontline Android Auto

This template project serves as a starter point for Android botting that relies on Computer Vision template matching via OpenCV and executing gestures like tapping and scrolling to accomplish a automation goal. It uses MediaProjection Service to programmatically take screenshots and the Accessibility Service to execute gestures. The framework itself, Automation Library, is well annotated with documentation to serve as explanations and usage hints.

Application.Preview.Demo.mp4

Provided Features

• A Home page that also houses the Message Log to allow the user to see informational logging messages.
• A Settings page that utilizes SharedPreferences to share data across the application and its associated Services.
• Floating overlay button to issue START/STOP signals to the bot process.
• Fleshed out template matching functions via OpenCV
• Notifications to alert users of various changes during workflow.
• Accessibility Service that will allow the bot process to execute gestures on the screen like tapping a specific point.
• Automatically checking for new app updates from your designated GitHub repository.
• Ability for sending you messages via Discord private DMs.

Requirements

1. Android Device or Emulator (Nougat 7.0+)
   1. (Experimental) Tablets supported with minimum 1600 pixel width like the Galaxy Tab S7. If oriented portrait, browsers like Chrome needs to have Desktop Mode turned off and situated on the left half of the tablet. If landscape, browsers like Chrome needs to have Desktop Mode turned on and situated on the left half of the tablet.
   2. Tested emulator was Bluestacks 5 with the following settings:
      • P64 (Beta)
      • 1080p
      • 240 DPI
      • 4+ GB of Memory
2. Display is set to FHD (1080p). If not, then will need to adjust scales in the Extra Settings page to accommodate.

Instructions

1. Download the project and extract.
2. Go to https://opencv.org/releases/ and download OpenCV 4.5.1 (make sure to download the Android version of OpenCV) and extract it.
3. Create a new folder inside the root of the android folder named opencv and copy the extracted files in /OpenCV-android-sdk/sdk/ from Step 2 into it.
4. Build the Javascript portion of the project by running yarn install in the root of the project folder as well.
5. You can now build and run on your Android Device or create your own .apk file.
6. Clean Project and then Rebuild Project under the Build menu.
7. After building is complete, you can test the capability of this framework in the Android Studio's emulator.
8. After you familiarized yourself with what the framework can do, you can refactor the application's package name in various parts of the project and in settings.gradle. Also adjust the app name in app.json and in strings.xml.
9. If you want your application automatically check for the latest updates from your GitHub repo using AppUpdater, you can either reuse the already provided app/update.xml in the template or do the following:
   1. Upload a .xml file to your Github repo using the provided example app/update.xml with your updated version number, release notes, and link to the Releases page of your GitHub repo.
   2. Update the setUpdateXML() with the RAW link to your new update.xml.
   3. Now when a user has a lower version number compared to the latest version in your Releases page in your GitHub repo, they will be prompted with a dialog window like this:

Some things to note while developing

1. ImageUtils class reads in images in .webp format to reduce apk file size. You can change this if you wish.
2. All images are recommended to be categorized in separate folders inside the /assets/ folder. Be sure to update the folderName variables inside the various functions in ImageUtils. Or you could remove the need to organize them and just put all image assets into one place. Just make sure to update the code references to the folderName variables.
3. When working on a horizontal screen, the coordinate axis gets flipped as well. So if your vertical orientation dimensions is 1080x2400, then the horizontal orientation dimensions gets flipped to 2400x1080.
4. (on the old branch) If you want to create nested Fragment Preference settings, there is an example provided to showcase how to do that in SettingsFragment.kt and mobile_navigation.xml.

Important Classes to be familiar with

BotService

• Facilitates the display and moving of the floating overlay button.
• Able to start/stop the bot process on a new Thread and notify users of bot state changes like Success or Exception encountered.

ImageUtils

• Able to template match for a single image or multiple image locations on the screen.
• Able to detect text via Google's ML Kit (Read up on the Google's documentation and my usage of it at https://github.com/steve1316/granblue-automation-android/blob/main/app/src/main/java/com/steve1316/granblueautomation_android/utils/ImageUtils.kt for a better understanding of it)

MediaProjectionService

• Starts up the MediaProjection Service to allow the ImageUtils class to programmatically grab screenshots to perform template matching on it.

MessageLog

• Sends informational logging messages from the Game class to the Home page of the application to quickly view what is going on.
• Automatically saves logs into text files when the bot stops.

MyAccessibilityService

• Starts up the Accessibility Service to allow the Game class to execute gestures at specified (x,y) coordinates on the screen.
  • Supported gestures are: tap, swipe, and scroll via AccessibilityAction.
• Note: If you encounter this Exception: kotlin.UninitializedPropertyAccessException: lateinit property instance has not been initialized, it means that you must have terminated the application via Android Studio. This causes the Accessibility Service to bug out. This does not happen in regular use without interference from Android Studio. To fix this, you can toggle on/off the Accessibility Service until the Toast message pops back up signalling that the Service is now running properly again.

JSONParser

• Reads in data and settings from a .json file which you can then choose to store however you like, whether it be stored in a static object or put into the SharedPreferences.

NotificationUtils

• Allows the bot process to create and update Notifications to notify users of the status of the bot and whether or not the bot encounters an Exception.
• Sends a STOP signal to the bot process from the Notification's button via the StopServiceReceiver class.

Technologies used

1. MediaProjection - Used to obtain full screenshots
2. AccessibilityService - Used to dispatch gestures like tapping and scrolling
3. OpenCV Android 4.5.1 - Used to template match
4. Tesseract4Android 2.1.1 - For performing OCR on the screen
5. Google's Firebase Machine Learning OCR for Text Detection
6. AppUpdater 2.7 - For automatically checking and notifying the user for new app updates
7. Javacord 3.3.2 - For Discord integration
8. Twitter4j - For Twitter integration
9. Klaxon 5.5 - For parsing JSON files
10. React Native 0.64.3 - Used to display the UI and manage bot settings