daandtu / Android-Web-Scraper

Android Web Scraper is a simple library for android web automation. You can perform web task in background to fetch website data programmatically.

android-web-scraper

Android Web Scraper is a simple library for android web automation. You can perform web task in background to fetch website data programmatically.

Setup

The project is hosted on jCenter. Add the following to your dependencies:

implementation 'com.daandtu:android-web-scraper:1.0.1'

Add internet permission to AndroidManifest.xml:

<uses-permission android:name="android.permission.INTERNET"/>

Sample usage

Initialisation:

WebScraper webScraper = new WebScraper(this);
webScraper.setUserAgentToDesktop(true); //default: false
webScraper.setLoadImages(true); //default: false
webScraper.loadURL("https://www.github.com/");

If you want to see the browser automation in action:

layout.addView(webScraper.getView());

Interact with webpage elements:

Element el1 = webScraper.findElementByXpath("//*[@id=\"search\"]");
Element el2 = webScraper.findElementByName("img",3);
el1.setText("Android");
el2.click();
Element el3 = webScraper.findElementById("result");
String result = el3.getValue();

Setup OnPageLoadedListener:

webScraper.setOnPageLoadedListener(new WebScraper.onPageLoadedListener() {
            @Override
            public void loaded(String URL) {
                //TODO
            }
        });

Other methods:

Bitmap screenshot = webScraper.takeScreenshot(); //Pay attention with big webpages or use
Bitmap screenshot2 = webScraper.takeScreenshot(500,MAX);

String title = webScraper.getWebsiteTitle();

String html = webScraper.getHtml();

webScraper.clearHistory();
webScraper.clearCache();
webScraper.clearCookies();
webScraper.clearAll(); //Clear history, cache and cookies

webScraper.reload();