datafiniti / eighty-apps

Example 80apps for use on the 80legs platform

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

EightyApps

###Basic 80app format

onst EightyApp = require('eighty-app');
const app = new EightyApp();

app.processDocument = function(html, url, headers, status, cheerio, extras) {
    const data = {};
    return JSON.stringify(data);
};
app.parseLinks = function(html, url, headers, status, cheerio, extras) {
    return [];
}
module.exports = function() { 
    return app;
};

Testing

To test your 80apps, you should use our test site.

Note about "img" tags

Note that if you use the parseHTML method in EightyApp.js, "img" tags will be changed to "img80" tags. This is so the crawlers do not load the images when using the EightyApp to parse the html response (strangely "img" tags seem to be the only html elements affected by this). If you need to reference an "img" tag by its tag type explicitly (i.e. not by its class, id, or some other attribute) in some html, it will instead be an "img80" tag, but everything else should be the same.

Currently Available Cheerio (i.e. jQuery) Methods

The new version of Voltron - Mauler - uses an extended version of Cheerio, a lighter weight version of jQuery. You will still write functions in the same manner as before (ex: $html.find('selector').parent()); however, in order for Cheerio to obtain its faster speed, it only uses certain core jQuery functions. You can see the list of already implemented Cheerio functions that are available to you here: http://cheeriojs.github.io/cheerio/

Our extended version of Cheerio also includes a number of other functions that are available for your use. These include:

* .not
* .makeArray
* .each
* .filter
* .prop

We are currently working on implementing the pseudo selectors :eq and :first (ex: $html.find('div:eq(1)')); however, they are NOT currently implemented and WILL cause errors. Check here or our knowledge base for updated Cheerio functionality.

CrawlImages.js

To successfully use this app, the image url submmitted must contain an 'EightyFlag', a tag that indicates the url is an image.

http://www.example.com/exampleimage.jpg?80flag=type:image

About

Example 80apps for use on the 80legs platform


Languages

Language:JavaScript 100.0%