A Messenger Bot working as a Trip Advisor.
Trippy was developed with the intention to come up as a Messenger Bot which can help consumers to decide upon several factors including flights,Rails,Buses when planning a trip.
Technology Stack Used = Facebook Messenger APIs, JSON for data communication,CasperJS,PhantomJS,PHP for processing the data received on Call Back URL.
Architecture of Trippy :
- The Bot Receives message from the end user.
- Facebook Calls the CallBack URL on my hosted site passing a JSON file including SenderID,ReceipientID,Sender Params,Message etc.
- PHP code parses the Message and then executes the corresponding CasperJS Utility Script to Scrape data from different websites.
- The CasperJS Scripts basically use a HeadLess Browser provided by PhantomJS which loads the mobile version of sites.
- The CasperJS scripts scrape the data and then the control returns to PHP.
- PHP then parses the scraped data to return a well-formated response to end user depending upon the query.
Hosting Methodologies:
- The Domain https://thetrippy.co.in was set up for trippy to be accessible as a product.
- But for running NodeJS scripts from PHP you would need a Virtual Private Server apart from shared hosting. Also to test locally tools like ngrok/localtunnel can be used to make your localHost address accessible via Internet(WWW) especially to FaceBook as a callBack URL
Why CasperJS and PhantomJS combination ?
- I started with NodeJS Library Cherio but then Cherio is only meant for scraping Static HTML websites
- Then I went onto ScraperJS which has a static Scraper (Internally uses CherioJS) and a dynamic Scraper(Internally uses PhantomJS) but then at times dynamic Scraper suffers if the site loads its content using AJAX/Excessive JS leading to page loading issues.
- CasperJS is built on top of PhantomsJS and is the best Library available for scraping Dynamically loading websites.
- PHP also comes with CLI to scrape websites which internally uses FireFox Agent which faces certain browser support issues as not most sites these days support FireFox. Hence a heads up for Phantom which is a HeadLess browser.
Data Scraped By CasperJS Scripts using CLI Arguments
- Flights ###########################################################################
C:\Users\amitabh\Desktop\Trippy>phantomjs.exe --config=config.json casperjs/bin/ bootstrap.js --casper-path=casperjs --cli home.js flights Mumbai Bangalore 31/08 /2017
[ "flights", "Mumbai", "Bangalore", "31/08/2017" ] { "casper-path": "casperjs", "cli": true } just before running...... Continue running...... There were 70 flights Suggestions are as follows :
Info= Jet Airways 9W-339 03:00 6h 50m 09:50 Rs. 6,109 [error] [phantom] Done
Done
- Buses ##########################################################################
C:\Users\amitabh\Desktop\Trippy>phantomjs.exe --config=config.json casperjs/bin/ bootstrap.js --casper-path=casperjs --cli home.js buses Mumbai Bangalore 31/08/2 017
[ "buses", "Mumbai", "Bangalore", "31/08/2017" ] { "casper-path": "casperjs", "cli": true } URL loaded...... Continue running......
Canara Pinto Travels Volvo A/C -Multi Axle Semi sleeper (2+2) 15:00 07:30 16 h 30m2 Rest Stop 4.1/5(568 Ratings) 10 5 1450
Orange Tours And Travels A/C Sleeper (2+1) 16:00 09:00 17 h 00m 4.1/5(304 Ratings) 36 24 2000
Jabbar Travels Scania AC Multi Axle Semi Sleeper(2+2) 15:00 07:30 16 h 30m 4.3/5(255 Ratings) 44 20 1425
Anand travel Volvo A/C Multi Axle Sleeper(2+1) 16 h 00m 4.2/5(231 Ratings) 36 24 1950 INR 2000
VRL Travels I-Shift Multi Axle Volvo A/C Sleeper (2+1) 14:00 06:30 16 h 30m 4/5(282 Ratings) 36 24 1800
SRS Travels Scania AC Multi Axle Semi Sleeper(2+2) 14:30 06:00 15 h 30m 3.9/5(170 Ratings) 46 23 1000
VRL Travels I-Shift Multi Axle Volvo A/C Sleeper (2+1) 16:00 08:30 16 h 30m 3.8/5(151 Ratings) 36 24 1800
Anand travel A/C Sleeper (2+1) 18:30 10:30 16 h 00m 3.5/5(261 Ratings) 30 20 1950
Canara Pinto Travels A/C Sleeper (2+1) 16:00 08:00 16 h 00m1 Rest Stop 3.4/5(247 Ratings) 18 12 1750
VRL Travels Volvo Multi Axle SemiSleeper I-Shift B11R (2+2) 12:30 05:30 17 h 00m 3.9/5(104 Ratings) 52 25 1200
VRL Travels Volvo Multi Axle SemiSleeper I-Shift B11R (2+2) 15:30 08:30 17 h 00m 3.8/5(123 Ratings) 50 24 1200
VRL Travels Volvo Multi Axle SemiSleeper I-Shift B11R (2+2) 23:00 15:50 16 h 50m 3.7/5(133 Ratings) 52 25 1200
Sharma Transports Volvo A/C Multi Axle Semi Sleeper (2+2) 12:15 06:00 17 h 45m 3.8/5(112 Ratings) 39 20 1429
VRL Travels I-Shift Multi Axle Volvo A/C Sleeper (2+1) 22:30 15:00 16 h 30m 3.6/5(113 Ratings) 36 24 1800
Jabbar Travels A/C Sleeper (2+1) 16:45 08:45 16 h 00m 3.6/5(107 Ratings) 30 20 1615
SRS Travels Scania AC Multi Axle Semi Sleeper(2+2) 19:30 11:45 16 h 15m 3.6/5(85 Ratings) 47 23 1000
Canara Pinto Travels A/C Sleeper (2+1) 19:00 10:30 15 h 30m1 Rest Stop 2.8/5(165 Ratings) 20 14 1550
Sharma Transports Volvo A/C Multi Axle Sleeper(2+1) 20:30 13:15 16 h 45m 3.4/5(145 Ratings) 29 19 2095
SRS Travels Volvo A/C Multi Axle Sleeper(2+1) 15:00 07:15 16 h 15m 3.5/5(64 Ratings) 34 23 1600
[error] [phantom] Done
Done
- Trains ##############################################################################
C:\Users\amitabh\Desktop\Trippy>phantomjs.exe --config=config.json casperjs/bin/ bootstrap.js --casper-path=casperjs --cli home.js trains Kalyan Jabalpur 31/08/2 017 3A
[ "trains", "Kalyan", "Jabalpur", "31/08/2017", "3A" ] { "casper-path": "casperjs", "cli": true } CHHAPRA EXPRESS 11059 10:55 01:45 Rs. 1,200
MAHANAGARI EXP 11093 00:10 16:10 Rs. 1,210
LTT PPTA EXPRESS 12141 23:35 14:15 Rs. 1,255
LTT BSB SUP EXP 12167 00:35 16:30 Rs. 1,255
VSG PATNA EXP 12741 07:35 23:00 Rs. 1,255
LTT RJPB EXP 13202 22:15 15:55 Rs. 1,210
DARBHANGA EXP 11061 12:15 03:30 Rs. 1,205
KOLKATA MAIL 12322 21:30 13:10 Rs. 1,255
LTT BSB SUP EXP 12165 05:23 20:40 Rs. 1,250
JBP GARIB RATH 12188 13:30 06:00 Rs. 830
BHAGALPUR EXP 12336 08:05 23:20 Rs. 1,255
GORAKHPUR EXP 15017 06:35 00:55 Rs. 1,210
[error] [phantom] Done
Done
Screenshot
These Data Sets can then be used to answer user Queries like in case of flights :
-
User needs to check flights of only a specific AirLine
-
User needs to leave the source city in a specific time slot of a day say 3pm to 6pm
-
User needs to reach the destination in a specific time slot.
-
User needs the cheapest flight
-
Also we give suggestions to user in the form of cheapest flight available on 6 consecutive days (Like if he wants to travle on nth day we calculate the prices for n-3,n-2,.......n+3 th day.
So on.....