Winds - An Open Source Personalized RSS Reader

Open source & beautiful RSS reader built using React/Redux/Sails/Node 7 and Stream (getstream.io). Showcases personalized feeds (using machine learning similar to Facebook, Flipboard, Etsy, and Quora - powered by the getstream.io API).

This tutorial explains how the personalization API works (blogpost). Check out the hosted demo at http://winds.getstream.io.

Note: We launched this project in November of 2016. We're actively working on it and contributions are much appreciated!

Screenshots

Demo

Check out the hosted demo: winds.getstream.io

Installation

Step 0 - Git clone

git clone https://github.com/GetStream/Winds.git

Step 1 - Node & requirements

cd Winds

brew install nvm
nvm install 7.0.0

npm install .
npm install -g sails pm2

Step 2 - Secrets

Create a file called .env by duplicating .env_example using this command:

 cp .env_example .env

Now the .env file will have the following values:

STREAM_APP_ID = ""
STREAM_API_KEY = ""
STREAM_API_SECRET = ""
STREAM_ANALYTICS_TOKEN = ""
SENDGRID_USERNAME = ""
SENDGRID_PASSWORD = ""
SENTRY_DSN = ""
API_BASE_URL = "http://localhost:1337"
MONGO_URI = ""
FACEBOOK_API_KEY = ""
REDIS_AUTH = ""
JWT_SECRET = ""

Stream

Stream handles the feed personalization and storage. Accounts are free up to 3 million feed updates and handle personalization (machine learning) for up to 100 users.

Get started at getstream.io and visit the dashboard to get your credentials.

While you're in the dashboard, you'll also want to create the following feed types:

rss_feed (type = flat, realtime notifications = off)
timeline (type = flat, realtime notifications = on)
user (type = flat, realtime notificatons = off)
topic (type = flat, realtime notifications = on)

Email

To send email create an account on sendgrid.com and add your username and password. You can use other providers by customizing config/emails.js

Error Reporting (optional)

To track errors create an account on sentry.io. Next add your Sentry DSN to the .env file.

Database (optional)

Sails uses an ORM called Waterline, which supports many databases. If you don't provide the Mongo URI ,it will store your data on local disk. This is fine for trying out the app; however it is not a solution for a production level app. The full details are available in config/connections.js.

Step 3 - Scrape Some Data

A reader without any data isn't much fun though. Let's insert a few topics and RSS feeds into the database:

node load_initial_data.js

Step 4 - Redis

Redis is currently used as the primary session store. To install on macOS, simply use Homebrew:

brew install redis

Then run Redis with the following command:

redis-server

Step 5 - Launch!

Next, we need to run 2 cronjobs to ensure we keep on reading RSS articles and update the site's favicons. To make it easy to keep these cronjobs up and running, we use the amazing PM2 library:

pm2 start process_dev.json

This command runs 3 cron jobs and the app on port 1337. You can change the port in process_dev.json. Alternatively, you can use process_prod.json. The production config uses a background worker for scraping the RSS feeds. It will work better if you expect to scrape thousands of feeds.

Development

To run in development mode, run sails lift in the command line like so:

sails lift

You can now see your own RSS reader at: localhost:1337

Step 5 - Enjoy

Point your browser to: localhost:1337, follow topics, create an account and add feeds as you please.

Contributing to This Project

Project Layout

React

The React codebase is located in /assets/js. There you'll find the actions, components and reducers.

API

The API is located in /api. It uses Sails, so their documentation is a good place to start.

Design

The Sketch File files are available for download via Invision.

Ideas for Improvements

Contributions are much appreciated. Here are some ideas for improvements:

Secondary links (ie comments link for HNews and Lobsters)
Deploy to Heroku button
Follow suggestions (we're working on this)
Switching between feeds should be easier
Lightweight task queuing system for emails and discover endpoint
Keyboard shortcuts (vim style)
GraphQL style APIs so you have more flexibility for building your own mobile apps
Android & iOS apps
Support more sites (RSS data quality is pretty poor and often needs custom logic per site/feed)
Search article's you've read using Algolia
Folders/Groups

Roadmap

At the moment we're gathering feedback from the community before deciding on the changes for Winds 0.2

Adding Support for Your Favorite Feeds

Unfortunately, RSS is more of a guideline than a standard. There is a good chance that the feed you're trying to add isn't correctly parsed. If this happens, there are two things you can do:

You can submit an issue. Be sure to specify the exact url you tried to add. Every now and then we will go over these outstanding issues and try to resolve them.
If you're a developer you'll want to fork our project. As a starting point you can add a test in discover.test.js and run it

NODE_ENV=testing mocha test/bootstrap.test.js test/integration/**/*discover* -g sentry

Next you'll want to open up ScrapingService.js and DiscoverService.js. Most of the time you can resolve the problem by adding an if statement in these files. If statements are of course an ugly solution. After we learn more about the type of customization required per feed, we'll add support for subclassing and extending the scraping logic.

Running the Test Cycle

We use Mocha for the test cycle. It's a pretty default setup for Sails. The only tricky bit is that you to specify the NODE_ENV as testing. You also need to load test/bootstrap.test.js before executing other tests. Here is an example:

Run all tests:

NODE_ENV=testing mocha test/bootstrap.test.js test/integration/**/**

Style Guide

Running JS Beautify

find . -name '*.js' | grep -v node_modules | grep -v tmp | xargs -n 1 js-beautify -r

Details About Operations & Cron Jobs

RSS is quite a broken standard. It works, but barely so. You'll often have to customize the scraping logic for specific sites. Here are a few commands which make it easier to test your feeds:

Scrape the feeds containing avc in the URL. Scrape only 1 article at the time and run with concurrency 1.

node scrape_feeds.js -q avc -a 1 -c 1

Scrape all feeds that weren't updated in the last 3 minutes:

node scrape_feeds.js

Force all feeds to be scraped:

node scrape_feeds.js -f

Scrape the favicons:

node scrape_favicons.js -c 10 -q cnn

Native macOS Support

Winds supports a native macOS client through the use of https://github.com/electron/electron. An example application (macOS) pointed at the hosted version can be found in the releases section this repo.

To install the native macOS application using Homebrew, simply run brew cask install winds.

Building Your Own Native Client

To make the build process easier, we chose to go with the popular tool Nativefier, a command line tool that allows you to easily create a desktop application for any web site with succinct and minimal configuration. Apps are wrapped by Electron in an OS executable (.app, .exe, etc.) for use on Windows, macOS and Linux.

To start, you'll need to install the Nativefier module from NPM in your terminal:

npm install nativefier -g

Then, run the following command to build on macOS:

nativefier --name "Winds" "https://your-domain.com" --icon "icon.png" --insecure --show-menu-bar

The full API documentation can be found at: https://github.com/jiahaog/nativefier/blob/development/docs/api.md

jd-h / client