Tweetscape: The Supercharged Twitter Feed
Tweetscape surfaces the best "insider" information—and the conversation around it—as shared by the smartest people in a given topic (e.g. ETH, BTC, NFTs, or Tesla) on Twitter. Tweetscape curates article links shared by the most reputable accounts on Twitter for a number of topics (e.g. ETH, BTC, NFTs, or Tesla). Learn more here.
How it works
hive.one to determine who are the most reputable (i.e. the "smartest") people in a specific field (e.g. who are the experts in ETH, BTC, NFTs, or Tesla) on Twitter;
hive.one acts as a reputation layer for the internet, determining who you can trust through a weighted graph of who follows who (e.g. a reputable user following another user raises that other user's "attention score" by more than if some random Joe follows them).
Tweetscape then uses Twitter's API and that list of "smartest" people to get links to the articles most abundantly (and most recently) shared by the "smartest" people on Twitter for a given topic (e.g. ETH, BTC, NFTs, or Tesla). It also shows you the conversation around each link; you get to see the best links and what the smartest people are saying about them.
This repository is a React app built in Typescript with Remix, TailwindCSS, and PostgreSQL. Both the Remix app and PostgreSQL database are deployed worldwide as edge-based, clustered applications on Fly:
. ├── app │ ├── cookies.server.ts │ ├── db.server.ts │ ├── entry.client.tsx │ ├── entry.server.tsx │ ├── img.server.ts │ ├── root.tsx │ ├── routes │ │ ├── $cluster.tsx │ │ ├── img.$url.ts │ │ └── index.tsx │ ├── styles │ │ └── app.css │ └── utils.server.ts ├── db │ ├── copy.pgsql │ └── setup.pgsql ├── Dockerfile ├── fly.toml ├── LICENSE ├── package.json ├── postcss.config.js ├── public │ ├── favicon.ico │ ├── fonts │ │ ├── sans.css │ │ ├── sans.ttf │ │ ├── sans.woff │ │ ├── sans.woff2 │ │ ├── serif.css │ │ └── serif-vietnamese.woff2 │ └── pics │ ├── placeholder.png │ └── vanessa.jpg ├── README.md ├── remix.config.js ├── remix.env.d.ts ├── scripts │ ├── data.mjs │ └── utils.mjs ├── styles │ └── app.css ├── tailwind.config.js ├── tsconfig.json ├── wrangler.toml └── yarn.lock 10 directories, 66 files
app/directory contains all of the front-end Remix components, server-side business logic (files suffixed with
.server.tswill only be included in the server-side bundle), and pages.
public/directory contains publicly visible assets (e.g. fonts, pictures, favicons, etc).
scripts/directory defines a set of Node.js scripts used to seed our PostgreSQL database with data from Twitter's and Hive's APIs.
.pgsqlscripts that define our database schema. Use them with care (i.e. don't accidentally reset the production database).
styles/directory contains the entry point for the TailwindCSS compiler (which outputs
app/styles/app.csswhich you shouldn't ever edit manually).
- And, of course, the root directory contains a number of configuration files (for ESLint, Prettier, Tailwind, Fly, a
Dockerfilefor Fly deployments, etc).
I could've used an ORM like Prisma to automatically generate Typescript type definitions and my PostgreSQL database schema at the same time, but I like being able to use the full power of both languages separately, so I've redefined my data types twice: once in SQL (
db/setup.pgsql) and once in Typescript (
The data structures that I chose and the property names that I chose come directly from Twitter's API and Hive's API. I aimed to be as unopinionated as possible when storing their data; I stored everything that was returned in—what I think is—a perfectly normalized PostgreSQL schema.
It can be useful to install something like Beekeeper Studio to open the PostgreSQL database in a visual spreadsheet-like form in order to better understand what each table actually contains and how they relate to one another.
To setup the development environment for and to contribute to Tweetscape:
- Follow these instructions to install
nvm(our suggested way to use Node.js) on your machine. Verify that
nvmis installed by running:
$ command -v nvm
- Run the following command to install Node.js v16.14.0 (the current LTS that's specified in our
$ nvm use
- (Optional) Run the following command to set Node.js v16.14.0 as your default Node.js version (useful if you have multiple Node.js versions installed and don't want to have to remember to switch to v16.14.0):
$ nvm alias default 16.14.0
- Ensure that you have recent versions of Node.js and it's package manager
$ node -v 16.14.0 $ npm -v 8.3.1
- (Optional) Install the Cypress system dependencies if you plan on running our integration tests locally.
$ sudo apt-get install libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb
- Clone and
cdinto this repository locally by running:
$ git clone https://github.com/rooteco/tweetscape.git && cd tweetscape/
- Follow these instructions to install
yarn(our dependency manager for a number of reasons):
$ corepack enable
- Then, install of our project's dependencies with the following command:
- Copy over the
.envfile (which contains project secrets) from this private repository:
$ cp tweetscape-env/.env* tweetscape/
- Install the
flyctlCLI, login to the CLI, and setup port forwarding so you can access the PostgreSQL database and Redis instance from your machine:
$ fly proxy 5432 -a tweetscape-pg $ fly proxy 6379 -a tweetscape-redis
- (Optional) Install Beekeeper Studio to be able to visually manipulate the PostgreSQL data.
Or, you should be able to simply access the
psqlcommand interface directly:
$ psql postgres://postgres:<pwd-from-env-file>@localhost:5432/postgres
- Finally, you should be able to start a fully-functioning development server:
$ yarn dev
- (Optional) To run fully local versions of our deployed infrastructure, you'll have to first make sure you've got Docker properly installed and configured on your system. Then, you'll be able to start a local PostgreSQL database and Redis instance by running:
$ yarn concurrently yarn:dev:postgres yarn:dev:redis
Commit Message Format
I have very precise rules over how Git commit messages must be formatted. This format leads to easier to read commit history. Please refer to the following documentation for more info:
Commit Message Header
Commit messages that do not adhere to the following commit style will not be merged into
<type>(<scope>): <short summary> │ │ │ │ │ └─⫸ Summary in present tense. Not capitalized. No period at the end. │ │ │ └─⫸ Commit Scope: The page, API route, or component modified. │ └─⫸ Commit Type: ci|docs|feat|fix|perf|refactor|test|deps|chore
<summary> fields are mandatory, the
(<scope>) field is optional.
Must be one of the following:
- ci: Changes to our CI configuration files and scripts.
- docs: Documentation only changes.
- feat: A new feature.
- fix: A bug fix.
- perf: A code change that improves performance.
- refactor: A code change that neither fixes a bug nor adds a feature.
- test: Adding missing tests or correcting existing tests.
- deps: A change in dependencies.
- chore: A code change in utility scripts, build configurations, etc.
The scope should refer to the page, API route, or component modified.
This can be flexible however (e.g. the scope for a
docs: commit may be the
Use the summary field to provide a succinct description of the change:
- Use the imperative, present tense: "change" not "changed" nor "changes".
- Don't capitalize the first letter.
- No dot (.) at the end.