fun-with-serverless/whatsapp-logger

A 3d art showing whatsapp application turning into small water drops that fall

WhatsApp Group Manager

Introducing a new application that helps you keep track of your WhatsApp group discussions. If you have several WhatsApp groups, you may sometimes find it challenging to keep up with the conversations and discussions, especially if you are a new member. This application solves this problem by saving the content of your WhatsApp groups in a Google Sheet and to create a daily summary of the conversations using ChatGPT.

With this application, new members can easily review previous discussions, even if they weren't a part of the group when the discussions took place. The Google Sheet acts as a database of all the discussions, allowing new members to search for relevant information using keywords or specific dates.

In the future, we will provide access to any media shared in the group, such as images, videos, and audio clips. With these features, you will have all the information you need at your fingertips, making it easier than ever to stay connected with your WhatsApp groups.

Report Bug · Request Feature

Table of Contents

High level architecture
Getting Started
- Prerequisites
- Installation
Usage
Operation Costs
Contributing
License
Contact
Logo

High level architecture

This architecture consists of several AWS components that work together to allow for capturing and storing data from the popular messaging application, WhatsApp. The components in the architecture include:

WhatsApp, the web application that serves as the source of data
Admin Lambda, a tool used for viewing and updating the configuration of the application
S3 bucket, which stores a QR image generated by the system
ECS Fargate, a container service that runs the WhatsApp web listener application and listens to the incoming data from WhatsApp web using headless chrome
SNS, which acts as a channel for pushing WhatsApp messages
SQS, a messaging service that also receives the WhatsApp messages
Write to Sheet Lambda, a function that reads the messages from SQS and writes them to a Google Sheet
AWS Secrets Manager, which holds secrets such as google and admin credentials
Parameters Store, which holds the configuration information for the application
Google Sheet, which acts as the final storage location for the messages, organized by group and each message as a separate line.

Getting started

Prerequisites

Make sure to have the latest CDK version installed (V2).
An AWS enviornment.
Python 3.10 (I highly recommend using pyenv).
Python Poetry
Add Poe the Poet to Poetry by running poetry self add 'poethepoet[poetry_plugin]'

Installation

Clone this repository.
The application uses CDK as IaC framework.
Run poetry install to install relevant dependencies.
Next run the poetry poe deploy. It will run the CDK deployment script. Approve the deployment of the various stacks. Sit tight, it will take a couple of minutes.
When the installation is complete you should get two links - 1. to the admin dashboard and 2. to the admin password stored in AWS.
Get the secret password, by going to the secret manager, scroll down and click Retrieve secret value
Go to the admin dashboard, the user name is adminand the password is the one you have copied.

Setting up OpenAI

You can create daily summary of your various WhatsApp groups.
Behind the scene this app is using OpenAI ChatGPT engine to summerize the conversations.
You need to setup an OpenAI dev account.
Signup to OpenAI
Create a secret key by going to the manage account and choose API Keys
Copy the key and paste it the OpenAI Key in the admin panel.
Every one hour, the list of available groups in your WhatsApp account are collected and shown under Groups Configuration.
You can choose per group where do you want the summary to be written at.

Setting up Google

Create a new spreadsheet in google sheet.
In case you want to save yur whatsapp chats into google sheets, you need to configure a google cloud account.
Head to Google Developers Console and create a new project (or select the one you already have).
In the box labeled "Search for APIs and Services", search for “Google Drive API” and enable it.
In the box labeled "Search for APIs and Services", search for “Google Sheets API” and enable it.

Service account is a special type of Google account intended to represent a non-human user that needs to authenticate and be authorized to access data in Google APIs.

Since it’s a separate account, by default it does not have access to any spreadsheet until you share it with this account. Just like any other Google account.

Here’s how to get one:

Enable API Access for a Project if you haven’t done it yet.
Go to “APIs & Services > Credentials” and choose “Create credentials > Service account key”.
Fill out the form
Click “Create” and “Done”.
Press “Manage service accounts” above Service Accounts.
Press on ⋮ near recently created service account and select “Manage keys” and then click on “ADD KEY > Create new key”.
Select JSON key type and press “Create”.
You will automatically download a JSON file with credentials. It may look like this:

{
    "type": "service_account",
    "project_id": "api-project-XXX",
    "private_key_id": "2cd … ba4",
    "private_key": "-----BEGIN PRIVATE KEY-----\nNrDyLw … jINQh/9\n-----END PRIVATE KEY-----\n",
    "client_email": "473000000000-yoursisdifferent@developer.gserviceaccount.com",
    "client_id": "473 … hd.apps.googleusercontent.com",
    ...
}

Remember the path to the downloaded credentials file. Also, in the next step you’ll need the value of client_email from this file.
Very important! Go to your spreadsheet and share it with a client_email from the step above. Just like you do with any other Google account.
Now go to the admin dashboard from the previous section.
Paste the json into the Google Secret text box.
Copy the the spreadsheet url you created in step one and paste it into the Sheet URL text box.
Click save and you are done.

Connecting WhatsApps

Behind the scenes this application uses WhatsApp web to pull chat content.
In order to connect to WhatsApp you need to scan a QR code with your Real WhatsApp instance, that is, the one that runs on a real phone.
Please be aware that this process is experimental and may result in your WhatsApp number being identified as a bot and disconnected. It is recommended to use a disposable number instead.
In the admin dashboard click on WebApp Client Status, and scan the image with your WhatsApp app.
You are good to go. See your spreadsheet gets updated.

Usage

After configuring the OpenAI and WhatsApp integration, the application will begin to collect a list of available groups associated with your WhatsApp account. These available groups can be viewed under the Groups Configuration tab, as shown in the image below:

For each group, you have the option to define whether you want it to be summarized and, if so, where you want the summary to be sent. You have three options to choose from:

Myself - The summary messages will be sent to your own chat. This is a good option if you want to keep the summary private. .
Original Group - The summary will be written in the original group where the discussion occurred.
None - This option will stop the daily summary for that specific group. This is the default for all new groups.

Once you've chosen where to send the summary, you can select the language in which the summary will be written. Currently, the application supports eight languages:

English - the default
Hebrew
Mandarin Chinese
Spanish
Hindi
Arabic
French
German

Operation Costs

The Serverless components in use, such as Lambda, DynamoDB, and SQS, are very inexpensive and are fully covered under the free tier. The only elements that cost money are a NAT instance and a single Fargate instance.

A t3.micro NAT instance costs around $7.4 per month,
while a Fargate instance with 0.25 vCPU and 1GB of memory costs approximately $10.4 per month. Using an ARM architecture may result in some savings for the Fargate instance.

Overall, the total monthly cost is approximately $17.8.

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

Fork the Project
Create your Feature Branch (git checkout -b feature/AmazingFeature)
Commit your Changes (git commit -m 'Add some AmazingFeature')
Push to the Branch (git push origin feature/AmazingFeature)
Open a Pull Request

Running the WhatsApp Listener locally

To run this application locally, you need to provision several resources in your AWS environment first:

Copy .env-example to a new file named .env.
Deploy the entire application using poetry poe deploy.
After a successful deployment, you will see the following items as output:
- SQSEVENTURL
- WHATAPPSNSTOPICARN
- QRBUCKETNAME
- EVENTBRIDGEARN
Copy these values to the relevant fields in .env.
From the root directory, run ./scripts/run_whatsapp_listener_locally.sh.

License

Distributed under the Apache License Version 2.0 License. See LICENSE for more information.

Contact

Efi Merdler-Kravitz - @TServerless

Logo

The project's logo was created by Dall-E 2 with the following description A 3d art showing whatsapp application turning into small water drops that fall

(back to top)

fun-with-serverless / whatsapp-logger