Final project for the "Cloud Computing" course at La Sapienza University of Rome.
The goal of this project is to develop an application that provides a search engine for images of bacteria. The search engine should be able to retrieve the most similar images to a given query image, uploaded by the user. The project will be deployed on Amazon Web Services and the ”algorithmic part” to compute similarity among images will be done using Weaviate.
| Upload the image | Search the most similar bacterias | Analyze the specs |
|---|---|---|
We will use a
tool like Weaviate pre-trained neural network to extract features from the images
and create a fast index of reference items.
When a query image is provided, the search engine will compare the features of the
query image with the features of the reference items in the index and return the most
similar images.
The key point of the Weaviate usage is the embedding of images in a vectorial databese.
When the user add an image (or any other data object) to Weaviate, the vector
database computes a vector embedding for that image using a given model. This
vector is then placed into an index, allowing the database to quickly perform searches
by finding the most similar vectors in the database to a given query vector.
In order to allow the scaling of our application we would like to implement it through a set of microservices such that the single functions can be scaled individually. But identifying the set of microservices in which an application can be split in isn’t a trivial task, however we identified three main components of our application:
- A web user interface from which our user can interact with the application;
- An API call for querying the database;
- The vector database itself.
The implementative solution involves the use of an EC2 instance on which to directly upload the already containerized application. In this first, fairly trivial example, a call to lambda function via a specific event on the web page (in an S3 bucket) is then always expected, and the lambda code will then provide for using the EC2 computational resource to launch the two docker images. Finally, the computed output will be returned dynamically and asynchronously (with respect to the request from the client). The following steps then power to good scalability performance:
- Set up an Auto Scaling group: we create an Auto Scaling group using our Launch Configuration. We specify the desired capacity, minimum and maximum number of instances, and scaling policy. The Auto Scaling group will automatically launch and terminate instances based on the defined criteria.
- Configure a load balancer: we set up an Elastic Load Balancer (ELB), in particular an Application Load Balancer (ALB) to distribute incoming traffic across our EC2 instances. We configure the load balancer to listen on the appropriate ports and protocols for our web app, i.e. port 80 for the HTTP protocol.
- Add instances to the load balancer: we associate the instances launched by the Auto Scaling group with our load balancer. This ensures that traffic is evenly distributed among the instances (fairness performance).
- Monitor and adjust scaling: finally we can monitor the performance of our web app and adjust the scaling policies as needed.
"GanziManzi" team
