shiyu22 / covidex

A multi-stage neural search engine for the COVID-19 Open Research Dataset

COVID-19 Open Research Dataset Search

This system contains the API server, neural models, and UI client, a neural search engine for the COVID-19 Open Research Dataset (CORD-19) and is refer to covidex.

At the COVID-19 Dataset Search system, Milvus is used to get the related articles. Let's start to have fun with the local deployment.

Local Deployment

Requirements

• Install CUDA 10.1

• Install Anaconda

  $ wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
  $ bash Anaconda3-2020.02-Linux-x86_64.sh

• Install Java 11

  $ sudo apt-get install openjdk-11-jre openjdk-11-jdk

Run Server

1. Start Docker container

The server system uses Milvus 0.10.0. Refer to the Install Milvus for how to start Milvus server.

$ docker run -d --name milvus_cpu_0.10.0 \
-p 19530:19530 \
-p 19121:19121 \
-v /home/$USER/milvus/db:/var/lib/milvus/db \
-v /home/$USER/milvus/conf:/var/lib/milvus/conf \
-v /home/$USER/milvus/logs:/var/lib/milvus/logs \
-v /home/$USER/milvus/wal:/var/lib/milvus/wal \
milvusdb/milvus:0.10.0-cpu-d061620-5f3c00

Point out the Milvus host and port in the api/app/settings.py file, please modify them for your own environment.

2. Prepare Anaconda environment

# Create an Anaconda environment named covdiex for Python 3.7
$ conda create -n covidex python=3.7
# Activate the covdiex environment
$ conda activate covidex
# Install Python dependencies
$ pip install -r api/requirements.txt

3. Build the Anserini indices and Milvus index

# updated all indices at api/index/
$ sh scripts/update-index.sh
# load all data to Milvus and build HNSW index
$ python milvus/index_milvus_hnsw.py --port=19530 --host=127.0.0.1

The port and host parameters indicate the Milvus host and port, please modify them for your own environment.

4. Run the server

# make sure you are in the api folder
$ cd api
$ uvicorn app.main:app --reload --port=8000

The server wil be running at localhost:8000 with API documentation at /docs

RUN UI Client

• Install Node.js 12+ and Yarn.

• Install dependencies

  # make sure you are in the client folder
  $ cd client
  $ yarn install

  If you changed the port of the server, please modify the parames at src/shared/Constants.ts at line 17 for your own enviroments.

• Start the server

  $ yarn start

The UI client will be running at localhost:3000, enter it in the browser to open the interface.

• Search somthing about COVID-19

  

• Get the related articles