This code walks you through the installation and usage of scispaCy for natural language processing. For our example, we use data from CORD-19, a large collection of articles about the Covid-19 pandemic.
scispaCy is a full, open-source spaCy pipeline for Python designed for analyzing biomedical and scientific text. It is a very powerful tool, especially for named entity recognition (NER), but it can be somewhat confusing to understand. The goal of this code is to show scispaCy in easy to understand terms. I hope it makes navigating the world of entity extraction a little easier.
The code is divided into 5 main sections
- Installing the necessary packages
- Importing the packages
- Reading a single text file
- Reading in lots of data
- Methods to help read in data
This part is pretty straightforward. We install scispacy and spacy along with the specific NLP models available in scispacy. The models are installed using their URLs, found here.
Importing these packages is the same as any other Python packages. Nothing new here.
We use pandas to read in the csv file we want. All we need is the path to the file. Once that is done, we pick a specific text to extract from that file and pass it through one of the models. Then, we display our results.
This section expands the previous code to loop over the entire csv file and continually grab the text we want and use NER to grab the entities and their attributes.
The helper methods are optional, but like all methods they help make the code a little more concise. The first 4, labeled "Methods to add entity/value pairs to table" each employ a different scispaCy model to add entities to the target table. The last method, labeled "Method to Extract Entity/Value Pairs" employs a given scispaCy model to perform entity extraction on a given text.
If you perform everything correctly, you should get your entity/value pairs along with the article doi in a nice looking CSV file.
If you're looking for more detailed instructions, check out the post I wrote about this code here.