gadenbuie / scorecard-db

Script for porting College Scorecard data to SQLite

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

College Scorecard Database

Turn College Scorecard data into parquet files, a SQLite database, CSV and RDS (R data storage) files.

License: GPL-3

Dependencies and Setup

This project uses renv to manage dependencies. To install the dependencies, run renv::restore() in the R console on R 4.4.0.

renv::restore()

Instructions

We use targets to manage the data prep pipeline. To download the source data and build all outputs, run the following in the R console:

targets::tar_make()

Raw College Scorecard Data

The raw College Scorecard data is available as a set of parquet files in data/parquet. The complete scorecard dataset is stored as a set of parquet files in data/parquet/merged. Two informational tables contain the data dictionary (data/parquet/dd_info.parquet) and the variable labels (data/parquet/dd_labels.parquet) for categorical variables in the scorecard dataset.

scorecard <- targets::tar_read("path_data_full_merged") |> arrow::read_dataset()
dd_info   <- targets::tar_read("path_data_full_info")   |> arrow::read_parquet()
dd_labels <- targets::tar_read("path_data_full_labels") |> arrow::read_parquet()

Tidy College Scorecard Tables

The most approachable tables are the school_tidy and scorecard_tidy targets.

school_tidy <- targets::tar_read("school_tidy")
skimr::skim_tee(school_tidy)
── Data Summary ────────────────────────
                           Values
Name                       data  
Number of rows             11300 
Number of columns          25    
_______________________          
Column type frequency:           
  character                9     
  factor                   3     
  logical                  10    
  numeric                  3     
________________________         
Group variables            None  

── Variable type: character ────────────────────────────────────────────────────
  skim_variable   n_missing complete_rate min max empty n_unique whitespace
1 name                    0         1       3  93     0    10525          0
2 city                    0         1       3  23     0     2915          0
3 state                   0         1       2   2     0       59          0
4 zip                     0         1       5  10     0     8757          0
5 url                  4943         0.563  15 123     0     5435          0
6 deg_predominant       998         0.912   8  11     0        4          0
7 deg_highest          1193         0.894   8  11     0        4          0
8 locale_type          5415         0.521   4   6     0        4          0
9 locale_size          5415         0.521   5   7     0        6          0

── Variable type: factor ───────────────────────────────────────────────────────
  skim_variable         n_missing complete_rate ordered n_unique
1 control                       1        1.00   FALSE          3
2 adm_req_test               8685        0.231  FALSE          4
3 religious_affiliation     10449        0.0753 FALSE         60
  top_counts                              
1 For: 5908, Non: 2760, Pub: 2631         
2 Con: 1205, Not: 1015, Req: 273, Rec: 122
3 Rom: 232, Uni: 85, Bap: 56, Pre: 54     

── Variable type: logical ──────────────────────────────────────────────────────
   skim_variable    n_missing complete_rate    mean count              
 1 is_hbcu               5412         0.521 0.0168  FAL: 5789, TRU: 99 
 2 is_pbi                5412         0.521 0.0105  FAL: 5826, TRU: 62 
 3 is_annhi              5412         0.521 0.00272 FAL: 5872, TRU: 16 
 4 is_tribal             5412         0.521 0.00594 FAL: 5853, TRU: 35 
 5 is_aanapii            5412         0.521 0.0350  FAL: 5682, TRU: 206
 6 is_hsi                5412         0.521 0.0909  FAL: 5353, TRU: 535
 7 is_nanti              5412         0.521 0.00543 FAL: 5856, TRU: 32 
 8 is_only_men           5412         0.521 0.0102  FAL: 5828, TRU: 60 
 9 is_only_women         5412         0.521 0.00510 FAL: 5858, TRU: 30 
10 is_only_distance      2832         0.749 0.00709 FAL: 8408, TRU: 60 

── Variable type: numeric ──────────────────────────────────────────────────────
  skim_variable n_missing complete_rate      mean         sd       p0      p25
1 id                    0         1     2550768.  8357052.   100654   182632. 
2 latitude           5412         0.521      37.3       5.87    -14.3     33.9
3 longitude          5412         0.521     -90.4      18.2    -171.     -97.5
       p50      p75       p100 hist 
1 367422   455666.  49664501   ▇▁▁▁▁
2     38.6     41.2       71.3 ▁▁▆▇▁
3    -86.3    -78.9      171.  ▂▇▁▁▁
scorecard_tidy <- targets::tar_read("scorecard_tidy")
skimr::skim_tee(scorecard_tidy)
── Data Summary ────────────────────────
                           Values
Name                       data  
Number of rows             183306
Number of columns          24    
_______________________          
Column type frequency:           
  character                1     
  logical                  6     
  numeric                  17    
________________________         
Group variables            None  

── Variable type: character ────────────────────────────────────────────────────
  skim_variable n_missing complete_rate min max empty n_unique whitespace
1 academic_year         0             1   7   7     0       27          0

── Variable type: logical ──────────────────────────────────────────────────────
  skim_variable                 n_missing complete_rate mean count
1 cost_med_similar                 183306             0  NaN ": " 
2 cost_med_overall                 183306             0  NaN ": " 
3 amnt_earnings_med_10y_similar    183306             0  NaN ": " 
4 amnt_earnings_med_10y_overall    183306             0  NaN ": " 
5 rate_completion_med_similar      183306             0  NaN ": " 
6 rate_completion_med_overall      183306             0  NaN ": " 

── Variable type: numeric ──────────────────────────────────────────────────────
   skim_variable             n_missing complete_rate        mean          sd
 1 id                                0       1       1163825.    5311085.   
 2 n_undergrads                  20563       0.888      2285.       5101.   
 3 cost_avg                     104675       0.429     15809.       8216.   
 4 cost_avg_income_0_30k        105507       0.424     13996.       7669.   
 5 cost_avg_income_30_48k       115448       0.370     14736.       7723.   
 6 cost_avg_income_48_75k       120294       0.344     16807.       7750.   
 7 cost_avg_income_75_110k      129900       0.291     19224.       7843.   
 8 cost_avg_income_110k_plus    138593       0.244     21352.       9036.   
 9 amt_earnings_med_10y         138347       0.245     35399.      14829.   
10 rate_completion              134721       0.265         0.333       0.237
11 rate_admissions              181562       0.00951       0.722       0.221
12 score_act_p25                156788       0.145        20.2         3.69 
13 score_act_p75                156794       0.145        25.4         3.55 
14 score_sat_verbal_p25         156906       0.144       484.         72.3  
15 score_sat_verbal_p75         156905       0.144       592.         69.3  
16 score_sat_math_p25           156771       0.145       486.         75.7  
17 score_sat_math_p75           156773       0.145       594.         72.3  
             p0        p25        p50        p75     p100 hist 
 1  100654      164562     213987     416670     49664501 ▇▁▁▁▁
 2       0         117        490       2050       253594 ▇▁▁▁▁
 3 -103168        9319      15281      21090.      112050 ▁▁▇▁▁
 4 -117833        7952      13336      19036.      111962 ▁▁▇▂▁
 5  -44508        8618.     14055      19740       113384 ▁▇▃▁▁
 6  -17804       10711.     16289      21716       113427 ▂▇▁▁▁
 7  -18045       13149.     18933      24285.      114298 ▁▇▁▁▁
 8  -17487       14415      20448      26533       113314 ▁▇▁▁▁
 9    8400       25700      33200      42400       250000 ▇▁▁▁▁
10       0           0.145      0.3        0.498        1 ▇▇▅▂▁
11       0.0106      0.612      0.773      0.889        1 ▁▁▂▆▇
12       1          18         20         22           35 ▁▁▇▃▁
13       2          23         25         27           36 ▁▁▂▇▂
14     100         440        480        520          799 ▁▁▇▃▁
15     100         540        590        630          800 ▁▁▂▇▂
16     100         440        475        520          799 ▁▁▇▃▁
17     100         550        588        630          800 ▁▁▂▇▂

For ease of use, the variables in school_tidy and scorecard_tidy have been renamed from the original names. I don’t yet have a mapping back to the original names, but you can reference the data_dictionary target to see information about the original names.

data_dictionary <- targets::tar_read("data_dictionary")
head(data_dictionary$info, 10)
# A tibble: 10 × 11
   name_of_data_element        dev_category developer_friendly_n…¹ api_data_type
   <chr>                       <chr>        <chr>                  <chr>        
 1 Unit ID for institution     root         id                     integer      
 2 8-digit OPE ID for institu… root         ope8_id                string       
 3 6-digit OPE ID for institu… root         ope6_id                string       
 4 Institution name            school       name                   autocomplete 
 5 City                        school       city                   autocomplete 
 6 State postcode              school       state                  string       
 7 ZIP code                    school       zip                    string       
 8 Accreditor for institution  school       accreditor             string       
 9 URL for institution's home… school       school_url             string       
10 URL for institution's net … school       price_calculator_url   string       
# ℹ abbreviated name: ¹​developer_friendly_name
# ℹ 7 more variables: data_type <chr>, index <chr>, variable_name <chr>,
#   source <chr>, shown_use_on_site <chr>, notes <chr>, has_labels <lgl>

About

Script for porting College Scorecard data to SQLite

License:Other


Languages

Language:R 100.0%