MongoRefine

This is an highly experimental data refining / wrangling package. It's inspired by OpenRefine and created as headless refine replacement. The key feature is NoSQL data support using MongoDB database collections and MongoDB engine to apply data Mutations.

Everything in this repository could change. Lack of documentation is not by mistake but is the result of experimentational nature of this code.

Requirements

This library uses pymongo and glom Python libraries. Make sure to have them installed or use pip install -r requirements.txt. You may need also rich and requests Python libraries to run examples in examples directory.

Installation

Use python setup.py install to install.

Examples

See more examples in examples directory.

Character case manipulations

from mongorefine import Refiner

# Collect to local database 'russianit' with collection 'pubprofile'
refiner = Refiner(dbname='resprj', collname='payments')
refiner.scan()
print(refiner.metadata())

# Get single column with project country
column = refiner.get_column('projectCountry')

# Let's measure how many records with this column exists
print(column.count())

# Lower case of column values using 'full' mode (slow but supports any encoding)
column.lower(mode='full')

# Iterate first 10 values 
print(column.items(0, 10))

# Upper case of column values using 'full' mode (slow but supports any encoding)
column.upper(mode='full')

# Iterate first 10 values 
print(column.items(0, 10))

# Upper case of column values using 'quick' mode (default). Uses built-in encoding in MongoDB database instance. 
column.upper(mode='title')

# Iterate first 10 values 
print(column.items(0, 10))

Key classes

Refiner

Base class to work with MongoDB database collection.

Supported functions:

scan - scans collection for columns metadata. Required for columns method
metadata - returns metadata collected by scan
cursor - returns collection cursor
get_rowset - returns RowSet object by query
get_column - returns Column object by name
columns - returns list of columns (require scan )
count - returns number of records
distinct - returns distinct values of column by provided name.
replace_substring - replaces substring inside the column
column_string_manipulation - applies one of string manipulations: trim, upper, lower and e.t.c
remove_column - removes columns by name
add_column - adds new column with static value
rename_column - rename single column
rename_columns - rename list of columns
transform_records - transforms every record in query and replace records
join_columns - joins several columns into the new single one
split_column - splits columns using divider char to the number of columns
transform_column - transforms every column value by query

Column

Column / field level manipulation class.

Supported functions:

count - returns count of records with this column
trim - removes space or chars provided from left, right or both sides of the string
upper - converts values to upper case string
lower - converts values to lower case string
title - converts values to title string
items - returns column items as an array. Could be iterated
metadata - returns column metadata
distinct - returns column distinct values
cursor - returns column only db cursor
transform - transforms column values
new - generates a new column from this one
concat - concatenate each column value with prefix and/or suffix
number - converts value to integer
string - converts value to string
todate - converts value to datetime
float_ - converts value to float
rename - renames a column
replace - replaces column value
drop - deletes this column
scan - scans column and generates it's metadata

RowSet

Class to manipulate with group/set of documents/rows/records.

Supported functions

count - returns count of records / rows in RowSet
star - marks records with star boolean value
flag - marks records with flag boolean value
remove - deletes all rows in RowSet
transform - transforms all records in RowSet

ivbeg / mongorefine