DEPRECATED
This package is deprecated. Please use TableTransforms.jl instead.
MLPreprocessing
| Package Status | Package Evaluator | Build Status |
|---|---|---|
 |
  |
  |
Overview
Utility package that provides end user friendly methods for feature scalings and polynomial
basis expansion. Feature scalings work on Matrix, Vector and DataFrames. It is possible to
have observations stored as columns or rows of a matrix. In order to distinguish between these cases
one can provide the parameter obsdim, where obsdim=1 corresponds to "observations as rows" and
obsdim=2 to "observations as columns". Transformations can be computed on a subset
of columns/rows by defining a vector operate_on.
StandardScaler
Standardization of data sets result in variables with a mean of 0 and variance of 1.
A common use case would be to fit a StandardScaler to the training data and later
apply the same transformation to the test data. StandardScaler is used with the
functions fit(), transform() and fit_transform() as shown below.
fit(StandardScaler, X[, μ, σ; obsdim, operate_on])
fit_transform(StandardScaler, X[, μ, σ; obsdim, operate_on])X : Data of type Matrix or DataFrame.
μ : Vector or scalar describing the translation.
Defaults to mean(X; dims=obsdim)
σ : Vector or scalar describing the scale.
Defaults to std(X; dims=obsdim)
obsdim : Specify which axis corresponds to observations.
Defaults to obsdim=2 (observations are columns of matrix)
For DataFrames obsdim is obsolete and rescaling occurs
column wise.
operate_on: Specify the indices of columns or rows to be centered.
Defaults to all columns/rows.
For DataFrames this must be a vector of symbols, not indices.
E.g. operate_on=[1,3] will perform centering on columns
with index 1 and 3 only (if obsdim=1, else rows 1 and 3)
Note on DataFrames:
Columns containing missing values are skipped.
Columns containing non numeric elements are skipped.
Examples:
Xtrain = rand(100, 4)
Xtest = rand(10, 4)
x = rand(4)
Dtrain = DataFrame(A=rand(10), B=collect(1:10), C=[string(x) for x in 1:10])
Dtest = DataFrame(A=rand(10), B=collect(1:10), C=[string(x) for x in 1:10])
scaler = fit(StandardScaler, Xtrain)
scaler = fit(StandardScaler, Xtrain, obsdim=1)
scaler = fit(StandardScaler, Xtrain, obsdim=1, operate_on=[1,3])
transform(Xtest, scaler)
transform!(Xtest, scaler)
transform(x, scaler)
transform!(x, scaler)
scaler = fit(StandardScaler, Dtrain)
scaler = fit(StandardScaler, Dtrain, operate_on=[:A,:B])
transform(Dtest, scaler)
transform!(Dtest, scaler)
Xscaled, scaler = fit_transform(StandardScaler, X, obsdim=1, operate_on=[1,2,4])
scaler = fit_transform!(StandardScaler, X, obsdim=1, operate_on=[1,2,4])Note that for transform! the data matrix X has to be of type <: AbstractFloat
as the scaling occurs inplace. (E.g. cannot be of type Matrix{Int64}). This is not
the case for transform however.
For DataFrames transform! can be used on columns of type <: Integer.
FixedRangeScaler
FixedRangeScaler is used with the functions fit(), transform() and fit_transform()
to scale data in a Matrix X or DataFrame to a fixed range [lower:upper].
After fitting a FixedRangeScaler to one data set, it can be used to perform the same
transformation to a new set of data. E.g. fit the FixedRangeScaler to your training
data and then apply the scaling to the test data at a later stage. (See examples below).
fit(FixedRangeScaler, X[, lower, upper; obsdim, operate_on])
fit_transform(FixedRangeScaler, X[, lower, upper; obsdim, operate_on])X : Data of type Matrix or DataFrame.
lower : (Scalar) Lower limit of new range.
Defaults to 0.
upper : (Scalar) Upper limit of new range.
Defaults to 1.
obsdim : Specify which axis corresponds to observations.
Defaults to obsdim=2 (observations are columns of matrix)
For DataFrames obsdim is obsolete and rescaling occurs
column wise.
operate_on: Specify the indices of columns or rows to be centered.
Defaults to all columns/rows.
For DataFrames this must be a vector of symbols, not indices.
E.g. operate_on=[1,3] will perform centering on columns
with index 1 and 3 only (if obsdim=1, else rows 1 and 3)
Note on DataFrames:
Columns containing NA values are skipped.
Columns containing non numeric elements are skipped.
Examples:
Xtrain = rand(100, 4)
Xtest = rand(10, 4)
x = rand(10)
D = DataFrame(A=rand(10), B=collect(1:10), C=[string(x) for x in 1:10])
scaler = fit(FixedRangeScaler, Xtrain)
scaler = fit(FixedRangeScaler, Xtrain, -1, 1)
scaler = fit(FixedRangeScaler, Xtrain, -1, 1, obsdim=1)
scaler = fit(FixedRangeScaler, Xtrain, -1, 1, obsdim=1, operate_on=[1,3])
scaler = fit(FixedRangeScaler, D, -1, 1, operate_on=[:A,:B])
Xscaled = transform(Xtest, scaler)
transform!(Xtest, scaler)
Xscaled, scaler = fit_transform(FixedRangeScaler, X, -1, 1, obsdim=1, operate_on=[1,2,4])
scaler = fit_transform!(FixedRangeScaler, X, -1, 1, obsdim=1, operate_on=[1,2,4])Lower Level Functions
The lower level functions on which StandardScaler and FixedRangeScaler are built on can also
be used seperately.
center!()
μ = center!(X[, μ; obsdim, operate_on])Shift X along obsdim by μ according to X = X - μ
where X is of type Matrix or Vector and D of type DataFrame.
fixedrange!()
lower, upper, xmin, xmax = fixedrange!(X[, lower, upper, xmin, xmax; obsdim, operate_on])Normalize X or D along obsdim to the interval [lower:upper]
where X is of type Matrix or Vector and D of type DataFrame.
If lower and upper are omitted the default range is [0:1].
standardize!()
μ, σ = standardize!(X[, μ, σ; obsdim, operate_on])Standardize X along obsdim according to X = (X - μ) / σ.
If μ and σ are omitted they are computed such that variables have a mean of zero.
Polynomial Basis Expansion
M = expand_poly(x[, degree=5, obsdim]) Perform a polynomial basis expansion of the given degree for the vector x.
julia> expand_poly(1:5, degree=3)
3×5 Array{Float64,2}:
1.0 2.0 3.0 4.0 5.0
1.0 4.0 9.0 16.0 25.0
1.0 8.0 27.0 64.0 125.0
julia> expand_poly(1:5, degree=3, obsdim=1)
5×3 Array{Float64,2}:
1.0 1.0 1.0
2.0 4.0 8.0
3.0 9.0 27.0
4.0 16.0 64.0
5.0 25.0 125.0
julia> expand_poly(1:5, 3, ObsDim.First()); # same but type-stable