README.txt for MLInterfaces, Jun 19 2015
hclustWidget has been added to demonstrate roles for shiny, fpc, and ggvis in
tuning cluster analysis.
README.txt for MLInterfaces, Sep 16 2008
The *B methods have been removed. The NAMESPACE is used to reduce
the number of symbols/classes/methods in play. Handling of
unsupervised learning via MLearn is in design.
README.txt for MLInterfaces, Sept 9 2007
Version 2.0 of MLInterfaces is imminent; I hope it will be available by
Sept 24. Version 1.11.15+ includes a number of new approaches to
simplify maintenance and extensibility
Basic approach:
A learner function defined in a package with a namespace, that uses a formula interface,
and that has a predict method, is the most reasonable case and the one that the
interface design is based on. A schema object is defined to provide access
to such learners and to define how their outputs are converted to standard
structures.
1) It is intended that all use will be through MLearn(formula, data, ...)
a) The simplest usage is MLearn(formula, data, learnerSchema, trainInds).
Instances of learnerSchema identify the package and function constituting
the learner, and include a converter function with standard calling sequence
that converts results of the learner function into a classifierOutput
or clusteringOutput instance. classifierOutput is currently defined, and
it manages the predictions (test and train) and any goodness of prediction
information produced. It also includes the call and the full representation
of the learner's output. When feature selection has been used in cross-validation,
the 'history' of feature selection is retained.
b) Cross-validation is carried out using
MLearn(formula, data, learnerSchema, xvalSpec, ...)
Instances of xvalSpec define the partitioning of the data. Of note is that
the former implementations of xval/xvalML did not generate MLOutput instances,
but the new implementation will generate objects identical in nature to those
produced in the train/test scheme described in 1a)
At present MLearn+xvalSpec yields has all functionality of previous versions (including
support for functionally specified partition) but the way of handling feature selection
is different. Instead of returning a set of feature scores, the function defining
feature selection needs to return a formula that includes the chosen features. An
example of this is given with help(MLearn), and also shown below. One feature
that is not retained with MLearn+xvalSpec is the xvalLoop generic with its support
for cluster computing. I will introduce that ASAP, when I have a working example.
c) So far, only supervised learning is handled in the new approach. Now that
cross-validation is working, I will start to deal with unsupervised methods.
d) A brief vignette that describes basic architecture with examples is in in
MLint_devel.Rnw. This will be elaborated soon.
2) Back-compatibility and deprecation. I have left the *B methods in place,
but they have been sequestered into a single file Bmethods.R. Eventually
they will be deprecated, possibly by Sept 24. The old version of MLearn,
which uses a string to identify the learner and a switch statement to
act, is retained for now but I do not intend to improve it in any way.
There was a problem with parameter capture in this design that motivated
the move to a schematic approach to learner specification. It can stay
indefinitely; we will just discourage use of strings and encourage the use of
schema objects to specify learners.
3) Benefits. The primary MLearn method definition is 18 lines long.
Converter functions are typically around 8 lines long, and vary primarily
because of the different approaches to the 'predict' generic taken in
various machine learning packages. We will not need a separate xval
wrapper.
4) Costs. The calling sequence is modified slightly, so we would have
MLearn(sp~CL+RW, crabs, ldaI, c(1:25, 101:125)) instead of
MLearn(sp~CL+RW, crabs, "lda", c(1:25, 101:125)). In most cases,
tuning parameters are passed using ... after the trainInd or
xvalSpec.
Some learners have such idiosyncratic implementations (no formula interface,
no predict method) that bridge methods need to be defined, and the schemas are
specified using closures to fix the tuning parameters: knnI(k=3,l=1) for example.
I would like to handle all tuning parameters uniformly, but this minority
of learners cannot be allowed to force major complexities on this package.
5) Examples
a) A simple formula+data frame test vs train exercise:
rf1 = MLearn(sp~CW+RW, data=crabs, randomForestI, kp, ntree=600 )
confuMat(rf1)
b) Cross-validation, LOO:
nn1cv = MLearn(sp~CW+RW, data=crabs[c(1:20,101:120),], nnetI, xvalSpec("LOO"), size=3, decay=.01 )
c) 5-fold cross-validation, partitions are balanced with respect to outcome class frequencies
nn2cv = MLearn(sp~CW+RW, data=crabs[c(1:20,101:120),], nnetI,
xvalSpec("LOG",5, balKfold.xvspec(5)), size=3, decay=.01 )
d) 5-fold cross-validation, feature selection using top 25% of features when ranked by
two-sample t. First define the fsFun:
fsFun.rowtQ3 = function(formula, data) {
# facilitation of a rowttests with a formula/data.frame takes a little work
mf = model.frame(formula, data)
mm = model.matrix(formula, data)
respind = attr( terms(formula, data=data), "response" )
x = mm
if ("(Intercept)" %in% colnames(x)) x = x[,-which(colnames(x) == "(Intercept)")]
y = mf[, respind]
respname = names(mf)[respind]
nuy = length(unique(y))
if (nuy > 2) warning("number of unique values of response exceeds 2")
#dm = t(data.matrix(x))
#dm = matrix(as.double(dm), nr=nrow(dm)) # rowttests seems fussy
ans = abs( rowttests(t(x), factor(y), tstatOnly=TRUE)[[1]] )
names(ans) = colnames(x)
ans = names( ans[ which(ans > quantile(ans, .75) ) ] )
btick = function(x) paste("`", x, "`", sep="") # support for nonsyntactic varnames
as.formula( paste(respname, paste(btick(ans), collapse="+"), sep="~"))
}
now deploy
library(golubEsets)
data(Golub_Train)
litg = Golub_Train[ 100:150, ]
g1 = MLearn(ALL.AML~. , litg, nnetI, xvalSpec("LOG",5, balKfold.xvspec(5), fsFun=fsFun.rowtQ3), size=3, decay=.01 )
confuMat(g1)
fsHistory(g1)
the fsFun is messy, and the acceptance criterion can be factored out to give more flexibility, eventually