Normalization causing degenerate dimensionality reduction
suragnair opened this issue · comments
Since the linear model here is allowed to have non-zero coefficients, I encountered cases in which the coefficient was negative.
Line 137 in c3ab177
As a result the normalized jacquard distances included negative values as well as some outliers. The code removes positive outliers before performing diffusion maps, but in this case the negative outliers ended up making the diffusion maps to output a degenerate solution. This can be fixed by using:
model <- lm(y ~ 0 +x + I(x^2), data) and changing the coefficients to use only beta1 and beta2.