Muljayan / ny-stock-exchange-analysis

This is part of the coursework for CM704 Data Mining module of MSc. in Big Data Analytics of Robert Gordon University

Objectives

1. Preprocess the dataset as specified in the data mining process.
   1. Handle Missing Values and Outliers if any
   2. Produce Q-Q Plots and Histograms of the features, and apply the transformations if required.
   3. If it is required, apply suitable feature coding techniques.
   4. Scale and/or standardized the features, produce relevant graphs to show the scaling/ standardizing effect.
   5. If necessary, apply feature discretization, and produce a relevant graph to show the discretization
2. Perform Feature Engineering by executing the following task:
   1. Appropriately use PCA (Principal Component Analysis) or SVD (Singular Value Decomposition) for feature reduction.
   2. Identify significant and independent features using appropriate techniques. Show how you selected the features using suitable graphs.
3. Apply the following techniques to predict the value of Y (Estimated Shares Outstanding) for the test dataset (K =10)
   1. Linear Regression with Cross Validation
   2. Lasso Regression with Cross Validation
   3. Ridge Regression with Cross Validation
4. Using suitable evaluation matrices, compare the applicability of different regression models on the given Dataset.