Applying RNN
hpoit opened this issue · comments
Could I receive some guidance on adjusting the net to correctly predict new targets from train.csv?
https://s3-sa-east-1.amazonaws.com/nu-static/workable-data-science/data-science-puzzle.zip
@hpoit
I would prefer not to to download and open zip files.
Perhaps if you provided a more specific question (ie. what are you trying to adjust? learning rate, size of net?) with some sample code and a sample of your data, I might be able to give you some guidance. :)
Hi Andy, this is how train.csv looks like
julia> showcols(train)
24976x108 DataFrames.DataFrame
| Col # | Name | Eltype | Missing |
|-------|-------------|------------|---------|
| 1 | x016399044a | Int64 | 0 |
| 2 | x023c68873b | UTF8String | 0 |
| 3 | x0342faceb5 | Int64 | 0 |
| 4 | x04e7268385 | Int64 | 0 |
| 5 | x06888ceac9 | Int64 | 0 |
| 6 | x072b7e8f27 | Float64 | 0 |
| 7 | x087235d61e | Int64 | 0 |
| 8 | x0b846350ef | Float64 | 0 |
| 9 | x0e2ab0831c | Float64 | 0 |
| 10 | x12eda2d982 | Float64 | 0 |
| 11 | x136c1727c3 | Float64 | 0 |
| 12 | x173b6590ae | Float64 | 0 |
| 13 | x174825d438 | Int64 | 0 |
| 14 | x1f222e3669 | Float64 | 0 |
| 15 | x1f3058af83 | Int64 | 0 |
| 16 | x1fa099bb01 | Int64 | 0 |
| 17 | x20f1afc5c7 | Float64 | 0 |
| 18 | x253eb5ef11 | Float64 | 0 |
| 19 | x25bbf0e7e7 | Int64 | 0 |
| 20 | x2719b72c0d | Float64 | 0 |
| 21 | x298ed82b22 | Float64 | 0 |
| 22 | x29bbd86997 | Float64 | 0 |
| 23 | x2a457d15d9 | Int64 | 0 |
| 24 | x2bc6ab42f7 | Float64 | 0 |
| 25 | x2d7fe4693a | Float64 | 0 |
| 26 | x2e874bc151 | Float64 | 0 |
| 27 | x361f93f4d1 | UTF8String | 0 |
| 28 | x384bec5dd1 | Int64 | 0 |
| 29 | x3df2300fa2 | Float64 | 0 |
| 30 | x3e200bf766 | Int64 | 0 |
| 31 | x3eb53ae932 | Float64 | 0 |
| 32 | x435dec85e2 | Float64 | 0 |
| 33 | x4468394575 | Float64 | 0 |
| 34 | x49756d8e0f | Float64 | 0 |
| 35 | x4fc17427c8 | Float64 | 0 |
| 36 | x55907cc1de | Float64 | 0 |
| 37 | x55cf3f7627 | Float64 | 0 |
| 38 | x56371466d7 | Int64 | 0 |
| 39 | x5b862c0a8f | Int64 | 0 |
| 40 | x5f360995ef | Int64 | 0 |
| 41 | x60ec1426ce | Float64 | 0 |
| 42 | x63bcf89b1d | Float64 | 0 |
| 43 | x6516422788 | Float64 | 0 |
| 44 | x65aed7dc1f | Int64 | 0 |
| 45 | x6db53d265a | Int64 | 0 |
| 46 | x7734c0c22f | Float64 | 0 |
| 47 | x7743f273c2 | Float64 | 0 |
| 48 | x779d13189e | Float64 | 0 |
| 49 | x77b3b41efa | Float64 | 0 |
| 50 | x7841b6a5b1 | Float64 | 0 |
| 51 | x789b5244a9 | Float64 | 0 |
| 52 | x7925993f42 | Float64 | 0 |
| 53 | x7cb7913148 | Int64 | 0 |
| 54 | x7fe6cb4c98 | Float64 | 0 |
| 55 | x8311343404 | Float64 | 0 |
| 56 | x87b982928b | Float64 | 0 |
| 57 | x8a21502326 | Float64 | 0 |
| 58 | x8c2e088a3d | Int64 | 0 |
| 59 | x8d0606b150 | UTF8String | 0 |
| 60 | x8de0382f02 | Float64 | 0 |
| 61 | x8f5f7c556a | Int64 | 0 |
| 62 | x91145d159d | UTF8String | 0 |
| 63 | x96c30c7eef | Float64 | 0 |
| 64 | x96e6f0be58 | Float64 | 0 |
| 65 | x98475257f7 | Float64 | 0 |
| 66 | x99d44111c9 | Float64 | 0 |
| 67 | x9a575e82a4 | Int64 | 0 |
| 68 | x9b6e0b36c2 | Float64 | 0 |
| 69 | a14fd026ce | Int64 | 0 |
| 70 | a24802caa5 | Float64 | 0 |
| 71 | aa69c802b6 | Float64 | 0 |
| 72 | abca7a848f | Int64 | 0 |
| 73 | ac826f0013 | Float64 | 0 |
| 74 | ae08d2297e | Int64 | 0 |
| 75 | aee1e4fc85 | Float64 | 0 |
| 76 | b4112a94a6 | Float64 | 0 |
| 77 | b709f75447 | Float64 | 0 |
| 78 | b835dfe10f | UTF8String | 0 |
| 79 | b9a487ac3c | Float64 | 0 |
| 80 | ba54a2a637 | Float64 | 0 |
| 81 | bdf934caa7 | Float64 | 0 |
| 82 | beb6e17af1 | Int64 | 0 |
| 83 | c0c3df65b1 | Int64 | 0 |
| 84 | c1b8ce2354 | Float64 | 0 |
| 85 | c58f611921 | Float64 | 0 |
| 86 | d035af6ffa | Float64 | 0 |
| 87 | d2c775fa99 | Int64 | 0 |
| 88 | d4d6566f9c | Float64 | 0 |
| 89 | dcfcbc2ea1 | Int64 | 0 |
| 90 | e0a0772df0 | Float64 | 0 |
| 91 | e16e640635 | UTF8String | 0 |
| 92 | e5efa4d39a | Float64 | 0 |
| 93 | e7ee22effb | Int64 | 0 |
| 94 | e86a2190c1 | Int64 | 0 |
| 95 | ea0f4a32e3 | Float64 | 0 |
| 96 | ed7e658a27 | Float64 | 0 |
| 97 | ee2ac696ff | Float64 | 0 |
| 98 | f013b60e50 | Float64 | 0 |
| 99 | f0a0febd35 | Float64 | 0 |
| 100 | f1f0984934 | UTF8String | 0 |
| 101 | f66b98dd69 | Float64 | 0 |
| 102 | fbf66c8021 | Float64 | 0 |
| 103 | fdf8628ca7 | Int64 | 0 |
| 104 | fe0318e273 | Float64 | 0 |
| 105 | fe8cdd80ba | Float64 | 0 |
| 106 | ffd1cdcfc1 | Float64 | 0 |
| 107 | id | Int64 | 0 |
| 108 | target | Float64 | 0 |
I'm sorry I understand this isn't an issue about the package, but a question on how to use it.
I have ran a hello world of Mocha.jl on MNIST but not yet for this one. My understanding is that RNNs use shared params over time (not space like CNNs) to extract patterns of sequences, and are proper for time series data, which I'm assuming this is for having originated from a bank.
I would need some guidance on how to apply this package on this 108 column by 24976 row matrix of variables of no particular meaning. The UTF8String columns are SHA-256 hashed categorical variables. I am pretty sure each row represents a client, so 24976 clients at the time. The hashed values are probably client name, address, e-mail, telephone, etc. so I wouldn't worry about them.
Generally, RNNs are used to predict the next variable(s) in a sequence. They are very useful for learning sequential sequence, but this data does not seem to be sequential. This case seems better suited to a standard DNN, or a perhaps a CNN.
Also note the "Hello World" of RNN is generally a char-RNN that learns the next character in a sentence. If you are just starting, I suggest having a look at Karpathy"s online example.
http://cs.stanford.edu/people/karpathy/recurrentjs/
Thanks Andy
Andy check these out
http://deeplearning.cs.cmu.edu/notes/shaoweiwang.pdf
http://stats.stackexchange.com/questions/127542/convolutional-neural-network-for-time-series
Maybe we could work together on this one. I haven't found any RNN ready in Julia for time series. My focus isn't financial, but anything that is worth being observed through time.
@johnmyleswhite @StefanKarpinski
This is the target column
julia> train[108] 24976-element DataArrays.DataArray{Float64,1}: 0.536383 0.426536 0.498649 1.01357 4.00876 1.77013 0.895514 0.348059 0.547179 2.38998 3.36091 1.4124 -0.0996988 0.40257 0.896381 1.97511 0.315973 1.35555 5.32482 0.591043 0.610452 1.45285 0.179439 3.16545 ⋮ 7.6247 0.340788 0.205181 2.04826 0.150869 0.775302 0.214532 0.1074 0.46036 0.241159 1.73751 0.390916 3.03904 2.71701 2.94019 0.309083 0.998051 0.762129 0.377102 3.88774 2.39431 0.50462 23.0103 2.53425
... which appears to be the output of a logit. This can be done on a CNN, sure, but with no use of shared params over time (memory).
I'm also comparing your rnn.jl code to https://github.com/kzahedi/RNN.jl/blob/master/src/RNN.jl and trying to understand the differences.
Sequences. Depending on your background you might be wondering: What makes Recurrent Networks so special? A glaring limitation of Vanilla Neural Networks (and also Convolutional Networks) is that their API is too constrained: they accept a fixed-sized vector as input (e.g. an image) and produce a fixed-sized vector as output (e.g. probabilities of different classes). Not only that: These models perform this mapping using a fixed amount of computational steps (e.g. the number of layers in the model). The core reason that recurrent nets are more exciting is that they allow us to operate over sequences of vectors: Sequences in the input, the output, or in the most general case both.
@Andy-P
I have an additional question about usage. How would you apply this library to doing time-series prediction, as in the link below?
http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/
I guess one of the main things I don't quite grasp in your example is the usage of the "wil" NNMatrix and how that would be used outside of the example (if it would even be necessary).
Thanks!
Chris
@pazzo83 This library is longer supported. I suggest you use MxNet.jl. There is a good example of an LSTM here