Richman

Pieter, Rudolf, Laenita, Vernike

29/04/2020

0.1 I might be wrong

Everything in this document is not 100% correct, this is what I think it does using my own discretion and googles help.

0.2 What is an Autoencoder

An auto encoder is is an unsupervised machine learning technique, it is trained by using supervised models.Autoencoders are a family of neural networks for which the input is the same as the output. They work by compressing the input into a latent-space representation, and then reconstructing the output from this representation.

0.3 What is Keras

Keras is a high-level neural networks API developed with a focus on enabling fast experimentation. This means that Keras is appropriate for building essentially any deep learning model, from a memory network to a neural Turing machine.

0.4 prerequisite

library(keras)
library(data.table)
library(dplyr)
library(ggplot2)
library(reshape2)
library(gnm)
library(HMDHFDplus)
library(stringr)
library(forecast)

0.5 Data manupilatoin

loading data and then adding a new column by taking the logs of male mx and naming it logmx.

dat = HMDHFDplus::readHMD("GBRTENW.Mx_1x1.txt") %>% 
  data.table
dat[,logmx:=log(Male)]
head(dat)

##    Year Age   Female     Male    Total OpenInterval     logmx
## 1: 1841   0 0.136067 0.169189 0.152777        FALSE -1.776739
## 2: 1841   1 0.059577 0.063208 0.061386        FALSE -2.761324
## 3: 1841   2 0.036406 0.036976 0.036689        FALSE -3.297486
## 4: 1841   3 0.024913 0.026055 0.025480        FALSE -3.647546
## 5: 1841   4 0.018457 0.019089 0.018772        FALSE -3.958643
## 6: 1841   5 0.013967 0.014279 0.014123        FALSE -4.248965

0.6 Functions

Scale min max function:

The first function defines min_dat as the minimum for the data “dat”, etc with the function max_dat for the maximum. The dat_scale is the scaled function for this code, by subtracting the observation with the min_dat observation and then dividing by the difference from the maximum observation and the minimum observation.

scale z function:

obtaining the mean and the sd from the data, scaling the observation in respect to the mean and sd.

Each function has will have two outputs, one for the data and one for the test data, unsure why but will look at it.

scale_min_max = function(dat,dat_test)  {
  min_dat = min(dat)
  max_dat = max(dat)
  dat_scaled=(dat-min_dat)/(max_dat-min_dat)
  dat_scaled_test = (dat_test-min_dat)/(max_dat-min_dat)
  return(list(train = dat_scaled, test = dat_scaled_test, min = min_dat, max=max_dat))
}

scale_z = function(dat,dat_test)  {
  mean_dat = mean(dat)
  sd_dat = sd(dat)
  dat_scaled=(dat-mean_dat)/(sd_dat)
  dat_scaled_test = (dat_test-mean_dat)/(sd_dat)
  return(list(train = dat_scaled, test = dat_scaled_test, mean_dat = mean_dat, sd_dat=sd_dat))
}

0.7 Training and testing data

We only take the values of the data that is higher than year 1949 and less than the age of 100 and make it our new dataset. We specify the training data set from our new dataset by only taking the years smaller than 2000 and our test dataset will be for the years after and including 2000.

Next we take the scaled values using the min_max scale of the observations of the logmx values of both the test and training dataset. We than add the scaled mx vales of each respective dataset back.

The train_rates that is specifed, the function dcast works as follows: A matrix/table is formed with the first column as year going from 1950 to 2016 top to bottom, for each specified year row the following columns have the scaled log mx of the training data from age 0-99. The function dcast is a shortcut to change the data format in the table.

dat = dat[Year>1949 & Age<100]
train = dat[Year < 2000]
test = dat[Year >= 2000]

scaled = scale_min_max(train$logmx, test$logmx)

train$mx_scale = scaled$train #adding to resspective datasets
test$mx_scale = scaled$test
train_rates = train %>%
  dcast.data.table(Year~Age, value.var = "mx_scale")
head(train_rates)

##    Year         0         1         2         3         4         5         6
## 1: 1950 0.6356891 0.3403410 0.2917020 0.2561202 0.2274679 0.2451742 0.2218990
## 2: 1951 0.6377657 0.3539856 0.2882443 0.2550938 0.2358810 0.2258669 0.2054003
## 3: 1952 0.6288870 0.3411003 0.2809043 0.2441541 0.2216200 0.2032768 0.1892247
## 4: 1953 0.6254083 0.3297534 0.2716298 0.2460730 0.2218990 0.2118416 0.1945277
## 5: 1954 0.6193904 0.3098776 0.2483983 0.2265368 0.2006059 0.1880949 0.1754771
## 6: 1955 0.6179227 0.3026814 0.2587450 0.2166201 0.1966496 0.1756891 0.1830612
##            7         8         9        10        11        12        13
## 1: 0.2096842 0.2133573 0.1654901 0.1737664 0.1854130 0.1659537 0.1930900
## 2: 0.1987314 0.1790265 0.1765329 0.1834567 0.1722475 0.1786148 0.1882840
## 3: 0.1865704 0.1664154 0.1693700 0.1587866 0.1621893 0.1769525 0.1555301
## 4: 0.1792318 0.1585394 0.1577948 0.1442998 0.1481716 0.1442998 0.1798453
## 5: 0.1668751 0.1428834 0.1402872 0.1570450 0.1302080 0.1495222 0.1636160
## 6: 0.1859932 0.1590331 0.1327373 0.1376283 0.1442998 0.1638520 0.1597695
##           14        15        16        17        18        19        20
## 1: 0.1948842 0.2232834 0.2435833 0.2511041 0.2713632 0.2758103 0.2811489
## 2: 0.2006059 0.2085894 0.2249221 0.2235582 0.2391459 0.2642897 0.2856953
## 3: 0.1971737 0.2101501 0.2107682 0.2352656 0.2440402 0.2516374 0.2788450
## 4: 0.1790265 0.1903429 0.2058846 0.2269368 0.2261353 0.2447220 0.2550938
## 5: 0.1731180 0.1918166 0.2155924 0.2179276 0.2302150 0.2293069 0.2509971
## 6: 0.1808604 0.2141074 0.2126021 0.2306019 0.2415036 0.2393840 0.2612113
##           21        22        23        24        25        26        27
## 1: 0.2779266 0.2834854 0.2887015 0.2980292 0.2919238 0.3046730 0.3023459
## 2: 0.2746073 0.2928065 0.2746937 0.2822830 0.2917760 0.2955560 0.2871701
## 3: 0.2812303 0.2790112 0.2763219 0.2813929 0.2681125 0.2727778 0.2752105
## 4: 0.2726899 0.2690251 0.2757248 0.2631453 0.2655162 0.2642897 0.2684784
## 5: 0.2619890 0.2623758 0.2717185 0.2513177 0.2584453 0.2582451 0.2493801
## 6: 0.2642897 0.2706490 0.2589443 0.2658909 0.2692974 0.2558133 0.2475183
##           28        29        30        31        32        33        34
## 1: 0.3002429 0.3058510 0.3248773 0.2826049 0.3075306 0.3205192 0.3199478
## 2: 0.2838833 0.2964811 0.3024802 0.3063706 0.3029490 0.3101284 0.3234437
## 3: 0.2735656 0.2830861 0.2813929 0.2983780 0.2992108 0.3049359 0.3072739
## 4: 0.2837243 0.2808227 0.2900619 0.2747800 0.2933182 0.3029490 0.2873242
## 5: 0.2842797 0.2639096 0.2829259 0.2810674 0.2829259 0.2949110 0.2973282
## 6: 0.2599357 0.2775065 0.2746937 0.2631453 0.2867839 0.2860854 0.2910340
##           35        36        37        38        39        40        41
## 1: 0.3355726 0.3298058 0.3353237 0.3481532 0.3569745 0.3659761 0.3781509
## 2: 0.3313664 0.3321898 0.3372994 0.3486401 0.3493885 0.3654843 0.3745082
## 3: 0.3219911 0.3194886 0.3351242 0.3439020 0.3491249 0.3514314 0.3664282
## 4: 0.3108151 0.3131549 0.3287530 0.3317789 0.3444539 0.3479311 0.3566053
## 5: 0.2985173 0.3070165 0.3194310 0.3317789 0.3438098 0.3481088 0.3605607
## 6: 0.3067586 0.2959839 0.3122372 0.3246031 0.3464994 0.3560285 0.3555732
##           42        43        44        45        46        47        48
## 1: 0.3902415 0.3949853 0.4070679 0.4248317 0.4337807 0.4517532 0.4607288
## 2: 0.3968557 0.3959100 0.4052849 0.4291097 0.4339439 0.4463127 0.4583361
## 3: 0.3796617 0.3891454 0.3994778 0.4126220 0.4291310 0.4413169 0.4571200
## 4: 0.3766191 0.3812836 0.3914766 0.4117778 0.4281040 0.4338828 0.4518399
## 5: 0.3827524 0.3801607 0.3991429 0.4097403 0.4231808 0.4351604 0.4435444
## 6: 0.3749267 0.3780156 0.3916263 0.4015480 0.4224140 0.4327139 0.4479049
##           49        50        51        52        53        54        55
## 1: 0.4792101 0.4864730 0.4892123 0.5059927 0.5221611 0.5310244 0.5359129
## 2: 0.4824490 0.4905385 0.5022587 0.5134196 0.5294743 0.5373357 0.5417357
## 3: 0.4641769 0.4774787 0.4942355 0.5086763 0.5140231 0.5284922 0.5343566
## 4: 0.4644401 0.4753702 0.4854807 0.5057697 0.5213332 0.5258384 0.5322607
## 5: 0.4628200 0.4761934 0.4809147 0.5005441 0.5160371 0.5302010 0.5307873
## 6: 0.4632741 0.4742581 0.4841694 0.5023356 0.5127230 0.5286651 0.5384009
##           56        57        58        59        60        61        62
## 1: 0.5497138 0.5610744 0.5724363 0.5852744 0.5920987 0.6007551 0.6167817
## 2: 0.5577329 0.5696708 0.5828954 0.5909452 0.6020290 0.6135869 0.6291723
## 3: 0.5556959 0.5627301 0.5789078 0.5843602 0.5866264 0.5993982 0.6162947
## 4: 0.5500560 0.5636904 0.5743602 0.5853261 0.5918261 0.5973263 0.6183029
## 5: 0.5496781 0.5620172 0.5749923 0.5847663 0.5889972 0.5988644 0.6125293
## 6: 0.5459077 0.5621640 0.5754965 0.5814965 0.5910827 0.6000290 0.6146740
##           63        64        65        66        67        68        69
## 1: 0.6290958 0.6409872 0.6497720 0.6510361 0.6621406 0.6708778 0.6833272
## 2: 0.6355023 0.6475619 0.6573825 0.6605755 0.6753092 0.6849969 0.6945536
## 3: 0.6279670 0.6376307 0.6488485 0.6498873 0.6644487 0.6729161 0.6856236
## 4: 0.6300272 0.6381732 0.6444234 0.6490750 0.6675838 0.6773398 0.6848250
## 5: 0.6284612 0.6374245 0.6465662 0.6502954 0.6613414 0.6741688 0.6853232
## 6: 0.6268548 0.6381796 0.6472083 0.6517548 0.6673382 0.6724030 0.6859730
##           70        71        72        73        74        75        76
## 1: 0.6891376 0.6968221 0.7149960 0.7236977 0.7329225 0.7430821 0.7519847
## 2: 0.7000821 0.7078837 0.7241776 0.7309137 0.7454608 0.7514078 0.7662975
## 3: 0.6886360 0.6962200 0.7151045 0.7219715 0.7310684 0.7394937 0.7542435
## 4: 0.6921079 0.6957041 0.7149864 0.7270210 0.7313068 0.7403406 0.7546599
## 5: 0.6936236 0.6976097 0.7100300 0.7246305 0.7334544 0.7399530 0.7494547
## 6: 0.6942650 0.6995528 0.7155215 0.7233756 0.7358098 0.7447257 0.7554327
##           77        78        79        80        81        82        83
## 1: 0.7621880 0.7696863 0.7877196 0.7917675 0.7975538 0.8140077 0.8240725
## 2: 0.7726214 0.7864264 0.7948175 0.8014987 0.8081714 0.8215844 0.8330007
## 3: 0.7592657 0.7737228 0.7846458 0.7876056 0.7948377 0.8133033 0.8198399
## 4: 0.7574631 0.7735718 0.7833630 0.7889465 0.7967966 0.8129760 0.8194634
## 5: 0.7602223 0.7719252 0.7841855 0.7860631 0.7977350 0.8099881 0.8235271
## 6: 0.7647577 0.7753439 0.7899577 0.7927060 0.7992360 0.8133963 0.8229256
##           84        85        86        87        88        89        90
## 1: 0.8371691 0.8433746 0.8567774 0.8616965 0.8672325 0.8808917 0.8910789
## 2: 0.8461454 0.8507358 0.8666195 0.8764612 0.8799987 0.8878377 0.8954187
## 3: 0.8343163 0.8404715 0.8533044 0.8599919 0.8649045 0.8785918 0.8846597
## 4: 0.8314338 0.8389939 0.8527607 0.8581602 0.8664416 0.8736083 0.8803374
## 5: 0.8325885 0.8362177 0.8498793 0.8622164 0.8685994 0.8735643 0.8826654
## 6: 0.8365529 0.8462620 0.8545672 0.8627139 0.8712400 0.8843051 0.8857023
##           91        92        93        94        95        96        97
## 1: 0.8924089 0.9057682 0.9090777 0.9310690 0.9380756 0.9297472 0.9355948
## 2: 0.8990201 0.9051179 0.9282982 0.9296035 0.9203183 0.9352436 0.9382336
## 3: 0.8889953 0.9052910 0.9108876 0.9127358 0.9367530 0.9292258 0.9374399
## 4: 0.8817043 0.8938290 0.8967353 0.9099509 0.9162425 0.9373110 0.9492758
## 5: 0.8909937 0.8985886 0.9155038 0.9153234 0.9358767 0.9432458 0.9215413
## 6: 0.8958020 0.9055831 0.9102336 0.9196639 0.9292569 0.9384446 0.9372379
##           98        99
## 1: 0.9458013 0.9443410
## 2: 0.9510170 0.9899509
## 3: 0.9333059 0.9506794
## 4: 0.9607723 0.9582700
## 5: 0.9761418 0.9748660
## 6: 0.9532125 1.0000000

0.8 Generalized non-linear model

Fitting a generalized non-linear model to the train dataset using the Poisson log link and specifying the nonlinear interaction between age and year. After that we, we feed our data into the model and predict the dependent variable mx and adding it to the training dataset again as seen in the table below Pred_lc is the predicted male mx from the lee carter model.

fit = gnm(Male~ as.factor(Age) + Mult(as.factor(Age), as.factor(Year), inst = 1) -1,family = poisson(link = "log"),data=train)

## Initialising
## Running start-up iterations..
## Running main iterations........
## Done

train[,pred_LC:=predict(fit, type="response")]
head(train)

##    Year Age   Female     Male    Total OpenInterval     logmx  mx_scale
## 1: 1950   0 0.025786 0.033686 0.029839        FALSE -3.390673 0.6356891
## 2: 1950   1 0.002251 0.002317 0.002285        FALSE -6.067482 0.3403410
## 3: 1950   2 0.001219 0.001491 0.001358        FALSE -6.508308 0.2917020
## 4: 1950   3 0.000947 0.001080 0.001015        FALSE -6.830794 0.2561202
## 5: 1950   4 0.000713 0.000833 0.000774        FALSE -7.090477 0.2274679
## 6: 1950   5 0.000658 0.000978 0.000822        FALSE -6.930001 0.2451742
##         pred_LC
## 1: 0.0302345993
## 2: 0.0019081142
## 3: 0.0011729501
## 4: 0.0008928028
## 5: 0.0007514293
## 6: 0.0006883229

0.9 Lee carter model

A data table “coefs” is made, using the variables and coefficients from the GNM. We add in extra row. next we specify that the first 1-100 values as ax, 101-200 are bx and 201-250 are K.

We define the variables ax, bx and K from our coefs table.

and I cant remember Lee carter stuff in CS2 but i think the c1 and stuff is used to get the model and forecast k.

coefs = data.table(names = fit %>% coef %>% names, coef=fit %>% coef)
coefs[,row:=.I]
coefs[row %in% c(1:100),var:="ax"]
coefs[row %in% c(101:200),var:="bx"]
coefs[row %in% c(201:250),var:="k"]

ax =coefs[var == "ax"]$coef
bx =coefs[var == "bx"]$coef
k =coefs[var == "k"]$coef

c1 = mean(k)
c2 = sum(bx)
ax = ax+c1*bx
bx = bx/c2
k = (k-c1)*c2

0.10 Forecast Lee carter model

so form our values of K above we forecast 17 more observations ahead, the effects of the trend on mortality. We fit a lee carter model.

forecast_k  =k %>% forecast::forecast(17) # forecast k 17 observatons ahead
k_forecast = forecast_k[[2]] # this is the actual time series.

fitted = (ax+(bx)%*%t(k)) # lee carter model to all values.

fitted_test = (ax+(bx)%*%t(k_forecast)) %>% melt # lee caret model to forecasted k, the time series with melt ?
test$pred_LC =   fitted_test$value %>% exp # exponanting the values of the forecasted k 
results = data.table(model = "Lee-Carter", MSE_OutOfSample = test[,sum((Male - pred_LC)^2)])
results # mean square error of the predicted vs actual

##         model MSE_OutOfSample
## 1: Lee-Carter       0.2623647

0.11 Comparing

The graph shows the predicted values from the gnm to the actual values in the year 2016.

test[Year==2016] %>%  #plotting for the year 2016
ggplot(aes(x=Age, y = log(pred_LC),colour="Predicted"))+ 
  geom_point(size = 0.5, alpha=0.5)+
  facet_wrap(~Year)+
  geom_line(aes(x=Age, y=log(Male),colour="Actual")) + 
   theme(panel.background = element_rect(fill = "#ADD8E6", colour = "#6D9EC1",size = 2, linetype = "solid"))+
  ylab("Log(mx)")+
  ggtitle("Values of mx for each age")#plot of actual vs predicted

0.12 Setting up Network

x is a list of Numinputs and rates, these value that is specified is the train_rates specified as above from the dcast function, they are sampled without replacement and not adding the specified years and putting all that values as a matrix for numinputs and rates, they have the same value.

We specify how numinputs are defined, like activation function etc. The same for rates. Since the observations of Numinputs and Rates are the same, that is fed into the keras model, this is why its an autoencoder, the input and output layers are the same.

adam is the learning rate specified as 0.001 and the model is further compiled with this learning rate and then using the specified loss function that will be minimized mape(mean absolute percentage error)

x = list(NumInputs=as.matrix(train_rates[,c(-1),with=F]), Rates= as.matrix(train_rates[,c(-1),with=F])) # same value

NumInputs <- layer_input(shape = c(100), dtype = 'float32', name = 'NumInputs')

encode = NumInputs %>%  
  layer_dense(units = 4, activation = 'tanh', name = "EncodeRates1")


Rates = encode %>%  
  layer_dense(units = 100, activation = 'sigmoid', name = "Rates") 

model <- keras_model(
  inputs = c(NumInputs), 
  outputs = c(Rates))

adam = optimizer_adam(lr = 0.001)

model %>% compile(
  optimizer = adam,
  loss = "mape")

0.13 model 1

A learning rate schedule is specified for if n<20 is 0.01 and if n<40000 is 0.001 otherwise 0.0001 and i don’t know why. lr_sched is then specified as the schedule.

The model that is obtained from the greedy auto encode looks a pre determined model that will be use for the autoencoding.

The encode and Rates variables are specified for the layers in the neural network with their activation function and units in the layer. We will name this encoderates 1 and encode in the 2 layers of the inputs and for the output, Rates and DecodeRates 1.

A second Keras model is specified for this newly defined Numinputs and Rates, the same principles apply. A learning rate of 0.001 is defined. We specify the weights of the model from the encoderates1 and Rates for the model, this is basically for parameter reduction if I’am not mistaken. These layers are not trainable, they are used as is.

lr_schedule = function(n,lr) {
  ifelse(n<20,0.01,ifelse(n<40000,0.001,0.0001)) } 
lr_sched = callback_learning_rate_scheduler(schedule = lr_schedule) # the schedule

model = load_model_hdf5("greedy_auto_encode1.mod") # autoencode model.

encode = NumInputs %>%  
  layer_dense(units = 4, activation = 'tanh', name = "EncodeRates1") %>% 
  layer_dense(units = 2, activation = 'linear', name = "encode")
Rates = encode %>%  
  layer_dense(units =4, activation = 'tanh', name = "DecodeRates1") %>% 
  layer_dense(units = 100, activation = 'sigmoid', name = "Rates") 

model_stage2 <- keras_model(
  inputs = c(NumInputs), 
  outputs = c(Rates))

adam = optimizer_adam(lr = 0.001)

set_weights(get_layer(model_stage2, "EncodeRates1"), get_weights(get_layer(model, "EncodeRates1")))
set_weights(get_layer(model_stage2, "Rates"), get_weights(get_layer(model, "Rates")))

layer = get_layer(model_stage2, "EncodeRates1")
layer$trainable = FALSE

layer = get_layer(model_stage2, "Rates")
layer$trainable = FALSE

0.14 model stage 2

The model is compiled with the learning rate and loss function again the same steps is followed in the previous chunk as before.

model_stage2 %>% compile(
  optimizer = adam,
  loss = "mape")

model_stage2 = load_model_hdf5("greedy_auto_encode2.mod")

encode = NumInputs %>%  
  layer_dense(units = 4, activation = 'tanh', name = "EncodeRates1") %>% 
  layer_dense(units = 2, activation = 'linear', name = "encode")

Rates = encode %>%  
  layer_dense(units =4, activation = 'tanh', name = "DecodeRates1") %>% 
  layer_dense(units = 100, activation = 'sigmoid', name = "Rates") 

model_stage3 <- keras_model(
  inputs = c(NumInputs), 
  outputs = c(Rates))

adam = optimizer_adam(lr = 0.001)

set_weights(get_layer(model_stage3, "EncodeRates1"), get_weights(get_layer(model_stage2, "EncodeRates1")))
set_weights(get_layer(model_stage3, "encode"), get_weights(get_layer(model_stage2, "encode")))
set_weights(get_layer(model_stage3, "DecodeRates1"), get_weights(get_layer(model_stage2, "DecodeRates1")))
set_weights(get_layer(model_stage3, "Rates"), get_weights(get_layer(model_stage2, "Rates")))

0.15 model stage 3

The model stage 3 will be done as usual.

model_stage3 %>% compile(
  optimizer = adam,
  loss = "mse")

model_stage3=load_model_hdf5("greedy_auto_encode3.mod")

model = model_stage3

0.16 Training matrix

A matrix/table pre_train is formed by predicting the log mx scaled from our model using the numinputs. We see for the head of the table, we have the predicted log mx scaled values for v1-v100 which i think is the age. the format is changed such that after the year coll um is added, the values are grouped by v1 for each year and so forth for each v. A column is added age for each v and year age 0-19. Another column is added mx_autencoder that is the same as value and i don’t know why that is done..

the last part of the chunk is as follows: we add the column mx_autoencode From the previous specified part of pred_train. That added column is then exponented and then a lot of stuff, look at the line specified as “values adapted”. After that formula is done and replaced with the mx_autencode column, these values look very similar to the Pred_lc values, hmmmmmm. A table is produced that has the predicted Sum of squares error of the Autoencode and pred_lc values in respect to the mx values of males listed as “SSE” in the chunk.

The table “train_metrics” is a table consisting of the MSE of both autencoder and pred_LC summed over each age for every year, an example is given. Delta is defined as Autoencode - pred_lc and added to the table of train_metrics.

pred_train = model %>% predict(x$NumInputs) %>% data.table #predicted log mx scaled
head(pred_train)

##           V1        V2        V3        V4        V5        V6        V7
## 1: 0.6467944 0.3513709 0.2940219 0.2594874 0.2360045 0.2323276 0.2142464
## 2: 0.6462150 0.3503927 0.2934213 0.2593034 0.2364643 0.2322569 0.2148480
## 3: 0.6304219 0.3309096 0.2745384 0.2414451 0.2188267 0.2126442 0.1967992
## 4: 0.6229088 0.3217203 0.2660468 0.2338167 0.2119249 0.2044829 0.1899060
## 5: 0.6110839 0.3085161 0.2527739 0.2206441 0.1978216 0.1900768 0.1754376
## 6: 0.6130433 0.3103139 0.2550939 0.2234713 0.2016609 0.1933188 0.1795096
##           V8        V9       V10       V11       V12       V13       V14
## 1: 0.2058837 0.1896771 0.1727161 0.1781367 0.1684881 0.1807704 0.1869070
## 2: 0.2063770 0.1899423 0.1736375 0.1781390 0.1684831 0.1803400 0.1864357
## 3: 0.1882235 0.1737578 0.1597397 0.1622582 0.1556360 0.1659728 0.1739339
## 4: 0.1812098 0.1673449 0.1548550 0.1557556 0.1503290 0.1596603 0.1683025
## 5: 0.1669549 0.1549488 0.1428759 0.1441467 0.1408114 0.1498341 0.1598022
## 6: 0.1708353 0.1581275 0.1467597 0.1468230 0.1430055 0.1515816 0.1611866
##          V15       V16       V17       V18       V19       V20       V21
## 1: 0.1972901 0.2176347 0.2281940 0.2265781 0.2422821 0.2553520 0.2754054
## 2: 0.1968053 0.2167621 0.2287828 0.2279591 0.2432730 0.2559257 0.2750873
## 3: 0.1862832 0.2058845 0.2235894 0.2309587 0.2449367 0.2551432 0.2703034
## 4: 0.1814149 0.2004851 0.2219457 0.2336679 0.2466576 0.2553778 0.2679387
## 5: 0.1743146 0.1938323 0.2166292 0.2331619 0.2459053 0.2535413 0.2647792
## 6: 0.1753475 0.1942569 0.2186760 0.2352618 0.2474894 0.2547770 0.2650865
##          V22       V23       V24       V25       V26       V27       V28
## 1: 0.2814087 0.2864996 0.2877319 0.2884917 0.2874861 0.2885768 0.2844077
## 2: 0.2806954 0.2854641 0.2863238 0.2868633 0.2858961 0.2869846 0.2829468
## 3: 0.2740743 0.2772328 0.2766849 0.2763826 0.2755393 0.2767911 0.2735299
## 4: 0.2705452 0.2727135 0.2712191 0.2703561 0.2696019 0.2709180 0.2681078
## 5: 0.2667522 0.2683176 0.2664840 0.2654135 0.2646782 0.2661244 0.2636534
## 6: 0.2666739 0.2679522 0.2657078 0.2643991 0.2637101 0.2651204 0.2627407
##          V29       V30       V31       V32       V33       V34       V35
## 1: 0.2927912 0.3012238 0.3067211 0.2978728 0.3079193 0.3172473 0.3196200
## 2: 0.2910424 0.2992925 0.3047984 0.2961967 0.3064156 0.3154572 0.3182470
## 3: 0.2803981 0.2873460 0.2927154 0.2864051 0.2963278 0.3043190 0.3083588
## 4: 0.2741849 0.2804130 0.2857285 0.2806105 0.2905689 0.2978400 0.3028018
## 5: 0.2693549 0.2749375 0.2801260 0.2762927 0.2856657 0.2926939 0.2977812
## 6: 0.2681526 0.2736512 0.2788776 0.2750551 0.2847935 0.2914979 0.2971052
##          V36       V37       V38       V39       V40       V41       V42
## 1: 0.3341047 0.3316818 0.3411655 0.3538644 0.3604109 0.3665736 0.3751164
## 2: 0.3327044 0.3306004 0.3401718 0.3528497 0.3596147 0.3661409 0.3747444
## 3: 0.3218860 0.3217654 0.3313583 0.3435981 0.3514558 0.3590403 0.3681788
## 4: 0.3159043 0.3169272 0.3266107 0.3386396 0.3471716 0.3555623 0.3649873
## 5: 0.3102037 0.3121283 0.3216437 0.3333641 0.3423041 0.3507775 0.3604978
## 6: 0.3096500 0.3117846 0.3214562 0.3332189 0.3423569 0.3513077 0.3610411
##          V43       V44       V45       V46       V47       V48       V49
## 1: 0.3940096 0.3963099 0.4048935 0.4199245 0.4317724 0.4446062 0.4577484
## 2: 0.3936296 0.3962395 0.4049999 0.4200925 0.4319733 0.4448720 0.4579318
## 3: 0.3860874 0.3903563 0.3998707 0.4147801 0.4269023 0.4397744 0.4525099
## 4: 0.3824674 0.3877524 0.3977654 0.4126562 0.4249142 0.4378389 0.4503517
## 5: 0.3772109 0.3831714 0.3934147 0.4080260 0.4204053 0.4331690 0.4455901
## 6: 0.3779235 0.3841531 0.3945883 0.4093515 0.4217487 0.4346415 0.4469697
##          V50       V51       V52       V53       V54       V55       V56
## 1: 0.4735135 0.4831736 0.4923094 0.5102655 0.5235850 0.5332921 0.5387620
## 2: 0.4737093 0.4834346 0.4925757 0.5104532 0.5237315 0.5335789 0.5391617
## 3: 0.4681050 0.4783850 0.4877530 0.5049542 0.5181529 0.5285267 0.5350351
## 4: 0.4658796 0.4764635 0.4859343 0.5027637 0.5158859 0.5266272 0.5336492
## 5: 0.4609404 0.4718320 0.4814726 0.4979158 0.5110563 0.5219261 0.5294599
## 6: 0.4623798 0.4732906 0.4828990 0.4993237 0.5124025 0.5234397 0.5310116
##          V57       V58       V59       V60       V61       V62       V63
## 1: 0.5563105 0.5682883 0.5812719 0.5903254 0.5965381 0.6055831 0.6239564
## 2: 0.5565612 0.5685199 0.5815220 0.5906163 0.5969127 0.6059348 0.6241558
## 3: 0.5515491 0.5635912 0.5765154 0.5860323 0.5931602 0.6024623 0.6195923
## 4: 0.5496298 0.5616884 0.5745967 0.5843375 0.5919101 0.6013104 0.6178131
## 5: 0.5450314 0.5571949 0.5699971 0.5799941 0.5880674 0.5977435 0.6136725
## 6: 0.5464709 0.5585834 0.5714373 0.5814317 0.5895045 0.5990831 0.6149339
##          V64       V65       V66       V67       V68       V69       V70
## 1: 0.6353061 0.6468638 0.6545454 0.6560551 0.6721210 0.6803191 0.6913426
## 2: 0.6354611 0.6470052 0.6547137 0.6563692 0.6723478 0.6805850 0.6915913
## 3: 0.6308429 0.6424891 0.6505933 0.6534775 0.6688582 0.6773592 0.6883062
## 4: 0.6289946 0.6406710 0.6489742 0.6525393 0.6675718 0.6762267 0.6871302
## 5: 0.6249005 0.6366867 0.6452562 0.6495198 0.6642444 0.6730315 0.6839211
## 6: 0.6260884 0.6378297 0.6463743 0.6506782 0.6653531 0.6741645 0.6850334
##          V71       V72       V73       V74       V75       V76       V77
## 1: 0.6966033 0.7016622 0.7196991 0.7285808 0.7383841 0.7456174 0.7577139
## 2: 0.6969367 0.7020497 0.7199600 0.7288628 0.7386590 0.7459139 0.7579452
## 3: 0.6943899 0.7001214 0.7170365 0.7261776 0.7360271 0.7435233 0.7552193
## 4: 0.6936201 0.6996742 0.7160295 0.7252954 0.7351609 0.7427837 0.7542677
## 5: 0.6908375 0.6972843 0.7130873 0.7225034 0.7324263 0.7402013 0.7515451
## 6: 0.6919680 0.6983972 0.7141544 0.7235647 0.7334642 0.7412337 0.7525191
##          V78       V79       V80       V81       V82       V83       V84
## 1: 0.7646455 0.7778105 0.7900686 0.7928755 0.8001782 0.8167810 0.8264642
## 2: 0.7648276 0.7778720 0.7900289 0.7929432 0.8002304 0.8167042 0.8263356
## 3: 0.7623053 0.7747540 0.7864969 0.7904493 0.7978102 0.8133799 0.8229014
## 4: 0.7613928 0.7734587 0.7849160 0.7894327 0.7968098 0.8118500 0.8212686
## 5: 0.7589419 0.7707693 0.7820956 0.7872449 0.7947153 0.8092749 0.8187099
## 6: 0.7597817 0.7714943 0.7827066 0.7878597 0.7952858 0.8097741 0.8191300
##          V85       V86       V87       V88       V89       V90       V91
## 1: 0.8403250 0.8460941 0.8595228 0.8684912 0.8743964 0.8851717 0.8914312
## 2: 0.8400964 0.8458945 0.8592337 0.8681735 0.8741284 0.8848233 0.8910727
## 3: 0.8360724 0.8422504 0.8550478 0.8639109 0.8702356 0.8805981 0.8870581
## 4: 0.8340722 0.8404480 0.8529066 0.8617018 0.8682425 0.8783652 0.8849070
## 5: 0.8312269 0.8378584 0.8500440 0.8588296 0.8655734 0.8755784 0.8823193
## 6: 0.8315674 0.8381811 0.8502969 0.8590403 0.8658103 0.8757190 0.8823968
##          V92       V93       V94       V95       V96       V97       V98
## 1: 0.8954905 0.9078665 0.9158488 0.9230182 0.9337584 0.9403613 0.9427309
## 2: 0.8952446 0.9075489 0.9154457 0.9226738 0.9333390 0.9400297 0.9423609
## 3: 0.8918336 0.9038699 0.9116219 0.9191104 0.9294442 0.9365052 0.9391181
## 4: 0.8900725 0.9019046 0.9094926 0.9171575 0.9272416 0.9345628 0.9372638
## 5: 0.8877519 0.8994992 0.9071443 0.9148947 0.9248368 0.9322728 0.9353126
## 6: 0.8879373 0.8995920 0.9070877 0.9149068 0.9247531 0.9322993 0.9351994
##          V99      V100
## 1: 0.9499943 0.9691490
## 2: 0.9497495 0.9684979
## 3: 0.9466217 0.9637784
## 4: 0.9449464 0.9607286
## 5: 0.9427894 0.9579377
## 6: 0.9429181 0.9573956

pred_train[,Year:=.I] # Adding in a collum of year from 1- 50
pred_train = pred_train %>% melt(id.vars="Year") %>% data.table() # changing the format to v1 for each year
head(pred_train)

##    Year variable     value
## 1:    1       V1 0.6467944
## 2:    2       V1 0.6462150
## 3:    3       V1 0.6304219
## 4:    4       V1 0.6229088
## 5:    5       V1 0.6110839
## 6:    6       V1 0.6130433

pred_train[,Age:=as.integer(str_sub(variable, 2))-1] # adding a collum age 0-19
head(pred_train)

##    Year variable     value Age
## 1:    1       V1 0.6467944   0
## 2:    2       V1 0.6462150   0
## 3:    3       V1 0.6304219   0
## 4:    4       V1 0.6229088   0
## 5:    5       V1 0.6110839   0
## 6:    6       V1 0.6130433   0

pred_train[,mx_autoencode:=value] # adds a colum mx_autoencoder that has the same value as value
head(pred_train)

##    Year variable     value Age mx_autoencode
## 1:    1       V1 0.6467944   0     0.6467944
## 2:    2       V1 0.6462150   0     0.6462150
## 3:    3       V1 0.6304219   0     0.6304219
## 4:    4       V1 0.6229088   0     0.6229088
## 5:    5       V1 0.6110839   0     0.6110839
## 6:    6       V1 0.6130433   0     0.6130433

pred_train %>% setkey(Year,Age)


train$mx_autoencode = pred_train$mx_autoencode #adding the values to the train dataset
head(train)

##    Year Age   Female     Male    Total OpenInterval     logmx  mx_scale
## 1: 1950   0 0.025786 0.033686 0.029839        FALSE -3.390673 0.6356891
## 2: 1950   1 0.002251 0.002317 0.002285        FALSE -6.067482 0.3403410
## 3: 1950   2 0.001219 0.001491 0.001358        FALSE -6.508308 0.2917020
## 4: 1950   3 0.000947 0.001080 0.001015        FALSE -6.830794 0.2561202
## 5: 1950   4 0.000713 0.000833 0.000774        FALSE -7.090477 0.2274679
## 6: 1950   5 0.000658 0.000978 0.000822        FALSE -6.930001 0.2451742
##         pred_LC mx_autoencode
## 1: 0.0302345993     0.6467944
## 2: 0.0019081142     0.3513709
## 3: 0.0011729501     0.2940219
## 4: 0.0008928028     0.2594874
## 5: 0.0007514293     0.2360045
## 6: 0.0006883229     0.2323276

train[,mx_autoencode:=exp(mx_autoencode*(scaled$max-scaled$min)+scaled$min)] #values adapted
train[,.(Autoencode = sum((Male-mx_autoencode)^2)/.N, Pred_LC = sum((Male-pred_LC)^2)/.N)] # MSE

##      Autoencode      Pred_LC
## 1: 0.0001259226 0.0001204623

train_metrics = train[,.(Autoencode = sum((Male-mx_autoencode)^2), Pred_LC = sum((Male-pred_LC)^2)), keyby = Year] # adding SSE to new table 
head(train_metrics) 

##    Year Autoencode     Pred_LC
## 1: 1950 0.02700372 0.013457625
## 2: 1951 0.03205880 0.034742071
## 3: 1952 0.01302112 0.007995853
## 4: 1953 0.01777636 0.018886777
## 5: 1954 0.05768625 0.050844405
## 6: 1955 0.09314369 0.075642035

train_metrics[, delta := Autoencode-Pred_LC]

train_metrics %>% # Plot of Delta 
  ggplot(aes(x=Year, y=delta))+ 
  geom_line() + 
  ggtitle("Plot of Delta for each year") + 
  xlab("Year")+
  ylab("Delta")+
  theme(panel.background = element_rect(fill = "#ADD8E6", colour = "#6D9EC1",size = 2, linetype = "solid"))+
  geom_hline(yintercept=0, linetype="dashed", color = "red")

0.17 Obtaining Codes

Codes model is specifed with inputs as NUminputs and outputs as encode with learning rate 0.01 and mape loss function.

The keras model predict V1 and V2 and placing it in a table with the corresponding year, it predicts something but i don’t know what. The two variables V1 and V2 is then set as a Time series from 1-50 (1950-1999)

dim1_forecasts forecasts V1 17 units ahead with drift and the same is done for dim2_forecasts.

codes_forecast table is created, it is the mean forecast of each V1 and V2 for the years 2000-2016. We Row bind these Codes_forecast such that the values continue from 1999 - 2016.

codes_model <- keras_model(
  inputs = c(NumInputs), 
  outputs = c(encode)) %>% compile(
    optimizer = adam,
    loss = "mape")

codes =  codes_model %>%
  predict(x$NumInputs) %>% data.table # predicts V1 and v2 
codes[,Year:= train[,unique(Year)]] # adding the year to the table codes
head(codes)

##           V1          V2 Year
## 1: -2.589451 0.193039715 1950
## 2: -2.583511 0.146560371 1951
## 3: -2.465765 0.106399596 1952
## 4: -2.353044 0.067193687 1953
## 5: -2.168850 0.038686931 1954
## 6: -2.233367 0.004854023 1955

ts_mat = as.ts(codes[,c(1:2),with=F]) # as time series

dim1_forecasts = ts_mat[,1] %>% forecast::rwf(h = 17,drift=T) %>% forecast(17)# forecasting
dim2_forecasts = ts_mat[,2] %>% forecast::rwf(h = 17,drift=T) %>% forecast(17)# forecasting

codes_forecast=data.table(V1 = as.double(dim1_forecasts$mean),V2 = as.double(dim2_forecasts$mean),Year=2000:2016) # adds colum and such
head(codes_forecast)

##           V1        V2 Year
## 1: 0.5986809 0.2240492 2000
## 2: 0.6624435 0.2246694 2001
## 3: 0.7262061 0.2252896 2002
## 4: 0.7899688 0.2259098 2003
## 5: 0.8537314 0.2265300 2004
## 6: 0.9174940 0.2271502 2005

codes = rbind(codes, codes_forecast)

0.18 Forecast

The first part of the chunk is the same principles as before.

I made a change in the plot, he plotted Pred_lc to year, but i think he wanted to plot autoencoder to year, so I plotted all of them.

This specific chunk is better to explain in the code rows.

CodeInputs <- layer_input(shape = c(2), dtype = 'float32', name = 'CodeInputs') #layer input.
DecodedRates = CodeInputs %>%  
  layer_dense(units = 4, activation = 'tanh', name = "DecodeRates1") %>% 
  layer_dense(units = 100, activation = 'sigmoid', name = "DecodedRates")
decode_model <- keras_model(
  inputs = c(CodeInputs), 
  outputs = c(DecodedRates)) %>% compile(
    optimizer = adam,
    loss = "mape")
set_weights(get_layer(decode_model, "DecodeRates1"), get_weights(get_layer(model, "DecodeRates1")))
set_weights(get_layer(decode_model, "DecodedRates"), get_weights(get_layer(model, "Rates")))



x_codes = list(encode = codes[,c(1,2),with=F] %>% as.matrix)
codes =  decode_model %>%
  predict(x_codes$encode) %>% data.table #predicts V1-V100
codes[,Year:=.I] # Adds years 
codes = codes %>% melt(id.vars="Year") %>% data.table() #data formating for V1 for each year and so on
codes[,Age:=as.integer(str_sub(variable, 2))-1] # adding ages
head(codes)

##    Year variable     value Age
## 1:    1       V1 0.6635555   0
## 2:    2       V1 0.6634684   0
## 3:    3       V1 0.6565518   0
## 4:    4       V1 0.6498291   0
## 5:    5       V1 0.6381762   0
## 6:    6       V1 0.6428274   0

codes[,Year:=Year+1949] # correcting the year
codes[,mx_autoencode:=value] # adding the previous MX_autencode values.
codes[,mx_nn:=exp(mx_autoencode*(scaled$max-scaled$min)+scaled$min)] # new collum add that looks like same formula as last time.
codes %>% setkey(Year,Age)
head(codes)

##    Year variable     value Age mx_autoencode       mx_nn
## 1: 1950       V1 0.6635555   0     0.6635555 0.043364524
## 2: 1950       V2 0.3656229   1     0.3656229 0.002913659
## 3: 1950       V3 0.3188880   2     0.3188880 0.001907589
## 4: 1950       V4 0.2954786   3     0.2954786 0.001542918
## 5: 1950       V5 0.2933152   4     0.2933152 0.001512960
## 6: 1950       V6 0.2771123   5     0.2771123 0.001306324

test$mx_autoencode = codes[Year>=2000]$mx_autoencode #we add the mx_autoencode values from the codes above to the test dataset from 2000 and up.
head(test)

##    Year Age   Female     Male    Total OpenInterval     logmx    mx_scale
## 1: 2000   0 0.005038 0.006073 0.005568        FALSE -5.103903 0.446658392
## 2: 2000   1 0.000304 0.000447 0.000377        FALSE -7.712952 0.158786557
## 3: 2000   2 0.000165 0.000278 0.000223        FALSE -8.187889 0.106383899
## 4: 2000   3 0.000171 0.000175 0.000173        FALSE -8.650725 0.055316563
## 5: 2000   4 0.000157 0.000137 0.000147        FALSE -8.895530 0.028305776
## 6: 2000   5 0.000108 0.000108 0.000108        FALSE -9.133379 0.002062413
##         pred_LC mx_autoencode
## 1: 0.0068457726    0.43899250
## 2: 0.0004773606    0.16486940
## 3: 0.0003048824    0.10951355
## 4: 0.0002381634    0.07240656
## 5: 0.0001971017    0.05901864
## 6: 0.0001697622    0.00000000

test[,mx_autoencode:=exp(mx_autoencode*(scaled$max-scaled$min)+scaled$min)] #replaces values of mx_autencoer with these values from the formula.
head(test)

##    Year Age   Female     Male    Total OpenInterval     logmx    mx_scale
## 1: 2000   0 0.005038 0.006073 0.005568        FALSE -5.103903 0.446658392
## 2: 2000   1 0.000304 0.000447 0.000377        FALSE -7.712952 0.158786557
## 3: 2000   2 0.000165 0.000278 0.000223        FALSE -8.187889 0.106383899
## 4: 2000   3 0.000171 0.000175 0.000173        FALSE -8.650725 0.055316563
## 5: 2000   4 0.000157 0.000137 0.000147        FALSE -8.895530 0.028305776
## 6: 2000   5 0.000108 0.000108 0.000108        FALSE -9.133379 0.002062413
##         pred_LC mx_autoencode
## 1: 0.0068457726  0.0056653856
## 2: 0.0004773606  0.0004723351
## 3: 0.0003048824  0.0002859983
## 4: 0.0002381634  0.0002043178
## 5: 0.0001971017  0.0001809714
## 6: 0.0001697622  0.0001060000

df <- data.frame(test[,sum((Male-mx_autoencode)^2)],test[,sum((Male - pred_LC)^2)]) #dataframe
colnames(df)<- c("MSE mx_autoencode","MSE Pred_LC") # colum names
df # SSE for the mx_autencoder en Pred_lc 

##   MSE mx_autoencode MSE Pred_LC
## 1         0.1170268   0.2623647

results= rbind(results, data.table(model = "Autoencoder",MSE_OutOfSample=test[,sum((Male-mx_autoencode)^2)]))
results # this code above adds the Autencode MSE to the results table.

##          model MSE_OutOfSample
## 1:  Lee-Carter       0.2623647
## 2: Autoencoder       0.1170268

test[Year==2016]%>% 
  ggplot(aes(x=Age, y = log(mx_autoencode),colour="Mx_Autoencoder Predicted"))+
  geom_point(size = 0.5, alpha=0.5)+
  facet_wrap(~Year)+
  geom_line(aes(x=Age, y=log(Male),colour="Actual"))+
  geom_line(aes(x=Age, y = log(pred_LC),colour="Pred_LC Predicted"))+
  ggtitle(label = "Values of mx for each age")+
  xlab("Age")+
  ylab("Log(mx)")+
  theme(panel.background = element_rect(fill = "#ADD8E6", colour = "#6D9EC1",size = 2, linetype = "solid")) #plot of mx

0.19 Regression

A new data set train_reg is made with the variables of the training data set, Year, Age, Male mx and mx_scale without replacement. The same is done for the test_reg.

After the scaling of the year that is weird for me, but from prior knowledge its better for the network to use scaled values, so i think that is why year is also scaled, the values of the Year_scale are the replaced in the original train_reg and test_reg datasets. Within chunk explanation is done again.

train_reg = train[,c(1,2,4,8),with=F] # New dataset
test_reg = test[,c(1,2,4,8),with=F] # New dataset

# Train

year_scale = scale_min_max(train_reg$Year,test_reg$Year) # scaled the new datasetes, but his scalling the year, weird, 

train_reg$Year = year_scale[[1]] #replace in dataset
test_reg$Year = year_scale[[2]] #replace in dataset

Year=  train_reg$Year # defines variable
Age =  train_reg$Age  # defines variable

x = list(Year= Year,Age = Age) #defines independant varialbes
y = (main_output= train_reg$mx_scale) #defines dependent varialbe as the mx_scaled one.

#test
Year = test_reg$Year # defines variable
Age = test_reg$Age  # defines variable

x_test = list(Year = Year,Age = Age) #defines independant varialbes
y_test = (main_output= test_reg$mx_scale) #defines dependent varialbe as the mx_scaled one.

0.20 Building layers

We define the layers for year and age, some stuff is really vague to me.

A dropout rate of 0.1 is used to prevent a model from over fitting. Dense is the units in the layers with an activation function relu (rectified linear unit).

Year <- layer_input(shape = c(1), dtype = 'float32', name = 'Year') #define layer 
Age <- layer_input(shape = c(1), dtype = 'int32', name = 'Age') #define layer
Age_embed = Age %>% #define embed age ? 
  layer_embedding(input_dim = 100, output_dim = 5,input_length = 1, name = 'Age_embed') %>%
  keras::layer_flatten()

main_output <- layer_concatenate(list(Year,Age_embed)) %>% #output variable layers drop out and dense. 
  layer_dense(units = 32, activation = 'relu') %>% 
  layer_dropout(0.1) %>%  
  layer_dense(units = 32, activation = 'relu') %>% 
  layer_dropout(0.1) %>% 
  layer_dense(units = 1, activation = 'sigmoid', name = 'main_output') 

model <- keras_model( #keras model
  inputs = c(Year,Age), 
  outputs = c(main_output))

adam = optimizer_adam(lr=0.001) # NOOOOOO idea what the following part do
lr_callback = callback_reduce_lr_on_plateau(factor=.80, patience = 50, verbose=1, cooldown = 100, min_lr = 0.0001)
model_callback = callback_model_checkpoint(filepath = "c:/r/best_mx_reg_callback.mod", verbose = 1,save_best_only = TRUE)

model %>% compile( #compiling the model
  optimizer = adam,
  loss = "mse")

fit = model %>% fit( # This is the final model.
  x = x,
  y = y, 
  epochs = 10, # I use this amount of epochs, it was 1500 but i did not want to wait so long.
  batch_size = 16,verbose = 1, shuffle = T, validation_split = 0.1, callbacks = list(lr_callback,model_callback))

0.21 Comparing all models

This best_mx_reg model predicts the explanatory variables (x_test) and adds it to the test data set. within chunk explanation is done again

model = load_model_hdf5("best_mx_reg.mod") #load a best mx model

test$mx_deep_reg = model %>%
  predict(x_test) 
test[,mx_deep_reg:=exp(mx_deep_reg*(scaled$max-scaled$min)+scaled$min)] # again modified to the formula
head(test)

##    Year Age   Female     Male    Total OpenInterval     logmx    mx_scale
## 1: 2000   0 0.005038 0.006073 0.005568        FALSE -5.103903 0.446658392
## 2: 2000   1 0.000304 0.000447 0.000377        FALSE -7.712952 0.158786557
## 3: 2000   2 0.000165 0.000278 0.000223        FALSE -8.187889 0.106383899
## 4: 2000   3 0.000171 0.000175 0.000173        FALSE -8.650725 0.055316563
## 5: 2000   4 0.000157 0.000137 0.000147        FALSE -8.895530 0.028305776
## 6: 2000   5 0.000108 0.000108 0.000108        FALSE -9.133379 0.002062413
##         pred_LC mx_autoencode  mx_deep_reg
## 1: 0.0068457726  0.0056653856 0.0044964767
## 2: 0.0004773606  0.0004723351 0.0004172431
## 3: 0.0003048824  0.0002859983 0.0002805146
## 4: 0.0002381634  0.0002043178 0.0002258803
## 5: 0.0001971017  0.0001809714 0.0002005641
## 6: 0.0001697622  0.0001060000 0.0001855847

test[,.(lc = sum((Male-pred_LC)^2),autoencode_forecast= sum((Male-mx_autoencode)^2), Deep_reg = sum((Male-mx_deep_reg)^2))]# This is the MSE for PRed_LC, Autoencodeing and the best reg model.

##           lc autoencode_forecast   Deep_reg
## 1: 0.2623647           0.1170268 0.08143053

results = rbind(results, data.table(model = "Deep_reg", MSE_OutOfSample =test[,sum((Male-mx_deep_reg)^2)]))
results # MSE out of sample for the 3 models.

##          model MSE_OutOfSample
## 1:  Lee-Carter      0.26236472
## 2: Autoencoder      0.11702681
## 3:    Deep_reg      0.08143053

gbrtenw = fread("gbrtenw.csv")


Y2016 = test[Year==2016] #makes a new dataset for only the year 2016
Y2016$mx_deep_reg_all = gbrtenw$mx_deep_reg_full #adding that collum, dont know what it is .

Y2016[,.(Pred_lc = sum((Male-pred_LC)^2),autoencode_forecast= sum((Male-mx_autoencode)^2),Deep_reg = sum((Male-mx_deep_reg)^2),Deep_reg_all = sum((Male-mx_deep_reg_all)^2))] # the testing results MSE

##       Pred_lc autoencode_forecast   Deep_reg Deep_reg_all
## 1: 0.01705496         0.007398312 0.01171775  0.006009346

Y2016[,c(2,4,9:12),with=F] %>% melt(id.vars="Age") %>% 
  ggplot(aes(x=Age, y=log(value)))+geom_line(aes(group = variable,colour=variable,linetype=variable))+
  geom_point(aes(colour=variable,shape=variable))+
  ggtitle(label = "Values of mx for each age")+
  ylab("Log(mx)")+
  theme(panel.background = element_rect(fill = "#ADD8E6", colour = "#6D9EC1",size = 2, linetype = "solid"))