From Pavlov Dogs to Deep Learning :
Evolution of AI and AI scientist
Dr. Dmitry O. Gorodnichy, 20 June 2018
www.IVIM.ca/dg/ai (https://bookdown.org/gorodnichy/ai1)
Outline
- What is AI and NN ?
- Background
- Video Recognition Systems (VRS)
- Fully-connected pseudo-inverse neural networks (PINN)
- How NN (AI) was done in the past
- Success based on intelligent preprocessing
- Demo: Face Recognition in Video (FRiV)
- How AI (NN) is done now
- Old FRiV ideas implemented via new Keras functionality
- Good and bad use of NN: Sequencial data example
- Vision for AI and AI scientist
- Intelligent multilingual text analysis example
Defining AI
What’s Common ?
| Cybernetics Center in Kiev |
Institut für Neuroinformatik in Bochum |
Brain Team in Google |
Defining AI
What’s Common ?
| Cybernetics Center in Kiev |
Institut für Neuroinformatik in Bochum |
Brain Team in Google |
It’s all about the same !
- Artificial Intelligence (AI)
- Neural networks (NN)
- Highly complex, very diverse, densly-connected, recurrent, multi-layered neural networks,
- Plus, leveraging many other Machine Learning and Stats techniques
My former colleagues: Ivakhnenko: library(GMDH), Kussul, Reznik
What I really want to know
I want to be able to predict future
| Observation |
Hypothesis |
Validation |
Application |
| … |
… |
… |
? |
I want to be able to predict future
| Scientific calculators made for everyone |
Larger population of programmers |
Deep Learning is everyone’s “scientific calculator” |
AI everywhere |
| … |
… |
… |
… |
| No code sharing |
Most AI coding done in in-house - C++ language of choice |
AI codes reusable, modular, scalable, shareable - R / RStudio IDE of choice |
Full automation - AI helping built AI |
“But something in our minds will always stay”
Our Objective:
To built highly intelligent systems
- capable of recognizing, predicting, advising, optimizing,
- bridging the gap between people and technology,
- bridging the gap between people and people
- through the use neural networks,
- leverged by many other AI / Stats tools
- programmed in AI-developer-driven IDE such as RStudio
- interfaced with Python, C++ (if needed)
Many religions, one God.
Many notations, one AI.
“One Vision”
This is about all Recognition systems
… which we built in the past, and
… which we can built now and in future
How did we do it then?
Dmitry O. Gorodnichy, Associative neural networks as means for low-resolution video-based recognition, International Joint Conference on Neural Networks (IJCNN’05), NRC 48217.
“Best Presentation” award at IJCNN 1999
D.O. Gorodnichy, A.M. Reznik, (1997) “Increasing Attraction of Pseudo-Inverse Autoassociative Networks” , Neural Processing Letters, volume 5, issue 2, pp. 123-127,
- Compare to: Hochreiter, S.; Schmidhuber, J. (1997). “Long Short-Term Memory”. Neural Computation. 9 (8): 1735–1780.
- Ph.D. Dissertation: “Investigation and Design of High Performance Fully-Connected Neural Networks”, Glushkov Cybernetics Center, Ukraine, Kiev, 1997.
From Past to Present to Future
Different terminology - same idea
- Neural networks, Neurocomputation, Connectionist models -> Deep Learning
- Image Processing / Computer Visions feature/edge detections –> Convolution networks
- Attractor-based Networks Neural –> Recurrent Neural Networks,
- Desaturation –> Regularization technique against over-fitting
- Flood-fill processing –> “Òne-hot” coding in Tensorflow
- All custom-made codes in MS C++ –> in R/RStudio with
library(keras)
Now with much more efficient programming workflow and support !
Intro Tensorflow with Keras in RStudio
- “Simply the Best!”
library(keras): it installs library(tensorflow)
- Leverages R/RStudio power and beauty:
- pipeline
%>% worksflow
- most suitable for Tensor data flows,
- RStudio IDE and web-servers, which are designed by AI scientists themselves
- Plenty of excellent free tutorials:
Example 1: FRiV redone with Keras
- MNIST digit recognition is used the “Hello world” example:
# Define model
model <- keras_model_sequential() %>%
layer_conv_2d(filters = 32, kernel_size = c(3,3), activation = 'relu',
input_shape = input_shape) %>%
layer_conv_2d(filters = 64, kernel_size = c(3,3), activation = 'relu') %>%
layer_max_pooling_2d(pool_size = c(2, 2)) %>%
layer_dropout(rate = 0.25) %>%
layer_flatten() %>%
layer_dense(units = 128, activation = 'relu') %>%
layer_dropout(rate = 0.5) %>%
layer_dense(units = 10, activation = 'softmax')
# Compile model
model %>% compile(
loss = loss_categorical_crossentropy,
optimizer = optimizer_adadelta(),
metrics = c('accuracy')
# Train model
model %>% fit(
x_train, y_train, validation_split = 0.2,
batch_size = batch_size, epochs = epochs
)
- Extending to processing sequences of images:
...
Example 2: Predicting future (sequential data)
Data from Open Government -> Historical Border Wait Times:
library(data.table);library(ggplot2);library(lubridate);
library(magrittr); library(stringr);
dt <- fread("http://cbsa-asfc.gc.ca/data/bwt-taf-2016-07-01--2016-09-30-en.csv")
Cleaning and selecting data:
dt <- dt[str_detect(Location, "BC")] # subset: BWT in British Columbia
dt[, Updated := as.POSIXct(Updated, format = "%Y-%m-%d %H:%M ") ]
dt$BWT <- str_replace(dt$`Travellers Flow`, "No delay", "0") %>%
str_replace("Closed", "0") %>% str_replace("Not applicable", "0") %>%
str_replace("Missed entry", "NA") %>% as.numeric()
dt <- dt[, .(Updated, BWT, Location)]
Example 2: Visualized
theme_set(theme_minimal())
ggplot(dt[Updated<dmy("01/09/2016")], aes(Updated,BWT,col=Location)) +
geom_step() + facet_grid(Location ~ .)
Knowing Past data, can we use “Deep Learning” to predict Future data ?
Your answer ?
model2 <- keras_model_sequential() %>%
layer_gru(units = 32, dropout = 0.2, recurrent_dropout = 0.2,
input_shape = list(NULL, dim(dt)[[-1]])) %>%
layer_dense(units = 1)
model2 %>% compile(
optimizer = optimizer_rmsprop(), loss = "mae"
)
model2 %>% fit_generator(
train_gen, steps_per_epoch = 500, epochs = 40,
validation_data = val_gen, validation_steps = val_steps
)
Conclusions
Looking into the Future
What’s common ?
It will be possible
… to automatially deduct that
Two booksin different languages (e.g., “Book of Joy” by Dalai Lama and Desmond Tutu, written in 2015, and “Deux Vies” by Concordia Antarova, written in Russian in 1955) are about the same…
… through intelligent NN-based text analysis (eg. RNN, LSTM, PINN).
