# data[].attributes.body
They are a family of neural networks for processing sequential data, thus order and structure matters.

To go from multilayer networks to recurrent networks, ideas found in machine learning and statistical models of the 1980s: sharing parameters across diﬀerent parts of a model

Why the need of parameter sharing? It allows us to extend the model and check how well generalises, since we are able to apply the model to different forms (different lengths in this case).

Can be a 1D convolution like text recognition. In practice RNN operates in minibatches t, and the time step of each minibatch doesn't have to be related to the real world.

Can be applied to 2D spatial data like images, when the time serial must be considered like in videos.

# data[].attributes.body
We could say that a RNN is composed by n computational graphs which are a set of structured computations. That computation implies mapping an input and parameters to outputs and loss.

We can see a computational graph in its folded or unfolded view: the unfolded view provides an explicit description of which computations to perform, and helps us to illustrate the information flow forward in time (computing outputs and losses) and backwards in time (computen gradients) since we know explicitly which path the information is following.

Regarding this graph:

This RNN has no outputs,

This RNN processes info from x (input node) and incorporates it to the state h, that is passed forward time.

The black square on the folded view is the delay of a single time step.

Unfolded computational graph --> each node is associated to a 1 time instance

For a x(t) signal, the state h(t) is represented by a function dependent of the previous state h(t-1) and gradients.

# data[].attributes.body
In many applications, however, we want to output a prediction of y(t) that

may depend on the whole input sequence x. Ex. In speech recognition if a sound might be interpreted in two ways, maybe we need information about the context in order to figure out what is the likely sound.
It combines an RNN that moves forward through time, beginning from the start of the sequence, with another RNN that moves backward through time, beginning from the end of the sequence.

As the name suggests, bidirectional RNNs combine an RNN that moves forward

through time, beginning from the start of the sequence, with another RNN that

moves backward through time, beginning from the end of the sequence. That means to learn to map the input sequences x to target sequences y with loss L(t) at each step t. The h recurrence propagates information forward in time (to the right) while g recurrence propagates information backward in time (to the left). At each point t the o(t) can benefit from the past on its h(t) and from the future through g(t).

# data[].attributes.body
In theory, RNNs are absolutely capable of handling such long-term dependencies, however in practice, RNNs don’t seem to be able to learn them.

The basic idea behind an LSTM is to control the how the information flow is formed, that means to control which information is going to be keep for long, which part of the recent information flow (or short-term memory) is going to be part of the history, etc. LSTMs have a repeating unit structure, same as RNN,  but the repeating module instead of a single neural network layer, there are four that interact btw each other, this four layers are seen like gates through which the information flow is moved forming a circuit. This gates are composed by a sigmoid nn and an element-wise multiplication operation.

LSTMs have a repeating unit structure usually called cells, same as RNN,  but the repeating module instead of a single neural network layer, there are four that interact btw each other.

# data[].attributes.body
Left: One input produces an output each time step, existing connections btw hidden units. The graph maps:

x, input sequence --> o, outputs

L, loss measures how far each o is from training target y

when using softmax outputs, we assume o is the unnormalized log      probabilities.

The RNN has input to hidden connections parametrised by a weight matrix U. Hidden-to-hidden recurrent connections parametrised by a weight matrix W. Hidden-to-output connections parametrised by a weight matrix V.

It's a powerful model: any possible information about the past, can be shared to the hidden representation and transmitted to the future.

Center: One input produces an output each time step, existing connections only from output to hidden units each time step. The only recurrence is the feedback connection from the output to the hidden layer. This model is less powerful than the previous one: the figure is trained to put a specific output value into the output o, and o is the only information that is allowed to send to the future. No direct connections from h are going forward. Previous h is connected to the present only indirectly via the predictions it was used to produce. Unless o is high-dimensional and rich, it will usually lack important information from the past. Thus, less powerful but easier to train, because each time step is ISOLATED from the others (that allows greater parallelization). This models are trained with teacher forcing.

Right: Recurrent connections btw hidden units, that read an entire sequence and produces a single output. It can be used to summarise a sequence producing a fixed-size representation (used as input for further processing like seq-2-seq model). There might be a target (right at the end) or the gradient on the output o(t) can be obtained by BPTT from further downstream modules.

# data[].attributes.headline
PROBLEM: Vanishing (and exploding) gradients

# data[].attributes.subject
PROBLEM: Vanishing (and exploding) gradients

# data[].attributes.body
The challenge of learning long-term dependencies is that gradients propagated over many stages tend to either to vanish (most of the time) or explode. If we assume that the parameters are stable, we will be comparing long-term dependencies (smaller weights resulting from the multiplication of many Jacobians -matrix of derivatives-) to the short-term ones (Hochreiter, 1991; Doya, 1993; Bengio et al., 1994; Pascanu et al., 2013).  If we can’t know the gradient, we can’t adjust the weights in a direction that will decrease error, and our network ceases to learn.

Exploding gradients treat every weight as though it were the proverbial butterfly whose flapping wings cause a distant hurricane. Those weights’ gradients become saturated on the high end; i.e. they are presumed to be too powerful. But exploding gradients can be solved relatively easily, because they can be truncated or squashed. Vanishing gradients can become too small for computers to work with or for networks to learn – a harder problem to solve.

Below you see the effects of applying a sigmoid function over and over again. The data is flattened until, for large stretches, it has no detectable slope. This is analogous to a gradient vanishing as it passes through many layers.

https://uwaterloo.ca/data-analytics/sites/ca.data-analytics/files/uploads/files/rnn1.pdf

# data[].attributes.body
Learn gate --> combines short-term memory and event ignoring part the information by wise multiplying by a factor i(t). The combination is performed by putting them through a linear function which consists of joining the vectors by multiplying by a weight matrix adding a bias and finally squashing the result with a tanh activation function. This combination leads to a new pseudo state that can be called as ~C(t), which later use will lead to the new updated cell state from C(t-1) to C(t).

To calculate i(t) we take the previous short-term info and event as an input to a small nn, and pass them to a linear function with new weights and bias and squashing them with a sigmoid function. The output is i(t) which is element-wise multi plicated to ~C(t).

Forget gate --> through which its possible rid of irrelevant  long-term memory information flow (history), that might be irrelevant for the outcome. After the cleanse, the remaining information will be added to the history. The info dropping is performed by element wise multiplication of the history -the cell state, C(t-1)- by a forget factor f(t).

To calculate f(t) it is used short-term memory and the event put them through a linear function multiplying by a new matrix and adding a bias, squashing with a sigmoid. This way new information is incorporated to the history.

Remember gate -->  The output of this gate is new updated long-term memory, thus what we will remember for the future. It adds wisely the remaining info output by the forget gate and the new learnt info from the learn gate. After this gate we have a brand new C(t) state, a new updated info workflow.

Use gate --> this gate is responsible to decide which information to use from what we previously know + what we just learned to make a prediction. The prediction is based on a filtered version of the long-term info updated flow C(t). This filter o(t) is an element wise multiplication factor, which is built using the short-term info and the event info into a small nn,  passing through a linear function with new weight matrix and bias, and squash the result with the sigmoid function.

Despite this is a frequent architecture which offers very good results, there are so many others possible.

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
https://deeplearning4j.org/lstm.html

# data[].attributes.body
LSTM variant, introduced by Gers & Schmidhuber (2000), is adding “peephole connections.” This variation also considers the cell state as an input for the forget, learn and use gates, which means that history and recent information are combined into the different nn that are run at each gate.
http://colah.github.io/posts/2015-08-Understanding-LSTMs/#variants-on-long-short-term-memory

# data[].attributes.headline
Backpropagation or BPTT (backpropagation through  time)

# data[].attributes.subject
Backpropagation or BPTT (backpropagation through  time)

# data[].attributes.body
No specialised algorithms are necessary, the generalised backpropagation algorithm is applied to the unrolled computational graph. Gradients obtained by backpropagation then are used to train.

In feedforward networks moves backward from the final error through the outputs, weights and inputs of each hidden layer, assigning those weights responsibility for a portion of the error by calculating their partial derivatives – ∂E/∂w, or the relationship between their rates of change. Those derivatives are then used by our learning rule, gradient descent, to adjust the weights up or down, whichever direction decreases error. Recurrent networks rely on an extension of backpropagation called backpropagation through time, or BPTT. Time, in this case, is simply expressed by a well-defined, ordered series of calculations linking one time step to the next, which is all backpropagation needs to work.

Neural networks, whether they are recurrent or not, are simply nested composite functions like f(g(h(x))). Adding a time element only extends the series of functions for which we calculate derivatives with the chain rule.

# data[].attributes.attachment_link.readable_text
Contents

The purpose of this post is to give students of neural networks an intuition about the functioning of recurrent neural networks and purpose and structure of a prominent RNN variation, LSTMs.

Recurrent nets are a type of artificial neural network designed to recognize patterns in sequences of data, such as text, genomes, handwriting, the spoken word, or numerical times series data emanating from sensors, stock markets and government agencies.

They are arguably the most powerful and useful type of neural network, applicable even to images, which can be decomposed into a series of patches and treated as a sequence.

Since recurrent networks possess a certain type of memory, and memory is also part of the human condition, we'll make repeated analogies to memory in the brain.

GET STARTED WITH DEEP LEARNING

To understand recurrent nets, first you have to understand the basics of feedforward nets. Both of these networks are named after the way they channel information through a series of mathematical operations performed at the nodes of the network. One feeds information straight straight through (never touching a given node twice), while the other cycles it through a loop, and the latter are called recurrent.

In the case of feedforward networks, input examples are fed to the network and transformed into an output; with supervised learning, the output would be a label, a name applied to the input. That is, they map raw data to categories, recognizing patterns that signal, for example, that an input image should be labeled "cat" or "elephant."

A feedforward network is trained on labeled images until it minimizes the error it makes when guessing their categories. With the trained set of parameters (or weights, collectively known as a model), the network sallies forth to categorize data it has never seen. A trained feedforward network can be exposed to any random collection of photographs, and the first photograph it is exposed to will not necessarily alter how it classifies the second. Seeing photograph of a cat will not lead the net to perceive an elephant next.

That is, a feedforward network has no notion of order in time, and the only input it considers is the current example it has been exposed to. Feedforward networks are amnesiacs regarding their recent past; they remember nostalgically only the formative moments of training.

Recurrent networks, on the other hand, take as their input not just the current input example they see, but also what they have perceived previously in time. Here's a diagram of an early, simple recurrent net proposed by Elman, where the BTSXPE at the bottom of the drawing represents the input example in the current moment, and CONTEXT UNIT represents the output of the previous moment.

The decision a recurrent net reached at time step t-1 affects the decision it will reach one moment later at time step t. So recurrent networks have two sources of input, the present and the recent past, which combine to determine how they respond to new data, much as we do in life.

Recurrent networks are distinguished from feedforward networks by that feedback loop connected to their past decisions, ingesting their own outputs moment after moment as input. It is often said that recurrent networks have memory. Adding memory to neural networks has a purpose: There is information in the sequence itself, and recurrent nets use it to perform tasks that feedforward networks can't.

That sequential information is preserved in the recurrent network's hidden state, which manages to span many time steps as it cascades forward to affect the processing of each new example. It is finding correlations between events separated by many moments, and these correlations are called "long-term dependencies", because an event downstream in time depends upon, and is a function of, one or more events that came before. One way to think about RNNs is this: they are a way to share weights over time.

Just as human memory circulates invisibly within a body, affecting our behavior without revealing its full shape, information circulates in the hidden states of recurrent nets. The English language is full of words that describe the feedback loops of memory. When we say a person is haunted by their deeds, for example, we are simply talking about the consequences that past outputs wreak on present time. The French call this " Le passé qui ne passe pas," or "The past that does not pass away."

We'll describe the process of carrying memory forward mathematically:

The hidden state at time step t is h_t. It is a function of the input at the same time step x_t, modified by a weight matrix W (like the one we used for feedforward nets) added to the hidden state of the previous time step h_t-1 multiplied by its own hidden-state-to-hidden-state matrix U, otherwise known as a transition matrix and similar to a Markov chain. The weight matrices are filters that determine how much importance to accord to both the present input and the past hidden state. The error they generate will return via backpropagation and be used to adjust their weights until error can't go any lower.

The sum of the weight input and hidden state is squashed by the function φ - either a logistic sigmoid function or tanh, depending - which is a standard tool for condensing very large or very small values into a logistic space, as well as making gradients workable for backpropagation.

Because this feedback loop occurs at every time step in the series, each hidden state contains traces not only of the previous hidden state, but also of all those that preceded h_t-1 for as long as memory can persist.

Given a series of letters, a recurrent will use the first character to help determine its perception of the second character, such that an initial q might lead it to infer that the next letter will be u, while an initial t might lead it to infer that the next letter will be h.

Since recurrent nets span time, they are probably best illustrated with animation (the first vertical line of nodes to appear can be thought of as a feedforward network, which becomes recurrent as it unfurls over time).

how recurrent neural networks work #deeplearning4j #dl4j

In the diagram above, each x is an input example, w is the weights that filter inputs, a is the activation of the hidden layer (a combination of weighted input and the previous hidden state), and b is the output of the hidden layer after it has been transformed, or squashed, using a rectified linear or sigmoid unit.

Remember, the purpose of recurrent nets is to accurately classify sequential input. We rely on the backpropagation of error and gradient descent to do so.

Backpropagation in feedforward networks moves backward from the final error through the outputs, weights and inputs of each hidden layer, assigning those weights responsibility for a portion of the error by calculating their partial derivatives - ∂E/∂w, or the relationship between their rates of change. Those derivatives are then used by our learning rule, gradient descent, to adjust the weights up or down, whichever direction decreases error.

Recurrent networks rely on an extension of backpropagation called backpropagation through time, or BPTT. Time, in this case, is simply expressed by a well-defined, ordered series of calculations linking one time step to the next, which is all backpropagation needs to work.

Neural networks, whether they are recurrent or not, are simply nested composite functions like f(g(h(x))). Adding a time element only extends the series of functions for which we calculate derivatives with the chain rule.

Truncated BPTT

Truncated BPTT is an approximation of full BPTT that is preferred for long sequences, since full BPTT's forward/backward cost per parameter update becomes very high over many time steps. The downside is that the gradient can only flow back so far due to that truncation, so the network can't learn dependencies that are as long as in full BPTT.

Like most neural networks, recurrent nets are old. By the early 1990s, the vanishing gradient problem emerged as a major obstacle to recurrent net performance.

Just as a straight line expresses a change in x alongside a change in y, the gradient expresses the change in all weights with regard to the change in error. If we can't know the gradient, we can't adjust the weights in a direction that will decrease error, and our network ceases to learn.

Recurrent nets seeking to establish connections between a final output and events many time steps before were hobbled, because it is very difficult to know how much importance to accord to remote inputs. (Like great-great-*-grandparents, they multiply quickly in number and their legacy is often obscure.)

This is partially because the information flowing through neural nets passes through many stages of multiplication.

Everyone who has studied compound interest knows that any quantity multiplied frequently by an amount slightly greater than one can become immeasurably large (indeed, that simple mathematical truth underpins network effects and inevitable social inequalities). But its inverse, multiplying by a quantity less than one, is also true. Gamblers go bankrupt fast when they win just 97 cents on every dollar they put in the slots.

Because the layers and time steps of deep neural networks relate to each other through multiplication, derivatives are susceptible to vanishing or exploding.

Exploding gradients treat every weight as though it were the proverbial butterfly whose flapping wings cause a distant hurricane. Those weights' gradients become saturated on the high end; i.e. they are presumed to be too powerful. But exploding gradients can be solved relatively easily, because they can be truncated or squashed. Vanishing gradients can become too small for computers to work with or for networks to learn - a harder problem to solve.

Below you see the effects of applying a sigmoid function over and over again. The data is flattened until, for large stretches, it has no detectable slope. This is analogous to a gradient vanishing as it passes through many layers.

In the mid-90s, a variation of recurrent net with so-called Long Short-Term Memory units, or LSTMs, was proposed by the German researchers Sepp Hochreiter and Juergen Schmidhuber as a solution to the vanishing gradient problem.

LSTMs help preserve the error that can be backpropagated through time and layers. By maintaining a more constant error, they allow recurrent nets to continue to learn over many time steps (over 1000), thereby opening a channel to link causes and effects remotely. This is one of the central challenges to machine learning and AI, since algorithms are frequently confronted by environments where reward signals are sparse and delayed, such as life itself. (Religious thinkers have tackled this same problem with ideas of karma or divine reward, theorizing invisible and distant consequences to our actions.)

LSTMs contain information outside the normal flow of the recurrent network in a gated cell. Information can be stored in, written to, or read from a cell, much like data in a computer's memory. The cell makes decisions about what to store, and when to allow reads, writes and erasures, via gates that open and close. Unlike the digital storage on computers, however, these gates are analog, implemented with element-wise multiplication by sigmoids, which are all in the range of 0-1. Analog has the advantage over digital of being differentiable, and therefore suitable for backpropagation.

Those gates act on the signals they receive, and similar to the neural network's nodes, they block or pass on information based on its strength and import, which they filter with their own sets of weights. Those weights, like the weights that modulate input and hidden states, are adjusted via the recurrent networks learning process. That is, the cells learn when to allow data to enter, leave or be deleted through the iterative process of making guesses, backpropagating error, and adjusting weights via gradient descent.

The diagram below illustrates how data flows through a memory cell and is controlled by its gates.

There are a lot of moving parts here, so if you are new to LSTMs, don't rush this diagram - contemplate it. After a few minutes, it will begin to reveal its secrets.

Starting from the bottom, the triple arrows show where information flows into the cell at multiple points. That combination of present input and past cell state is fed not only to the cell itself, but also to each of its three gates, which will decide how the input will be handled.

The black dots are the gates themselves, which determine respectively whether to let new input in, erase the present cell state, and/or let that state impact the network's output at the present time step. S_c is the current state of the memory cell, and g_y_in is the current input to it. Remember that each gate can be open or shut, and they will recombine their open and shut states at each step. The cell can forget its state, or not; be written to, or not; and be read from, or not, at each time step, and those flows are represented here.

The large bold letters give us the result of each operation.

Here's another diagram for good measure, comparing a simple recurrent network (left) to an LSTM cell (right). The blue lines can be ignored; the legend is helpful.

It's important to note that LSTMs' memory cells give different roles to addition and multiplication in the transformation of input. The central plus sign in both diagrams is essentially the secret of LSTMs. Stupidly simple as it may seem, this basic change helps them preserve a constant error when it must be backpropagated at depth. Instead of determining the subsequent cell state by multiplying its current state with new input, they add the two, and that quite literally makes the difference. (The forget gate still relies on multiplication, of course.)

Different sets of weights filter the input for input, output and forgetting. The forget gate is represented as a linear identity function, because if the gate is open, the current state of the memory cell is simply multiplied by one, to propagate forward one more time step.

Furthermore, while we're on the topic of simple hacks, including a bias of 1 to the forget gate of every LSTM cell is also shown to improve performance. ( Sutskever, on the other hand, recommends a bias of 5.)

You may wonder why LSTMs have a forget gate when their purpose is to link distant occurrences to a final output. Well, sometimes it's good to forget. If you're analyzing a text corpus and come to the end of a document, for example, you may have no reason to believe that the next document has any relationship to it whatsoever, and therefore the memory cell should be set to zero before the net ingests the first element of the next document.

In the diagram below, you can see the gates at work, with straight lines representing closed gates, and blank circles representing open ones. The lines and circles running horizontal down the hidden layer are the forget gates.

It should be noted that while feedforward networks map one input to one output, recurrent nets can map one to many, as above (one image to many words in a caption), many to many (translation), or many to one (classifying a voice).

You may also wonder what the precise value is of input gates that protect a memory cell from new data coming in, and output gates that prevent it from affecting certain outputs of the RNN. You can think of LSTMs as allowing a neural network to operate on different scales of time at once.

Let's take a human life, and imagine that we are receiving various streams of data about that life in a time series. Geolocation at each time step is pretty important for the next time step, so that scale of time is always open to the latest information.

Perhaps this human is a diligent citizen who votes every couple years. On democratic time, we would want to pay special attention to what they do around elections, before they return to making a living, and away from larger issues. We would not want to let the constant noise of geolocation affect our political analysis.

If this human is also a diligent daughter, then maybe we can construct a familial time that learns patterns in phone calls which take place regularly every Sunday and spike annually around the holidays. Little to do with political cycles or geolocation.

Other data is like that. Music is polyrhythmic. Text contains recurrent themes at varying intervals. Stock markets and economies experience jitters within longer waves. They operate simultaneously on different time scales that LSTMs can capture.

Gated Recurrent Units (GRUs)

A gated recurrent unit (GRU) is basically an LSTM without an output gate, which therefore fully writes the contents from its memory cell to the larger net at each time step.

A commented example of a Graves LSTM learning how to replicate Shakespearian drama, and implemented with Deeplearning4j, can be found here. The API is commented where it's not self-explanatory. If you have questions, please join us on Gitter.

Here's what the LSTM configuration looks like:

GET STARTED WITH DEEP LEARNING

Here are a few ideas to keep in mind when manually optimizing hyperparameters for RNNs:

Watch out for overfitting, which happens when a neural network essentially "memorizes" the training data. Overfitting means you get great performance on training data, but the network's model is useless for out-of-sample prediction.

Regularization helps: regularization methods include l1, l2, and dropout among others.

So have a separate test set on which the network doesn't train.

The larger the network, the more powerful, but it's also easier to overfit. Don't want to try to learn a million parameters from 10,000 examples - parameters > examples = trouble.

More data is almost always better, because it helps fight overfitting.

Train over multiple epochs (complete passes through the dataset).

Evaluate test set performance at each epoch to know when to stop (early stopping).

The learning rate is the single most important hyperparameter. Tune this using deeplearning4j-ui; see this graph

In general, stacking layers can help.

For LSTMs, use the softsign (not softmax) activation function over tanh (it's faster and less prone to saturation (~0 gradients)).

Updaters: RMSProp, AdaGrad or momentum (Nesterovs) are usually good choices. AdaGrad also decays the learning rate, which can help sometimes.

Finally, remember data normalization, MSE loss function + identity activation function for regression, Xavier weight initialization

1) While recurrent networks may seem like a far cry from general artificial intelligence, it's our belief that intelligence, in fact, is probably dumber than we thought. That is, with a simple feedback loop to serve as memory, we have one of the basic ingredients of consciousness - a necessary but insufficient component. Others, not discussed above, might include additional variables that represent the network and its state, and a framework for decisionmaking logic based on interpretations of data. The latter, ideally, would be part of a larger problem-solving loop that rewards success and punishes failure, much like reinforcement learning. Come to think of it, DeepMind already built that...

2) All neural networks whose parameters have been optimized have memory in a sense, because those parameters are the traces of past data. But in feedforward networks, that memory may be frozen in time. That is, after a network is trained, the model it learns may be applied to more data without further adapting itself. In addition, it is monolithic in the sense that the same memory (or set of weights) is applied to all incoming data. Recurrent networks, which also go by the name of dynamic (translation: "changing") neural networks, are distinguished from feedforward nets not so much by having memory as by giving particular weight to events that occur in a series. While those events do not need to follow each other immediately, they are presumed to be linked, however remotely, by the same temporal thread. Feedforward nets do not make such a presumption. They treat the world as a bucket of objects without order or time. It may be helpful to map two types of neural network to two types of human knowledge. When we are children, we learn to recognize colors, and we go through the rest of our lives recognizing colors wherever we see them, in highly varied contexts and independent of time. We only had to learn the colors once. That knowledge is like memory in feedforward nets; they rely on a past without scope, undefined. Ask them what colors they were fed five minutes ago and they don't know or care. They are short-term amnesiacs. On the other hand, we also learn as children to decipher the flow of sound called language, and the meanings we extract from sounds such as "toe" or "roe" or "z" are always highly dependent on the sounds preceding (and following) them. Each step of the sequence builds on what went before, and meaning emerges from their order. Indeed, whole sentences conspire to convey the meaning of each syllable within them, their redundant signals acting as a protection against ambient noise. That is similar to the memory of recurrent nets, which look to a particular slice of the past for help. Both types of nets bring the past, or different pasts, to bear in different ways.

# data[].attributes.attachment_link.authors[].name
Chris V. Nicholson, Adam Gibson, Skymind team

# data[].attributes.attachment_link.title
A Beginner's Guide to Recurrent Networks and LSTMs

# data[].attributes.attachment_link.description
Contents The purpose of this post is to give students of neural networks an intuition about the functioning of recurrent neural networks and purpose and structure of a prominent RNN variation, LSTMs. Recurrent nets are a type of artificial neural network designed to recognize patterns in sequences of data, such as text, genomes, handwriting, the spoken word, or numerical times series data emanating from sensors, stock markets and government agencies.