Everything you need to know about recurrent neural networks

The human mind has different mechanisms for processing individual pieces of information and sequences. For instance, we have a definition of the word “like.” But we also know that how “like” is used in a sentence depends on the words that come before and after it. Consider how you would fill in the blanks in the following two sentences:

Would you like … coffee?

Would you like … a walk?

We see sequences everywhere. Videos are sequences of images, audio files are sequences of sound samples, music is sequences of notes. In all cases, there is a temporal dependency between the individual members of the sequence. Changing the order of frames in a video will render it meaningless. Changing the order of words in a sentence or article can completely change its meaning.

As with the human brain, artificial intelligence algorithms have different mechanisms for the processing of individual and sequential data. The first generation of artificial neural networks, the AI algorithms that have gained popularity in the past years, were created to deal with individual pieces of data such as single images or fixed-length records of information. But they were not suitable for variable-length, sequential data.

[Read: The key differences between rule-based AI and machine learning]

Recurrent neural networks (RNN), first proposed in the 1980s, made adjustments to the original structure of neural networks to enable them to process streams of data.

Feedforward vs recurrent neural networks

Multi-layer perceptrons (MLP) and convolutional neural networks (CNN), two popular types of ANNs, are known as feedforward networks. In feedforward networks, information moves in one direction. They receive input on one end, process the data in their hidden layers, and produce an output value. For instance, an image goes through one end, and the possible class of the image’s contents come out the other end.