Deep learning for natural language processing: advantages and challenges

Abstract

Deep learning refers to machine learning technologies for learning and utilizing ‘deep’ artificial neural networks, such as deep neural networks (DNN), convolutional neural networks (CNN) and recurrent neural networks (RNN). Recently, deep learning has been successfully applied to natural language processing and significant progress has been made. This paper summarizes the recent advancement of deep learning for natural language processing and discusses its advantages and challenges. We think that there are five major tasks in natural language processing, including classification, matching, translation, structured prediction and the sequential decision process. For the first four tasks, it is found that the deep learning approach has outperformed or significantly outperformed the traditional approaches. End-to-end training and representation learning are the key features of deep learning that make it a powerful tool for natural language processing. Deep learning is not almighty, however. It might not be sufficient for inference and decision making, which are essential for complex problems like multi-turn dialogue. Furthermore, how to combine symbolic processing and neural processing, how to deal with the long tail phenomenon, etc. are also challenges of deep learning for natural language processing. In our view, there are five major tasks in natural language processing, namely classification, matching, translation, structured prediction and the sequential decision process. Most of the problems in natural language processing can be formalized as these five tasks, as summarized in Table 1. In the tasks, words, phrases, sentences, paragraphs and even documents are usually viewed as a sequence of tokens (strings) and treated similarly, although they have different complexities. In fact, sentences are the most widely used processing units. It has been observed recently that deep learning can enhance the performances in the first four tasks and becomes the state-of-the-art technology for the tasks (e.g. [1–8]). Table 2 shows the performances of example problems in which deep learning has surpassed traditional approaches. Among all the NLP problems, progress in machine translation is particularly remarkable. Neural machine translation, i.e. machine translation using deep learning, has significantly outperformed traditional statistical machine translation. The state-of-the art neural translation systems employ sequence-to-sequence learning models comprising RNNs [4–6]. Deep learning has also, for the first time, made certain applications possible. For example, deep learning has been successfully applied to image retrieval (also known as text to image), in which query and image are first transformed into vector representations with CNNs, the representations are matched with DNN and the relevance of the image to the query is calculated [3]. Deep learning is also employed in generation-based natural language dialogue, in which, given an utterance, the system automatically generates a response and the model is trained in sequence-to-sequence learning [7]. Five tasks in natural language processing. Five tasks in natural language processing. Performances of Natural Language Processing Problems. Performances of Natural Language Processing Problems. Advantages and challenges of deep learning for natural language processing. good at pattern recognition problems data-driven, and performance is high in many problems end-to-end training: little or no domain knowledge is needed in system construction learn of representations: cross-modal processing is possible gradient-based learning: learning algorithm is simple mainly supervised learning methods not good at inference and decision making cannot directly handle symbols data-hungry and thus is not suitable when data size is small difficult to handle long tail phenomena model is usually a black box and is difficult to understand computational cost of learning is high unsupervised learning methods need to be developed still lacks of theoretical foundation good at pattern recognition problems data-driven, and performance is high in many problems end-to-end training: little or no domain knowledge is needed in system construction learn of representations: cross-modal processing is possible gradient-based learning: learning algorithm is simple mainly supervised learning methods not good at inference and decision making cannot directly handle symbols data-hungry and thus is not suitable when data size is small difficult to handle long tail phenomena model is usually a black box and is difficult to understand computational cost of learning is high unsupervised learning methods need to be developed still lacks of theoretical foundation Advantages and challenges of deep learning for natural language processing. good at pattern recognition problems data-driven, and performance is high in many problems end-to-end training: little or no domain knowledge is needed in system construction learn