Sebastian Ruder
follow
https://ruder.io
About
follow
https://ruder.io/about/
Tags
follow
https://ruder.io/tags/
Papers
follow
https://ruder.io/publications/
Talks
follow
https://ruder.io/talks/
News
follow
https://ruder.io/news/
FAQ
follow
https://ruder.io/faq/
Newsletter
follow
https://ruder.io/nlp-news/
NLP Progress
follow
https://nlpprogress.com/
Media
follow
https://ruder.io/media/
Contact
follow
https://ruder.io/contact/
follow
https://twitter.com/seb_ruder
follow
https://ruder.io/rss/index.rss
natural language processing Challenges and Opportunities in NLP Benchmarking Over the last years, models in NLP have become much more powerful, driven by advances in transfer learning. A consequence of this drastic increase in performance is that existing benchmarks
follow
nlp-benchmarking/
Sebastian Ruder
follow
author/sebastian/
follow
recent-advances-lm-fine-tuning/
language models Recent Advances in Language Model Fine-tuning This article provides an overview of recent methods to fine-tune large pre-trained language models.
follow
recent-advances-lm-fine-tuning/
Sebastian Ruder
follow
author/sebastian/
follow
research-highlights-2020/
transfer learning ML and NLP Research Highlights of 2020 This post summarizes progress in 10 exciting and impactful directions in ML and NLP in 2020.
follow
research-highlights-2020/
Sebastian Ruder
follow
author/sebastian/
follow
nlp-beyond-english/
cross-lingual Why You Should Do NLP Beyond English 7000+ languages are spoken around the world but NLP research has mostly focused on English. This post outlines why you should work on languages other than English.
follow
nlp-beyond-english/
Sebastian Ruder
follow
author/sebastian/
follow
10-tips-for-research-and-a-phd/
advice 10 Tips for Research and a PhD This post outlines 10 things that I did during my PhD and found particularly helpful in the long run.
follow
10-tips-for-research-and-a-phd/
Sebastian Ruder
follow
author/sebastian/
follow
research-highlights-2019/
natural language processing 10 ML & NLP Research Highlights of 2019 This post gathers ten ML and NLP research directions that I found exciting and impactful in 2019.
follow
research-highlights-2019/
Sebastian Ruder
follow
author/sebastian/
follow
unsupervised-cross-lingual-learning/
cross-lingual Unsupervised Cross-lingual Representation Learning This post expands on the ACL 2019 tutorial on Unsupervised Cross-lingual Representation Learning. It highlights key insights and takeaways and provides updates based on recent work, particularly unsupervised deep multilingual models.
follow
unsupervised-cross-lingual-learning/
Sebastian Ruder
follow
author/sebastian/
follow
state-of-transfer-learning-in-nlp/
transfer learning The State of Transfer Learning in NLP This post expands on the NAACL 2019 tutorial on Transfer Learning in NLP. It highlights key insights and takeaways and provides updates based on recent work.
follow
state-of-transfer-learning-in-nlp/
Sebastian Ruder
follow
author/sebastian/
events EurNLP The first European NLP Summit (EurNLP) will take place in London on October 11, 2019. It is an opportunity to foster discussion and collaboration between researchers in and around Europe.
follow
eurnlp/
Sebastian Ruder
follow
author/sebastian/
events NAACL 2019 Highlights This post discusses highlights of NAACL 2019. It covers transfer learning, common sense reasoning, natural language generation, bias, non-English languages, and diversity and inclusion.
follow
naacl2019/
Sebastian Ruder
follow
author/sebastian/
transfer learning Neural Transfer Learning for Natural Language Processing (PhD thesis) This post discusses my PhD thesis Neural Transfer Learning for Natural Language Processing and some new material presented in it.
follow
thesis/
Sebastian Ruder
follow
author/sebastian/
follow
aaai-2019-highlights/
events AAAI 2019 Highlights: Dialogue, reproducibility, and more This post discusses highlights of AAAI 2019. It covers dialogue, reproducibility, question answering, the Oxford style debate, invited talks, and a diverse set of research papers.
follow
aaai-2019-highlights/
Sebastian Ruder
follow
author/sebastian/
follow
4-biggest-open-problems-in-nlp/
natural language processing The 4 Biggest Open Problems in NLP This is the second post based on the Frontiers of NLP session at the Deep Learning Indaba 2018. It discusses 4 major open problems in NLP.
follow
4-biggest-open-problems-in-nlp/
Sebastian Ruder
follow
author/sebastian/
follow
10-exciting-ideas-of-2018-in-nlp/
transfer learning 10 Exciting Ideas of 2018 in NLP This post gathers 10 ideas that I found exciting and impactful this year—and that we"ll likely see more of in the future. For each idea, it highlights 1-2 papers that execute them well.
follow
10-exciting-ideas-of-2018-in-nlp/
Sebastian Ruder
follow
author/sebastian/
follow
emnlp-2018-highlights/
events EMNLP 2018 Highlights: Inductive bias, cross-lingual learning, and more This post discusses highlights of EMNLP 2018. It focuses on talks and papers dealing with inductive bias, cross-lingual learning, word embeddings, latent variable models, language models, and datasets.
follow
emnlp-2018-highlights/
Sebastian Ruder
follow
author/sebastian/
follow
hackernoon-interview/
natural language processing HackerNoon Interview This post is an interview by fast.ai fellow Sanyam Bhutani with me. It covers my background, advice on getting started with NLP, writing technical articles, and more.
follow
hackernoon-interview/
Sebastian Ruder
follow
author/sebastian/
follow
a-review-of-the-recent-history-of-nlp/
language models A Review of the Neural History of Natural Language Processing This post expands on the Frontiers of Natural Language Processing session organized at the Deep Learning Indaba 2018. It discusses major recent advances in NLP focusing on neural network-based methods.
follow
a-review-of-the-recent-history-of-nlp/
Sebastian Ruder
follow
author/sebastian/
follow
acl-2018-highlights/
natural language processing ACL 2018 Highlights: Understanding Representations and Evaluation in More Challenging Settings This post discusses highlights of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018). It focuses on understanding representations and evaluating in more challenging scenarios.
follow
acl-2018-highlights/
Sebastian Ruder
follow
author/sebastian/
natural language processing NLP"s ImageNet moment has arrived Big changes are underway in the world of NLP. The long reign of word vectors as NLP"s core representation technique has seen an exciting new line of challengers emerge. These approaches demonstrated that pretrained language models can achieve state-of-the-art results and herald a watershed moment.
follow
nlp-imagenet/
Sebastian Ruder
follow
author/sebastian/
follow
tracking-progress-nlp/
natural language processing Tracking the Progress in Natural Language Processing Research in ML and NLP is moving at a tremendous pace, which is an obstacle for people wanting to enter the field. To make working with new tasks easier, this post introduces a resource that tracks the progress and state-of-the-art across many tasks in NLP.
follow
tracking-progress-nlp/
Sebastian Ruder
follow
author/sebastian/
follow
highlights-naacl-2018/
natural language processing Highlights of NAACL-HLT 2018: Generalization, Test-of-time, and Dialogue Systems This post discusses highlights of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018). It focuses on Generalization, the Test-of-Time awards, and Dialogue Systems.
follow
highlights-naacl-2018/
Sebastian Ruder
follow
author/sebastian/
semi-supervised learning An overview of proxy-label approaches for semi-supervised learning While unsupervised learning is still elusive, researchers have made a lot of progress in semi-supervised learning. This post focuses on a particular promising category of semi-supervised learning methods that assign proxy labels to unlabelled data, which are used as targets for learning.
follow
semi-supervised/
Sebastian Ruder
follow
author/sebastian/
follow
text-classification-tensorflow-estimators/
tensorflow Text Classification with TensorFlow Estimators This post is a tutorial that shows how to use Tensorflow Estimators for text classification. It covers loading data using Datasets, using pre-canned estimators as baselines, word embeddings, and building custom estimators, among others.
follow
text-classification-tensorflow-estimators/
Sebastian Ruder
follow
author/sebastian/
follow
requests-for-research/
transfer learning Requests for Research It can be hard to find compelling topics to work on and know what questions to ask when you are just starting as a researcher. This post aims to provide inspiration and ideas for research directions to junior researchers and those trying to get into research.
follow
requests-for-research/
Sebastian Ruder
follow
author/sebastian/
follow
deep-learning-optimization-2017/
optimization Optimization for Deep Learning Highlights in 2017 Different gradient descent optimization algorithms have been proposed in recent years but Adam is still most commonly used. This post discusses the most exciting highlights and most promising recent approaches that may shape the way we will optimize our models in the future.
follow
deep-learning-optimization-2017/
Sebastian Ruder
follow
author/sebastian/
follow
word-embeddings-2017/
word embeddings Word embeddings in 2017: Trends and future directions Word embeddings are an integral part of current NLP models, but approaches that supersede the original word2vec have not been proposed. This post focuses on the deficiencies of word embeddings and how recent approaches have tried to resolve them.
follow
word-embeddings-2017/
Sebastian Ruder
follow
author/sebastian/
follow
multi-task-learning-nlp/
multi-task learning Multi-Task Learning Objectives for Natural Language Processing Multi-task learning is becoming increasingly popular in NLP but it is still not understood very well which tasks are useful. As inspiration, this post gives an overview of the most common auxiliary tasks used for multi-task learning for NLP.
follow
multi-task-learning-nlp/
Sebastian Ruder
follow
author/sebastian/
follow
highlights-emnlp-2017/
natural language processing Highlights of EMNLP 2017: Exciting datasets, return of the clusters, and more This post discusses highlights of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). These include exciting datasets, new cluster-based methods, distant supervision, data selection, character-level models, and many more.
follow
highlights-emnlp-2017/
Sebastian Ruder
follow
author/sebastian/
follow
learning-select-data/
domain adaptation Learning to select data for transfer learning Domain adaptation methods typically seek to identify features that are shared between the domains or learn representations that are general enough to be useful for both domains. This post discusses a complementary approach to domain adaptation that selects data that is useful for training the model.
follow
learning-select-data/
Sebastian Ruder
follow
author/sebastian/
follow
deep-learning-nlp-best-practices/
natural language processing Deep Learning for NLP Best Practices Neural networks are widely used in NLP, but many details such as task or domain-specific considerations are left to the practitioner. This post collects best practices that are relevant for most tasks in NLP.
follow
deep-learning-nlp-best-practices/
Sebastian Ruder
follow
author/sebastian/
multi-task learning An Overview of Multi-Task Learning in Deep Neural Networks Multi-task learning is becoming more and more popular. This post gives a general overview of the current state of multi-task learning. In particular, it provides context for current neural network-based methods by discussing the extensive multi-task learning literature.
follow
multi-task/
Sebastian Ruder
follow
author/sebastian/
follow
transfer-learning/
transfer learning Transfer Learning - Machine Learning"s Next Frontier Deep learning models excel at learning from a large number of labeled examples, but typically do not generalize to conditions not seen during training. This post gives an overview of transfer learning, motivates why it warrants our application, and discusses practical applications and methods.
follow
transfer-learning/
Sebastian Ruder
follow
author/sebastian/
follow
highlights-nips-2016/
meta-learning Highlights of NIPS 2016: Adversarial learning, Meta-learning, and more The Conference on Neural Information Processing Systems (NIPS) is one of the top ML conferences. This post discusses highlights of NIPS 2016 including GANs, the nuts and bolts of ML, RNNs, improvements to classic algorithms, RL, Meta-learning, and Yann LeCun"s infamous cake.
follow
highlights-nips-2016/
Sebastian Ruder
follow
author/sebastian/
follow
cross-lingual-embeddings/
cross-lingual A survey of cross-lingual word embedding models Monolingual word embeddings are pervasive in NLP. To represent meaning and transfer knowledge across different languages, cross-lingual word embeddings can be used. Such methods learn representations of words in a joint embedding space.
follow
cross-lingual-embeddings/
Sebastian Ruder
follow
author/sebastian/
follow
emnlp-2016-highlights/
natural language processing Highlights of EMNLP 2016: Dialogue, deep learning, and more This post discusses highlights of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016). These include work on reinforcement learning, dialogue, sequence-to-sequence models, semantic parsing, natural language generation, and many more.
follow
emnlp-2016-highlights/
Sebastian Ruder
follow
author/sebastian/
word embeddings On word embeddings - Part 3: The secret ingredients of word2vec Word2vec is a pervasive tool for learning word embeddings. Its success, however, is mostly due to particular architecture choices. Transferring these choices to traditional distributional methods makes them competitive with popular word embedding methods.
follow
secret-word2vec/
Sebastian Ruder
follow
author/sebastian/
follow
lisbon-machine-learning-summer-school-highlights/
events LxMLS 2016 Highlights The Lisbon Machine Learning School (LxMLS) is an annual event that brings together researchers and graduate students in ML, NLP, and Computational Linguistics. This post discusses highlights, key insights, and takeaways from the 6th edition of the summer school.
follow
lisbon-machine-learning-summer-school-highlights/
Sebastian Ruder
follow
author/sebastian/
follow
word-embeddings-softmax/
word embeddings On word embeddings - Part 2: Approximating the Softmax The softmax layer is a core part of many current neural network architectures. When the number of output classes is very large, such as in the case of language modelling, computing the softmax becomes very expensive. This post explores approximations to make the computation more efficient.
follow
word-embeddings-softmax/
Sebastian Ruder
follow
author/sebastian/
follow
word-embeddings-1/
word embeddings On word embeddings - Part 1 Word embeddings popularized by word2vec are pervasive in current NLP applications. The history of word embeddings, however, goes back a lot further. This post explores the history of word embeddings in the context of language modelling.
follow
word-embeddings-1/
Sebastian Ruder
follow
author/sebastian/
follow
optimizing-gradient-descent/
optimization An overview of gradient descent optimization algorithms Gradient descent is the preferred way to optimize neural networks and many other machine learning algorithms but is often used as a black box. This post explores how many of the most popular gradient-based optimization algorithms such as Momentum, Adagrad, and Adam actually work.
follow
optimizing-gradient-descent/
Sebastian Ruder
follow
author/sebastian/
Sebastian Ruder
follow
https://ruder.io
About
follow
https://ruder.io/about/
Tags
follow
https://ruder.io/tags/
Papers
follow
https://ruder.io/publications/
Talks
follow
https://ruder.io/talks/
News
follow
https://ruder.io/news/
FAQ
follow
https://ruder.io/faq/
Newsletter
follow
https://ruder.io/nlp-news/
NLP Progress
follow
https://nlpprogress.com/
Media
follow
https://ruder.io/media/
Contact
follow
https://ruder.io/contact/
follow
https://twitter.com/seb_ruder
follow
https://ruder.io/rss/index.rss
Sebastian Ruder
follow
https://ruder.io
Latest Posts
follow
https://ruder.io
Twitter
follow
https://twitter.com/seb_ruder
Ghost
follow
https://ghost.org