Philosopher Ludwig Wittgenstein was a central figure in "the linguistic turn." His insights into human communication can help us puzzle through the complexities of natural language processing.
In his first major work, Tractatus Logico-Philosophicus, Wittgenstein explored the question, how do human beings manage to communicate with one another? His answer was that our words trigger concrete pictures in other people’s minds. But here is the rub, Wittgenstein also observed that we are not very good at painting good pictures in other's minds, so miscommunication often ensues.
He describes a thought as “a logical picture of facts.” For example, when I say “A willow tree sways in the wind,” my words should invoke a concrete picture of the scene. If we were to take out all of the stopwords, or common words, we would be left with “willow”, “tree”, “sways”, “wind” and still be able to get the gist of the meaning. This is exactly what we do when preprocessing data for NLP models.
In his second major work, Philosophical Investigations, published posthumously, Wittgenstein’s theory evolves to better match the complexity of communication. More than words that describe picture facts, he says, language is a tool with which we play different games or patterns of intention. How we communicate and the context in which the discourse is embedded is as important as the words themselves. The example above about the willow tree is from a “stating facts” game. But if your best friend tells you “It’s all going to work out for the best.” Unless they are clairvoyant, they don’t actually know how it is going to work out. But they are not playing the “stating facts game”; they are playing the “comforting game.” Knowing what language “game” is important for understanding the meaning and intention behind the words.
In NLP terms, feature engineering is the process by which domain knowledge is applied to data. As a recap from Part 1, features are attributes or properties shared by all independent units. They are what make machine learning actionable, because they give the necessary context for a model to make a prediction or analysis.
Coming up with features is difficult, time-consuming, requires expert knowledge. "Applied machine learning" is basically feature engineering.
- Andrew Ng, Machine Learning and AI via Brain simulations
Feature engineering is key because this is where the art, creativity, and “folk wisdom” come in. It is through features that we imbue our cultural knowledge of how language works in the real world. NLP will remain an exciting area of machine learning because we are always inventing and evolving how we express ourselves to one another.