How do you use natural language processing to extract insights from market research texts?

NLP has been used in the B2C segment for sentiment analysis and gathering buying signals. However, I haven't seen many businesses In the B2B segment doing it, and those who do, have the edge. Having executed it personally, I can say it is a game changer and gives you very interesting insights for instance, you get a sense of where your target audience is vs where the buzz around the brand is.

In my experience, NLP has tremendous potential to improve communication and benefit a wide range of industries. As NLP model training improves so does its accuracy. The future is looking more and more interesting.

AI and NLP are totally dependent on the question itself. If you are asking an open-ended brand awareness question, that is a far simpler computational task than “How can Brand X better meet your needs?” I have yet to use any AI tool that is not also enhanced by a careful manual review of a subset of comments to assess integrity of the coding structure.

Sentiment Analysis normally sucks, because its importance is greatly over-rated. Framing the context is way more important to understand the meaning. By focusing on the context first and on sentiment after, accuracy is improved tenfolds. Another typical flaw of sentiment analysis is when it is applied to entire paragraphs of text. Sentiment can't be "averaged". A paragraph could be ironically positive in 90% of the sentences and then be very negative only in one single phrase, changing the interpretation of the whole thing. Again, context over sentiment. ADVICE: don't rely on Sentiment alone. You will end up with really poor data. Look for next-gen AI technologies that focus on the context instead.

Agree completely. Sentiment analysis is a limited and naive approach to understanding respondent emotion and generally ignores context. Let’s not forget that any coding structure obscures the granular feedback that could be the source of a great insight, in the same way that factor analysis reduces attribute data to underlying themes. Something (variance) is always lost.

Most of the topic modeling tools out there are based on a semantic approach, meaning that they find topics starting from keywords and semantic trees. The problem with this approach is that A) it is language-dependent and B) it automatically introduces an enormous amount of Selection Bias, making the resulting data useless (or even dangerously misleading). ADVICE: adopt a semantic-less approach. Look into geometrical structures. Avoid bias.

NLG is the go-to technology for this task. However, the importance of summarization is greatly overestimated. Summarization easily becomes over-simplification, which is extremely dangerous in today's over-complex scenarios. Use your head instead. Don't become a passive ChatGPT user, be always in control and remember that the quality of your questions - not an NLP model - determines the quality of your actions.

The most important thing - before even discussing NLP - is data organization. The Noise to Signal Ratio must be kept as low as possible. By putting together a lot of text from different sources, the aforementioned ratio rises to unbearable levels. One good piece of advice is to avoid Social Media or any other source that can not be unequivocally assigned to a specific product/service experience. Social Media Listening tools are useless because the Noise is too high making them unactionable. Choose sources that can be organized/segmented around specific experiences instead (like Consumer reviews, Call transcripts, etc.).