New Product Development (NPD) through Social Media-based Analysis by Comparing Word2Vec and BERT Word Embeddings

Abstract

This study introduces novel methods for sentiment and opinion classification of tweets to support the New Product Development (NPD) process. Two popular word embedding techniques, Word2Vec and BERT, were evaluated as inputs for classic Machine Learning and Deep Learning algorithms to identify the best-performing approach in sentiment analysis and opinion detection with limited data. The results revealed that BERT word embeddings combined with Balanced Random Forest yielded the most accurate single model for both sentiment analysis and opinion detection on a use case. Additionally, the paper provides feedback for future product development performing word graph analysis of the tweets with same sentiment to highlight potential areas of improvement.