The Role of the Market Researcher in the Era of Big Data and Artificial Intelligence

The Role of the Market Researcher in the Era of Big Data and Artificial Intelligence

Big data allows companies to collect and store large amounts of information from a wide array of sources and in varied formats.  It does not necessarily provide an integrated analytic framework to assess this information and determine how the information can be applied to a specific business concern.  The insights of a seasoned market researcher are necessary to conceive of an appropriate analytic framework to address a specific concern and to make choices about the quality of the information gleaned through big data.

Machine Learning and Predictive Analytics

Artificial intelligence (AI) can process information according to prescribed algorithms using machine learning and predictive analytics.  It can dramatically shorten the time needed to process and analyze large amounts of data to provide unbiased results. However, the applicability of AI is constrained by the defined context in which it operates and the prescribed algorithms that are available to it.  I am reminded of a time long ago when I was a grad student serving as a research assistant to an economics professor developing a computer program to assess water policy alternatives for a California County Water District. I dutifully supplied the inputs required for the program and ran it several times getting the same nonsensical result.  I reluctantly reported back to my professor that I had discovered a problem with the computer program he was developing. He responded that I had not discovered a “problem” with the program, rather I had discovered a “feature” of the program that would always provide this same result. In the same way, AI will not determine the “sense” of the result, it will only respond with results determined by the “features” (algorithms) that are available to it.  A skilled and experienced market researcher must assess the sense of the result obtained via AI and be cognizant of the algorithms employed by AI in obtaining the result.

The Importance of Behavioral Modeling

Many applications of big data and AI are concerned with understanding consumer behavior and forecasting the likely response to specific products or services.  For example, how will consumers respond to an increase (or decrease) in the relative price of a product, or how will they respond to a change in one or more attributes of the service being provided.  In such cases, market researchers often employ behavioral modeling and/or conjoint measurement modeling techniques to address such concerns. While these are powerful analytic tools, they are also entirely dependent upon the skills and insights of a market researcher to provide the appropriate variables for inclusion in the models and for interpreting the results of such models in the context of the larger business environment.  For example, due to the often observed randomness of consumer behavior, it is quite possible that a specific behavioral model does not explain a large amount of the variance in the outcome being predicted, even though the model does indicate the primary drivers of the outcome. A skilled market researcher can make an informed decision about the limitations of the model and offer suggestions on how to improve the predictive ability of the model.


In a near future that is becoming more enamored with big data and the continued evolution of AI, the role of the market researcher must turn more towards being a steward for the quality of the information being captured, defining the analytic framework to assess that information, and interpreting the results of the analyses being employed to fully exploit that information.


By: Gary Stieger