Big Data and New Product Development

Big Data and New Product Development How do firms develop ideas, turn them into products, and decide which ones to bring to market? Most firms do so through a series of steps known as the New Product Development (NPD) or the Stage Gate process: idea generation; idea screening; idea development and testing; business analysis; beta testing and market testing; technical implementation; commercialization; and new product pricing.These steps, or stages (formalized by Dr. Robert G. Cooper in his book Robert’s Rules of Innovation based on industry research), are loose, with steps performed concurrently, and/or eliminated if unnecessary, and flexible enough to provide for firm or industry variation. This framework is also ongoing, with firms, ideally in a state of continuous development. © Shutterstock.com | tandaVThe era of Big Data has created substantial opportunities for developing products aligned with consumer demands, forecasting their profitability, and production. Using the NPD framework in this article, we will discuss 1) the benefits of using big data in new product design, 2) transforming Big Data into actionable consumer insights, 3) developing new products using Big Data, 4) improving existing products using Big Data, 4) a case study of how Big Data informs and enhances Procter and Gamble’s new product development.THE BENEFITS OF USING BIG DATA IN NEW PRODUCT DEVELOPMENTUsing Big Data to inform new product development has many benefits. Firms can create products that connect with the consumer, provide increased consumer value, minimize the risks associated with a new product’s launch, and both allocate, and coordinate the use of, internal RD resources efficiently. Through data mining, firms can also identify consumer needs it might not otherwise ha ve captured. By continuously developing products that fulfill consumer needs, firms can deepen customer brand engagement and increase customer lifetime value. Through modeling and predictive analytics, the firm can forecast the performance of the product(s) in the market both pre- and post-launch in near-real-time, determine the optimal distribution chains, and optimize marketing strategies to acquire the greatest number of customers at the lowest cost.In sum, Big Data can help transforming big data into actionable consumer insights; develop new products; and improve existing ones.TRANSFORMING BIG DATA INTO ACTIONABLE CONSUMER INSIGHTSWhen firms pare business intelligence tools, data mining, predictive analytics, and other Big Data tools with traditional market research techniques in order to collect actionable insights about their consumers’ needs, and/or similar or related brands/products, firms are able to develop a proactive approach to new product development. They are able t o innovate by developing entirely new products, as well as identify opportunities to introduce new product features, new product extensions and/or improve existing product lines. By developing a proactive, rather than reactive approach in which they are responding to the actions of competitors, they are able to ensure product quality, brand consistency, and marketing effectiveness, and exert more influence in their market. Further, they are able to minimize the uncertainty that comes with new product launches, as failures can be quite expensive. This can be a springboard for stage one, idea generation, and aid greatly in stage two, the idea screening process.NPD teams â€" usually cross-functional groups consisting of marketers, engineers and data scientists, working in firms that implement Big Data architecture can mine their internal databases from across the firm, as well as firm data and industry data from external sources. Firms can filter and analyze this data to determine exis ting, latent, and untapped consumer needs; these needs may inspire product ideas and concepts. This analysis may also be used in the second stage, idea screening.Idea screening involves filtering out ideas that do not provide sufficient customer value, satisfy a profitable target market, face too much competition, and/or are difficult to produce. Internal data can answer these questions and more, allowing product developers to sharply refine the ideas they will pursue long before they conduct a single focus group.Stage three, which involves consumer outreach, involves a further refinement of ideas for profitability, supply chain logistics, originality, and consumer acceptance. Once again, Big Data can greatly aid in this endeavor. Firms may be able to pull detailed manufacturing data or supply chain data to determine the feasibility of production or distribution respectively. Data scientists can build mathematical models of the product’s hypothetical production and distribution, c osts, and use predictive analytics to develop revenue and profitability projections. Beyond determining overall feasibility, these models can help determine optimal conditions for product launch; enhance focus group discussions and surveys, allowing the firm’s market researchers to drill down on specific aspects of the hypothetical product; and further help discard unprofitable ideas.Stage four business analysis, involves projections â€" demand, performance, and profitability. Predictive analytics play a huge role here, though in the absence of historical data on which to draw, data scientists must forecast using different mathematical models. There are three primary methods for predicting new product success: the Bass model; the Fourt-Woodlock model; and the Assessor model.Bass model: Data scientists using this model try to estimate the shape of the demand curve for existing products and apply it to the new products.Fourt-Woodlock model: This model can be used to estimate produc t sales; it is based on the number of consumers who make trial purchases and those who repeat those purchases within the first year of the product being on the market.Assessor model: This model relies on assessments of the strength of the firm’s brand, and is used to project both brand preference and brand awareness over time, the latter by analysis of the firm’s planned marketing mix.Firms may use other measures to project product performance in the absence of historical sales, including internal capacity; online and offline conversion rates for similar products in the firm’s portfolio; sales performance forecasting (especially for firms using direct sales methods); analysis of the firm’s other new product launches, among others. Big Data can provide a multiplicity of variables with which to refine a firm’s forecasts. However, it is worth noting that all of the aforementioned measures typically entail a much higher degree of uncertainty than regression-based forecasts usi ng historical data.Stages two through four allow firms to broaden their criteria for what constitutes a profitable product or target market. Optimizing marketing, production, distribution and pricing, as well as employing market customization strategies, can allow firms to match the products to the consumer at the highest margin. Take a well-known manufacturer of thermal products that is considering introducing a new branded thermal curtain that is 30% more efficient than its closest competitor, but is expensive to produce. The producer might develop models of just those target consumers willing to buy the curtains at the highest price point; develop, test and refine marketing messages; forecast the demand for the product, accounting for seasonality; and enter into an agreement with a low cost third-party manufacturer to produce the branded curtains on demand and directly ship them to the customer. A product that might otherwise have been scuttled due to high costs, might become a c ash cow for the manufacturer.The product concept that survives elimination at this point, is now ready for prototyping and beta testing begins in the following stage. Using mathematical models of target consumers before beta tests can yield insights on potential adoption rates, necessary marketing and sales strategies, optimal distribution channels, and desired product features and functions. Here, the theoretical meets the practical, as consumers provide their feedback. The firms that maximize Big Data will scour this feedback not just for insights about the product being beta tested, but also its overall brand and other exiting products.DEVELOPING NEW PRODUCTS USING BIG DATAOnce beta testing is complete and successful, firms then begin to determine how to scale the product’s manufacturing and integrate it into existing operations. This includes everything from determining optimal suppliers to contingency planning. Firms can use optimization models to predict quality and yield; a ccount for variation in production processes down to the machine or individual level, as well as outputs; forecast demand in order to set target yields; employ mass customization strategies; and determine return on investment for every component in the production process. This data can strengthen decision-making, and yield both higher ROI and greater performance.Stage seven, or commercialization, involves the actual product launch. Optimization models can predict the national, regional, or local distribution targets most likely to yield the greatest levels of consumer adoption with the lowest customer acquisition cost. This can help inform the ideal launch location(s), which will in turn inform the distribution strategy. Further, Big Data management tools can be used to optimize the operational aspects of the distribution chain, from packaging to delivery scheduling.Modeling tools can also help optimize media planning the process of finding the media (advertising, public relations, digital) channels that will help a firm achieve its marketing goals. Digital advertising, in particular, provides a wealth of performance data that, when analyzed, can yield terabytes of insights about consumer behavior and consumer purchase behavior in real-time. Marketing analytics firms and in-house quantitative marketing teams can analyze the impact of marketing across channels and across media, allowing firms to evaluate their marketing performance and adjust their marketing strategies in real-time to meet and exceed marketing objectives.The last stage, which begins earlier in the framework, involves adjusting pricing to reflect actual (rather than projected) supply, production, and distribution costs, as well as market demand, sales, and responses from competitors . This also involves assessing the new product’s actual performance in context to the firm’s overall product portfolio.IMPROVING PRODUCTS USING BIG DATASometimes, the consumer insights captured through market res earch about a new product involve the firm’s existing products. For example, customer service feedback is often ripe with constructive criticism about a firm’s existing product line â€" insights that can be used when launching a brand extension of a product. Aggregating that data and feeding it to product marketers at the idea generation stage can be a great source of new product ideas and concepts. Further, firms can mine social networks, industry websites, and other online sources for relevant data about their brand and how their products meet (or fail to meet) consumer needs. Firms can use this information to develop solutions to products currently on the market, or build solutions into planned product extensions/next generation products.The Internet of Things â€" a lasting business trend in which firms connect products (all products, but especially those that have been historically unconnected, such as household appliances) through wireless technologies, has tremendous appli cations for proactive product development. By providing firms with real-time data about consumer usage, firms can identify and exploit opportunities to maximize customer revenue and increase product value to the customer. For example, a smart refrigerator, one programmed to keep stock of the items inside it, also may be programmed to retain diagnostic information to aid the firm in preventative and/or emergency maintenance efforts. A firm, upon learning, that repeated customer complaints have been received about a particular feature, can proactively improve it for free or at cost.A firm can also use predictive analytics to determine which product features it should introduce to next generation products that will generate the most return for the least cost. For example, a firm might beta test a new video game console with different features, such as new controllers, wireless apps, and games, and analyze usage and purchase behavior to determine which combination should be rolled out w ith a mass-market strategy. Alternatively, the firm can use this data to determine pricing for various customized versions of the console that will enable the firm to achieve its revenue and profit goals.PROCTER AND GAMBLE: A CASE STUDYProcter and Gamble (PG), the consumer goods manufacturer, is one firm that has leveraged Big Data successfully into its new product development process, by aggregating consumer data from multiple brand touchpoints and using it to both launch and promote new products. They use modeling and simulation tools extensively to minimize prototyping expenses. For example, they’ve used them to determine how the molecules in certain household products like dishwashing liquids will react over time to refine the product.PG is not only able to use Big Data as a springboard for new ideas; it is able to strategically plan, produce, and launch them. Among other internal business initiatives which optimize operations, PG has developed what its former CEO, Robert McDo nald refers to as consumer pulse, which aggregates and filters external data, such as comments and news mentions, using Bayesian analysis (a method of statistical inference used for the dynamic analysis of data sequences) on PGs products and brands in real-time, allowing them to react as market developments occur.PG has also implemented a system called Control Tower, which provides real-time data on all transportation activity at PG in over 80 countries. They’ve used this system to not only improve their transportation, but also to reduce their carbon footprint. They use a similar system called Distributor Connect, which lets them manage inventory in real-time. Moreover, the firm keeps connected to their retailers through a globally synchronized data warehouse that allows them to manage commercial transactions in a completely automated fashion.These systems all aggregate data, which are harnessed by PG’s marketers, data scientists, and engineers to develop new products and impro ve existing ones. The firm spends almost $2 billion dollars annually on RD, and in recent years has worked to systematize innovation by creating multiple groups responsible for creating new products and development. Rather than innovation being pigeon-holed into a single department, PG linked firm-wide company, business and innovation strategies together for senior executive leadership to review. Moreover, it harnessed Big Data tools and testing. Tide Dry Cleaners, a branded dry cleaning franchise, for example, was developed by leveraging consumer insights about the deficits of the existing dry cleaning industry, its brand, and its own insights into consumer household cleaning habits. There are Tide Dry Cleaners all over the country, featuring 24-hour pickup, drive-through service, and environmentally safe cleaning processes â€" all consumer preferences PG packages into a franchise and sells entrepreneurs for a hefty fee.