A few weeks ago, I wrote about the benefits of retail sales data for CPG manufacturers. And I summarized the four types of retail sales data which are formed by a combination of two parameters: data source (syndicated vs. retailer direct) and data focus (store vs. shopper). In this post, I’ll discuss data source in detail, describing the positives and negatives of both retailer direct and syndicated (aka Nielsen/IRI) data. Retail sales data can come either direct from the retailer (reflecting only sales for that retailer) or through a third-party syndicated provider (who pools data from many retailers to create a picture of the total market).
Retailer Direct Data
As you might guess, retailer direct data focuses on your products in a single retailer. It won’t tell you anything about competing stores. And often you won’t get information on any competing products, only your own items and maybe some category totals.
If you want to dig into the details of doing business with a particular retailer and build a stronger collaboration, then retailer direct sales data is a great place to start. Retailer direct data allows you to look at a retailer’s data their way and that’s valuable. But it’s not as good at giving you an overall picture of your business. If the retailer is big and geographically dispersed you may glean some general learnings from the data. But if general learnings are your primary objective, or you want to easily compare across retailers, then you need to use syndicated market data instead.
The best example of retailer direct data is Walmart’s Retail Link system, which is an entire data ecosystem available to all of Walmart’s vendors. Many other retailers provide data through data portals or other delivery mechanisms. Some retailers provide data to their suppliers that is really focused on supply chain and logistics with less emphasis on retail store sales. The format and scope of retailer data varies widely and every retailer can easily be a whole data project until itself. There are software and consulting companies that specialize in helping manufacturers manage all their different streams of retail data.
Some retailers even outsource data management and delivery to a third party! Kroger and Ahold are two big retailers who have taken this route. Even though the data is technically coming from a third-party and not the retailer itself, it’s still not the same as syndicated data.
Sometimes direct retailer data is provided for free, such as Walmart’s Retail Link. In other cases, the retailer or provider will charge for data access. The cost of the data largely depends on what’s provided and the strategic objectives of the retailer.
Retailer direct data may be available for some products and channels that aren’t covered by the syndicated data vendors. In some cases, retailer direct data may be your only data option.
Syndicated Data
Syndicated data is the primary subject of this blog. This third-party data covers the full landscape of brick and mortar shopper choices, from stores to products. It can include and aggregate all stores (well, almost all stores) in a geographic market or channel and capture all products in most major categories. You can see how your competitors are selling, analyze adjacent categories, and even look at sales trends for products with certain characteristics (e.g. everything that is gluten-free). Because it provides a complete picture of the market, it’s sometimes referred to as “market data”.
Nielsen, IRI, and SPINS are the primary providers of syndicated market data for the CPG industry. Any manufacturer can purchase their data and retailers typically receive analytical support in exchange for sharing their data with the providers. IRI and Nielsen cover Food, Drug, Mass, Club, Military, Convenience and Dollar channels. SPINS specializes in the Natural Grocery and Specialty Gourmet channels and Natural and organic products in the Conventional Grocery channel (through a partnership with IRI).
Unfortunately a few big retailers don’t cooperate with IRI/Nielsen and SPINS, such as Whole Foods, Aldi, Trader Joes, and Dollar Tree. Costco provides some data but only through IRI and the data is limited. IRI/Nielsen and SPINS use statistical projection and a variety of sources to account for the “missing” retailers in market level and national aggregates. NOTE: The list of non-cooperating vendors can change at any time, so don’t hold us to it! Ask the vendors for precise, current information on cooperating retailers.
Even though syndicated data vendors capture every product that scans in these stores, they don’t organize and process the data for every category. If there is no client willing to pay for the data, the vendors are much less likely to invest the time and money necessary. They’re also less likely to cover slow-moving categories, general merchandise categories in grocery, and home and garden categories in the mass merchandisers. If you want to find out if your product is covered, call the vendors and ask! They will be happy to help you figure it out.
Syndicated data generally requires a significant dollar investment. Nielsen and IRI both publish category trend information in white papers and webinars at no cost. You may also find good information in trade publications or even the general press. But if you want a detailed analysis of your category, in your key market, for a particular time period, then you’ll have to pay for it. Costs range from a couple of thousand dollars for a single report to millions of dollars a year for ongoing subscriptions that include data analysis software tools.
I went to touch on one data source that doesn’t fit neatly into either side of this syndicated vs. retailer direct dichotomy: Kroger data from dunnhumby. Though it is only Kroger, the individual Kroger banners are so numerous, varied, and geographically dispersed that it can give you a much more general perspective on sales than data from most single retailers. In addition, reports and analysis are available that will give you a full perspective on the category and competitors. The dunnhumby offerings closely mirror the type of information available from Nielsen and IRI. So I would put it on the Retailer Direct side, since it’s not covering the full market, but it’s nudging over towards Syndicated.
Convinced you need syndicated data but need help taking the next step? Contact us to learn more about our consulting services. We can help you move your syndicated data project forward.
Looking for training on this topic? I recommend Category Management Knowledge Group’s online class Understanding and Using Data for an introduction to retail sales data analysis techniques and data sources. If you already know the basics, check out Building Data Competency: POS Data which goes into much more detail on retailer direct and syndicated data.
Did you find this article useful? Subscribe to CPG Data Tip Sheet to get future posts delivered to your email in-box. We publish articles once or twice a month. We won’t share your email address with anyone.
I would like to better understand how many stores within a chain or market are typically tracked by the syndicated data providers. Do they have real census data? How is the quality of their audit data for displays?
Thanks
The vast majority of retailers supply Nielsen and IRI with data for all stores (“census”). I would say 98-99%. This means many of the measures can be census based (sales, distribution, any ACV weighted measures, many pricing measures).
Some measures do not rely on data coming in from retailers. Feature ads are manually collected and coded by Nielsen/IRI. So those measures are impacted by Nielsen/IRI feature coding decisions and methodologies and are not coming directly from the stores. However, Nielsen/IRI do collect the ads quite comprehensively – they are attempting to pick up every ad that is relevant to their markets.
Displays are the one true “sample” measure. According to my co-blogger Robin Simon, there are about 3,000 stores in the audit sample for displays which is about 10% of all $2MM+ stores. For most manufacturers, this will be the best measure of display that they have available. It’s not feasible to go into every store and the Nielsen/IRI approach and sample are comparable across products and trendable across periods since the same methodology is used for everyone. Personally, I have never heard anyone complain about or question the display numbers but I’m sure it’s happened because it will absolutely not be 100% correct.
We launched a new brand nationwide and after 3 months on the shelf the product finally showed up on IRI. If you took the Data as true it showed 1 percent of the actual sales. How is that possible? How can it be corrected? It is so far off the data is useless and dishonest.
Take a look at this post: https://www.cpgdatainsights.com/measure-sales/shipments-vs-retailsales/ which addresses your question. Make sure the product and market definitions are what you are expecting in the IRI data. Keep in mind that shipments from a manufacturer to the retailer are always higher than the sales from a retailer to consumers. The first several weeks of shipments are typically filing the pipeline and take a while to get all the way to the shelf and eventual to the consumer. This is especially true for categories that are not purchased that frequently (home cleaning, personal care, etc.). Don’t give up on the IRI data! Hope this helps.
HOW DOES ONE FIND THE DATA LISTED IN THE COMMENT YOU MADE BELOW?
“Nielsen and IRI both publish category trend information in white papers and webinars at no cost”.
THANKS
Both companies have lots of insights and information on their websites, including recorded webinars and white papers. A good way to stay on top of new information is to subscribe to their newsletters.
http://www.nielsen.com/us/en/newsletter.html
https://www.iriworldwide.com/en-US/company/Subscribe
Is it possible to get scanner data in Nielsen by individual states?
Yes they have “state line” markets but I doubt if they have it for all channels. Probably just Food and Drug.
Can you point me to some of the best/largest “software and consulting companies that specialize in helping manufacturers manage all their different streams of retail data”? Thanks for the great blog!
Although that is a very hot topic, we are only somewhat familiar with these companies and do not feel comfortable endorsing any specific software or firm in that space.
Do you know why Max ACV would differ for a specific item at 1 retailer when comparing SPINS to IRI data? I am noticing a 7% variance, and can’t explain why this would be the case… Thank you!
What time period are you looking at? That could be why the numbers are different between IRI and SPINS. If you are looking at anything other than individual weeks or standard 4-week periods they will usually be different. IRI shows a true max, among all the weeks in the period. SPINS shows the Max 4-week distribution within the period. So if you’re looking at a 12-week period, IRI shows the distribution for the max week during those 12 weeks but SPINS shows the distribution for the max 4-week period among the 3 4-week periods that make up the 12-week period. My guess is that you are looking at a period longer than 4 weeks and the SPINS value is higher than the IRI value. You should confirm this with your SPINS client service team!
What are the key differences between IRI and Nielsen data? Are they very similar or is one superior to the other?
What about their user interfaces? Is one easier or more technologically advanced?
How would a CPG company decide which market research provider is the best option for them?
Great questions! These are not simple questions and I’m glad you also contacted us via email. I’ll follow up with you offline.
Oh! I have the same question as mentioned above. I need to understand the difference between IRI and Nielsen’s data, but I cannot find that anywhere… Can you please point me in the right direction to understand the difference?
As mentioned above, this is not a quick answer. Please click on the “Contact” link and I’d be happy to set up a quick call to discuss further.
How do Nielsen / IRI / SPINS compare on cost for basic services? I know the cost will differ depending on how big your company is and exactly what you need, but all other things being equal, is one considered more expensive / cheaper than the other?
You are right that the cost depends on what you are getting. Key cost drivers are custom vs. syndicated, how many categories and how many SKUs each one has, ad hoc vs. standard reports and some other things. I feel like Nielsen and IRI are pretty comparable for equivalent things. If a company is having Nielsen and IRI bid against each other for the business than I have seen the non-incumbent come in at a lower price for the contract, although typically within 10-15%. Then you have to decide is the upheaval and change management required worth the cost savings. Many small companies get data from SPINS because they have (I believe) 3 different pre-defined packages, including the low end “small business” bundle (or something like that – don’t recall the exact name).
Where would I find data for a specialty market, such as Vape Products? These products are not typically found in big box stores, yet they sell in thousands of small retail stores nationwide.
If you are not aware of any companies that provide data for this market, could you recommend a source to find how the big box stores provide the data to IRI and Nielson (file format, data sets, etc). I have a programming background.
I currently rely on customer requests, marketing campaigns from manufacturers and distributors, and personal product testing to determine what new products to carry in my stores. This is imprecise, time consuming, and often leads to poor choices for new products.
I need data! Please help!
I don’t know of any data source for small retail stores. IRI/Nielsen do have information on the Convenience channel and that data may include e-cigarettes. I suggest you contact them to see what they have to offer. Another research company that sometimes covers channels outside Nielsen/IRI is the NPD Group. If you can’t find any POS data sources, you might look to survey research for a broader take on consumer reaction to potential new products. Survey research can often be done quite cost-effectively. I also don’t know the details on how data flows between IRI/Nielsen and their retail partners.
Sally – thank you for sharing your amazing expertise in this strange little niche. Your site is a great resource.