How to Track Shelves in Retail: Traditional vs Image Recognition-based Approaches

Nowadays, it's brands' responsibility to monitor how their products are placed on the shelves and understand if the pricing is competitive and their promotions are effective. Shelves are the battlefield for CPG companies that sell various kinds of fast-moving consumer goods. Though retailers themselves might be interested in the data (for example, for inventory purposes), CPG companies are the first consumers of data showing how their products are presented across various stores in different countries.

In this article, we will review more traditional approaches to shelf monitoring, such as Nielsen panels, RFID tags and shelves with integrated scales, as well as innovative solutions, including mobile apps with image recognition technology, specialized high-res cameras in stores, mobile robots, and drones with built-in cameras. After reading this article you will be able to understand the advantages and disadvantages of these approaches.

Traditional approaches to shelf monitoring

1. Nielsen panels;
2. RFID tags;
3. Shelves with integrated scales

Nielsen panels

Nielsen is a leading data and analytics company providing views of customer behavior globally. Nielsen panels are quite popular among brands. It is one of the traditional approaches to monitoring products and prices on the shelves.

Although Nielsen panels make it possible to obtain great insights about numerous product categories across multiple countries, they’re usually not entirely helpful. They cover certain stores and do not provide real-time analytics. They instead present data that was up to date several months before. What's more, these panels can cost a fortune; smaller brands can't afford to buy this data.

Solution disadvantages:

• expensive
• no real-time information
• limited number of stores covered

RFID tags

RFID technology was recently revived again after 20 years of being in oblivion. RFID consists of several components: an RFID tag, reader hardware, and supporting software. RFID tags are usually attached to the goods (for instance, clothing) and data stored on them can be read with the help of specialized device readers.

However, RFID solutions proved to be too expensive compared to other similar technologies. RFID signal frequencies are non-standardized. Some consumers consider this solution to be invasive.

Fig 1. RFID clothing tracking examples

Source: https://www.asiarfid.com/rfid-technology-changes-the-garment-industry.html 

Solution disadvantages:

• expensive
• not standardized

Shelves with integrated scales

This technique is designed to weigh all the products placed on the shelf and detect the number of goods on the shelf in real time.

For CPG companies, this solution doesn't meet their requirements and is not entirely reliable, as smart shelves are not designed to distinguish between various products of the same product line with comparable weight. They can provide a manufacturer with inaccurate data about the number of products in stock.

Solution disadvantages:

• can provide inaccurate data due to its inability to distinguish between products of the same brand line but having the same weight.

Shelf monitoring solutions based on image recognition

The popularity of solutions for shelf monitoring based on image recognition technology is inevitably growing because capturing images is fast and natural. The cost of these solutions is very affordable, even for smaller CPG companies. Additionally, this approach allows manufacturers to collect data in all stores where their goods are presented and provide near real-time data updates limited only by the speed of automated data processing (either on the device itself or in the cloud).

On-shelf availability monitoring solutions with integrated image recognition technology onboard include the following:

• mobile apps for merchandisers
• specialized high-res cameras in stores
• mobile robots and drones with built-in cameras

Mobile phones with pre-installed apps

There is a good practice by CPG companies to hire merchandisers to monitor shelves. They either provide them with mobile phones with pre-installed apps which can be helpful in many ways, or instruct them to install a specialized mobile app on their personal devices.

Using mobile apps for merchandising and monitoring shelves has several advantages.
First of all, mobile phones are quite popular these days, so merchandisers probably already have personal devices. In this case, it's quite cheap to deploy free mobile apps on their phones, with payments only for data processing.

Managers can also use mobile apps to provide merchandisers with real-time instructions, for example, to manage out-of-stock situations while they are still in the store.

Though it seems that it's quite an inexpensive solution at first sight, every merchandiser must be paid, and most frequently, they are paid on an hourly basis. Therefore, hiring merchandisers to monitor shelves in all retail stores, sometimes across different countries, can be too expensive for brands.

The other problem is the human factor. Merchandisers might get rewarded with extra money for doing their job at the highest possible level, so they may want to present the best data and pictures from the store. However, life on the shelf is in constant flux, and the situation an hour later can be different from the situation captured and uploaded by a merchandiser.

And here we come to the issue that merchandisers don't provide insights in real time; they take and present the picture at a specific time of the day. Of course, suppose we are talking about larger stores such as hypermarkets. In that case, manufacturers might have lots of products presented, so a merchandiser might have a particular workplace right in the store and spend lots of time there. On the other hand, some stores in remote locations may be visited by merchandisers only once in a while. Consequently, a manager will have only some limited data about the situation on the shelves.

Advantages:

• no capital expenditures (CAPEX)
• almost zero-entry threshold, only data processing should be paid for
• ability to quickly make changes in case of OOS events by providing a merchandiser with instructions

Disadvantages:

• expensive
• merchandisers visit stores occasionally, so data updates are not as frequent as may be required
• data provided by merchandisers is too “good” because they are financially interested

Let’s check out the Inspector Cloud mobile app and see how it helps with shelf monitoring. In the screenshots below, you can see the results of a merchandiser’s visit to a store.

Fig. 2. Screenshots of the Inspector Cloud mobile app

In-store video cameras

Another way to monitor on-shelf availability is to use video cameras. There are several types of video cameras that retailers can potentially use to capture on-shelf situations.

They are:

• CCTV, or security video cameras
• people counter cameras
• cameras with face recognition capabilities
• cameras for essential motion detection and object identification

None of the above-mentioned video camera types are applicable for shelf monitoring. Let’s see why.

CCTV cameras

CCTV cameras are widely used in stores for security reasons. They usually capture grayscale videos of low resolution. The cameras are placed in locations from which every store area can be scanned, most often on the ceiling. Often, the number of cameras is quite limited. 

That's why using this type of camera can be somewhat of a problem for on-shelf availability purposes. The goods and certainly the price tags are almost indiscernible. As a consequence, CCTV cameras cannot provide manufacturers with valuable data about their products on the shelves, as well as about their current prices and prices on the competitors’ items. 

Fig. 3. An example of CCTV camera image

Source: https://www.youtube.com/watch?v=TsLl9GG7acU

People counter cameras

The same can be said about people counter cameras. The primary goal of these cameras in stores is to count people in the queues and notify staff that more cashiers are needed. These video cameras are usually placed in a specific counting zone, so they do not suit the purpose of tracking products on the shelves. 

Fig. 4. An example of a people counter camera image

Source: https://www.securityworldmarket.com/int/News/Advert/people-counting-for-increased-retail-success2

Cameras with face recognition capabilities

Video cameras with face recognition capabilities are used to detect shoplifters or returning customers. The number of use cases is quite large. Still, these cameras won't be helpful in the case of on-shelf availability, as they are placed in locations of the store from which they can catch the most satisfactory angles for face detection and recognition purposes.

Fig. 5. An example of an image from a camera with face recognition capabilities

Source: https://www.aclu.org/blog/privacy-technology/surveillance-technologies/are-stores-you-shop-secretly-using-face

Сameras for essential motion detection and object identification

One more type of video camera is a camera for basic computer vision work, such as motion detection, basic object identification, and similar tasks. A widely known application case is at self-checkout convenience stores. However, though these cameras are designed to track the situation on shelves, this solution is not so good for on-shelf availability. In the image below, such a camera captures several shelves, the operator manually marks the product groups, and it’s quite obvious that this angle of view doesn’t make it possible to collect the proper data which would be suitable for on-shelf availability.

Fig. 6. An image from a camera in a self-checkout store demonstrating how products on shelves can be manually detected

Source: https://www.youtube.com/watch?v=jppQoHUeEl4

As we see, the solutions above don’t suit the needs of CPG companies for shelf monitoring. What are the requirements for video cameras that might help achieve merchandise display analysis

First, the video cameras should be of a high resolution. In Inspector Cloud, we develop and produce 12mpx cameras. Additionally, we plan to present 15mpx cameras on the market in the following months. Also, the cameras should be installed right in front of the shelf to be analyzed. So, the angle of view is another important requirement.

Smart video cameras by Inspector Cloud are entirely autonomous. They are connected to a wireless mobile network; they are equipped with capabilities to capture and detect products, price tags, and even more valuable data, and make an analysis of the obtained images (some data analysis tasks are performed on the cameras, though more complicated tasks are accomplished on the servers). Inspector Cloud cameras can even create one clear image from a few shots made at different periods of time, so the final picture can be easily analyzed.

Fig. 7. An Inspector Cloud camera monitoring a shelf in a store

Smaller video cameras installed on the opposite side of the shelf can also be a solution. With the help of these cameras, only a part of the shelf view is captured, so the resolution of the cameras can be lower. Technologically, this method differs from the one presented by Inspector Cloud.

Why use special cameras for shelf monitoring?

First, Inspector Cloud cameras are available 24/7 in the store. All photos are available in real time, so the whole picture can be viewed dynamically at any time, anywhere.

Once a camera is installed, the operating expenses (OpEx) are much lower. Most costs come from image processing. But capital expenditures (CapEx) can be huge because cameras cost a lot, and if a huge store needs cameras to monitor all shelves, that would cost.

Compared to the shelf monitoring approach using the help of merchandisers, this one is passive as no changes can be made right away.

Advantages:

• real-time monitoring and photo availability
• low OpEx

Disadvantages:

• no way to change the situation, only passive monitoring
• high CAPEX

Autonomous mobile robots (AMRs) and in-store drones

As we mentioned in the previous section related to cameras in retail, in most cases, more than one camera should be installed in a store to efficiently perform the tasks for which they are designed. However, some categories of goods do not require constant monitoring, and there is no need to use several cameras for this purpose.
Therefore, to automate the in-store routine associated with on-shelf availability, retail companies have become a hotbed for innovations. AMRs and in-store drones are good examples of how retail companies are adopting technology for their needs.

Fig. 8. An autonomous mobile robot

Source: https://losspreventionmedia.com/the-use-of-autonomous-mobile-robots-in-retail/

Fig. 9. An in-store drone

Source: https://www.zdnet.com/article/robots-in-store-drones-ai-and-machine-recognition-brick-and-mortar-retailers-are-fighting-back/

However, testing robots in shopping areas turned out to be unsuccessful. For instance, Walmart started using AMRs in their stores for monitoring on-shelf availability in 2017. In 2020, they stopped their relationship with the robot's manufacturer, most possibly because of ROI issues.
AMRs are quite large, bulky, too expensive, and can be an impediment for customers, as AMRs move around the store by the routes defined by the algorithms. In-store drones are not in favor in shopping areas as well. The reasons are more or less the same.
But the list of problems with AMRs and indoor drones doesn’t stop there. In general, people respond to robots quite differently: some customers can be just scared, while others may attack the robots.

To sum up, robots may seem to be an adequate solution, as they have an in-built camera and can process the obtained data. Still, they move all around the store and capture a particular shelf only a limited number of times throughout the day. Even though robots are mobile, they lack merchandisers' ability to make changes on the shelves if required. Therefore, the advantages of cameras and merchandisers don’t apply to AMRs and in-store drones. As we have already mentioned, robots are quite expensive, so the capital expenditures on them are relatively high, as well as operational costs, as they require constant maintenance: from time to time they need to be recharged, and they can fail or be vandalized, etc.

Advantages:

• no need to use multiple cameras because robots move and monitor all shelves

Disadvantages:

• combines all the disadvantages of using humans and video cameras in one
• expensive maintenance
• interfere with customers

Conclusion

There are various approaches to monitoring and controlling goods on store shelves. They each have advantages and disadvantages, as we mentioned above.

Here at Inspector Cloud, we believe that mobile apps and high-res specialized cameras complement one another. That's why we developed a solution combining both methods and helping brands make informed decisions. Our solution can monitor shelves with the necessary frequency; it is not just passive monitoring but an excellent approach to changing situations with merchandisers using a mobile app.

Our customers want to have accurate and robust control and benefit from the advantages of both approaches. They use cameras to cover certain parts of the stores to see the most important shelves and placements, and use human resources (merchandisers) to monitor all stores at a frequency they can afford.

The great part about it is that all data is collected by the advanced Inspector Cloud solution. With this complete data, critical strategic decisions can be made afterward.