With its most innovative vision system to date, Spookfish has deployed an automatic refrigerator inspection system for LG Electronics in their Greater Noida factory. This real time machine vision inspection system has been installed and is inspecting over 75 refrigerator models with an accuracy rate of 99.83%.
This project started with a seemingly simple requirement, “Detect any missing parts in a refrigerator assembly”. Although this is straightforward in smaller cases, the biggest challenge here was to accurately solve this problem across the entire range of LG fridges – comprising of over 75 models at a time, with new models introduced every month. Assembly of the refrigerators, carried out manually, is prone to errors in terms of missing trays, labels and some other parts. Machine vision inspection systems are usually well suited for environments with consistent positioning and lighting, but this system required live inspection of components on the assembly floor, with the refrigerators in motion on a conveyor.
Spookfish’s software intelligently learns examples of parts that make up the fridge for each and every fridge model and carries out exemplar-based verification for every sample. For every model to be inspected, the operator makes use of simple offline tools to select on a template, regions of interest around components whose presence is to be verified. The software then examines this information to learn the appearance of the various components for this model. This one-time labelling process then enables the software to inspect any refrigerator belonging to the learned model.
The live inspection process begins with reading the refrigerator model via a barcode scanner, which communicates with our software through serial communication. The model number, serial number and batch number for every refrigerator inspected is automatically interpreted through the scanned barcode. Based on the model, the software loads up the previously annotated template. Since the fridge is on a moving conveyor belt, the system has to complete the difficult task of negating any apparent perspective change in the refrigerator being inspected, with respect to the template. Using sophisticated algorithms, the software reliably registers images inspite of challenges such as:
- Poor and inconsistent lighting
- Perspective transformation, as the refrigerator can scale, rotate and translate with respect to the template, and the door may not open to the same angle in every inspection
- Reflections and highlights
The image below shows an example of how a template (example image of an annotated model) on the right is registered to the door of the refrigerator being inspected. The blue lines show similarity correspondences that have been computed between the two images using Spookfish’s bespoke image processing algorithms, which then aid in computing an accurate registration.
Once registration is complete, every annotated region of interest is individually examined and matched using state of the art algorithms to determine the presence/absence of each component within the refrigerator. The example shown below displays the output of the software suite, wherein the user receives a Pass or Fail as a signal on the desired communication port, based on the result of inspection. The visualisation clearly marks the components that have been examined, with a green box to confirm presence or a red box to confirm absence.
The entire inspection process for one refrigerator is completed within an average processing time of 500ms, and the system is currently in use on a daily basis to examine the correctness in assembly of several of LG’s refrigerators and is performing with an accuracy of 99.83%. Thanks to the inspection system, LG have since identified and fixed in their production line inconsistencies that had previously slipped through the cracks.
The refrigerator inspection system has also been deployed for outgoing quality checks, wherein randomly picked samples of various models are examined more closely, along with the labelling and packaging that they are eventually shipped in. LG have also rolled out the same outgoing quality check system for inspection of washing machines, air conditioners and microwave ovens.