Hyperspectral Imaging

This whitepaper is an encapsulation of the research conducted by Cyient in introducing a system designed for assessing fruit ripeness utilizing a hyperspectral camera paired with a 2D Convolutional Neural Network (CNN) architecture. The research project is a part of the Action Learning Project (ALP) within the Technology Leadership Program (TLP). The development of this architecture took place on Google Colab, a cloud-based platform offering GPU access to boost the performance of deep learning models like CNNs.

Evaluation of the architecture involved utilizing a dataset containing ripening avocados, kiwis, and various other fruits, subjecting them to testing with diverse sets of features. The architecture's effectiveness was further improved through the integration of an optimal HyperSpectral Image (HIS) visualizer and the incorporation of additional features, such as those provided by the Gray-Level Co-occurrence Matrix (GLCM). This architecture demonstrated robust performance on the Colab platform, producing results consistent with findings from similar studies conducted previously.

Introduction

HyperSpectral Imaging or HIS, is a cutting-edge non-destructive technique that captures and analyzes material properties across a broad electromagnetic spectrum. Unlike traditional imaging methods, it acquires data in multiple contiguous bands, unveiling a wealth of previously hidden information.

This technology extends beyond its niche origins, finding applications in diverse fields, from agriculture and food quality inspection to environmental monitoring and remote sensing. HSI excels in fruit ripeness measurement, offering precise insights into attributes like sugar content, firmness, and overall ripeness, benefiting agriculture and food industries.

As part of the Action Learning Project (ALP) within the Technology Leadership Program (TLP), an internal initiative led by CYIENT to its associates, a comprehensive analysis of fruit skin characteristics, ranging from unripe to overripe stages has been made. This analysis was made possible through the application of HSI, coupled with the adaptation and enhancement of the HyperSpectral Convolutional Neural Networks (HS-CNNs) presented in [1], developed in Pytorch Lightning.

With a portfolio of various fruit images captured using three specialized hyperspectral cameras1 , it was possible to analyze their skin characteristics and test different AI algorithms performance under different types of fruits. All the models were first prepared, tested and underwent troubleshooting using Visual Studio Code before being migrated into Google Colab2 for the very first time. This allowed for heavy processing using NVIDIA V100 and A100 Tensor Core GPU’s.

The research and development have not just accomplished the adaptation of the AI Model to Google Colab, running the latest state-of-theart GPUs, but has also been able to successfully expand the testing capabilities introducing an automatic optimal Hyperspectral Image selector and visualizer or the inclusion of other features such as the ones provided by GLCM (Gray-Level Co-occurrence Matrix).

Data Set

To validate the architecture entailed utilizing a dataset containing ripening avocados, kiwis, and various other fruits, subjecting them to testing with diverse sets of features. The dataset was grouped in two groups, one containing Avocado and Kiwi and other with the remaining fruits which included Kaki, Mango, and Papaya.

The main data set contained 1038 recordings of Avocados and 1522 recordings of Kiwis. It covered the ripening process from unripe to overripe. Two measurement series covering a total of 28 days in the years 2019 and 2020 were collected. Two cameras were used in this experiment, INNO-SPEC Redeye and Specim FX 10.

The Specim FX 10 has 224 channels, and a spectral range from 400 to 1000 nm. This range holds the VIS (VISible spectrum) range with the addition of the lower NIR (Near InfraRed) range. The INNO-SPEC Redeye 1.7 records 252 channels. Their spectral range ran from 950 to 1700 nm.

According to the camera specification and fruits samples, each dataset group was organized into three separate groups: Visible Spectrum (VIS), Visible Spectrum w/ Color (VIS-COR) and Near-InfraRed (NIR). The following table maps each fruit to the camera and spectrum range being used.

Dataset	Camera	Fruit Samples
VIS	SPECIM_FX10	Avocado, Kaki, Kiwi, Mango, and Papaya
VIS_COR	CORNERING_HSI	Avocado, Kaki, Mango, and Papaya
NIR	INNOSPEC_REDEYE	Avocado, Kiwi

Table 1 - Dataset mapping to each camera and fruit sampled

At a later stage a third camera was introduced, Cornering HIS, which was also used just for reference to measure Avocado, Kaki, Mango, and Papaya. All of these are represented in the following Figure-1 as a reference.

Figure 1 - Hyperspectral cameras used to collect the used dataset

Model Overview

Conventional CNNs employ 2D convolutions, which operate across spatial dimensions but lack the capability to handle spectral information. In contrast, 3D convolutions can capture both spectral and spatial information simultaneously, albeit with a heightened computational complexity. Another approach to concurrently harness spatial and spectral information involves applying 2D convolutions on transformed data.

Our HS CNN Model is a small 2D neural network, which was specialized for the application of ripening fruits. The input comprises a hyperspectral recording of a fruit, characterized by two spatial dimensions and a channel dimension. Extracting feature maps from this input involves three convolutional layers, where convolutions are divided into smaller separable convolutions to optimize parameter count. Average-pooling layers, due to their empirically superior performance, are used according to the experiments in [1].

Additionally, batch normalization is implemented to expedite the training process.

The final classification occurs in the convolutional neural networks (CNN) head, consisting of a global average pooling layer and a fully connected layer. The global average pooling layer significantly reduces parameters, leading to more stable predictions compared to a fully connected head of similar size. In this instance, the network classifies three distinct categories visible in the output of the final layer.

This architecture is specifically designed for hyperspectral recordings with around 200 channels of wavelengths, when new cameras with different number of channels are introduced, adjustments to the hidden layers need to take place.

Figure 2 summarizes the architecture of the Convolutional Neural Network provided in Measuring the Ripeness of Fruit with HSI and Deep Learning [1][2].

Figure 2 - Architecture of the Convolutional Neural Network has provided in Meas- uring the Ripeness of Fruit with HIS and Deep Learning [1][2]

Model Preparation & Training

Using a normal desktop computer with Visual Studio Code installed we prepared the necessary environment to setup the HS-CNN network, including the Weights & Biases (W&B) platform for logging, visualizing, and analyzing machine learning experiment. This initial setup was used to make all the necessary changes to the existing models.

Although Google Colab is more powerful and stable, for testing and troubleshooting it doesn’t offer such advanced debugging possibilities as any traditional software.

Later, all the code was migrated to Google Colab, a cloud based Jupyter notebook service provided by Google primarily designed for data analysis, machine learning, and collaborative research.

Figure 3 - Development & Testing High Level Architecture

Using Avocado and Kiwi hyperspectral Images samples we run several model ranging from classical ML (Machine Learning) such as the Support Vector Machine (SVM) algorithms used in early classification of hyperspectral images to deep learning models such as our HS CNN. In total seven models were tested: SVM [5], CNN, ResNet-18, ResNet-18-SE (Squeeze-and-Excitation), DeepHS-Net, DeepHS-Net-SE and SpectralNet [6].

For training purposes, several augmentation technics were used such as flip, rotate, noise, cut, crop, intensity and color jitter as well has different learning rates, batch sizes and maximum number of epoch although an early stopping method was already in place based on the validation loss to prevent over-fitting. By default, all training started with batches of 32, and a learning rate of 0.001.

Obtained Results

For Specim FX 10 hyperspectral samples looking at Avocado, and Kiwi, and using our HS-CNN, the results were heavily satisfactory with values achieving 93,30% The results shown in Figure-4 were obtained using the following settings for our HS-CNN model which we define has “Modified” model. for Ripeness for Avocado and 66,67% for Kiwi. The INNO-SPEC Redeye didn’t surpass the 60% mark on both Avocado and Kiwi.

Slide1-Jan-08-2024-10-49-48-7111-AM — Figure 4 – Comparison Between UNNOSPEC and SPECIM cameras

The results shown in Figure-4 were obtained using the following settings for our HS-CNN model which we define has “Modified” model.

Augmentation Features	Base Model	Modified Model
Color Jitter	True	True
Random Crop	True	True
Random Cut	True	True
Random Flip	True	True
Random Intensity Scale	False	True
Random Noise	False	True
Random Rotate	True	True

Table 2 - HS-CNN Settings - Augmentation Features

Other changes include the use of a batch size of 64, a learning rate of 0,001 and a maximum epoch of 100.

We then proceeded to benchmark all seven models using Avocado as a reference to understand how they stand with the collected hyperspectral images. In this comparison, illustrated in Figure-5, the CNN model overtakes any other model when tested against Avocado samples.

Figure 5 - Benchmarking Between Deep Learning Networks

Additional Enhancements

The performance of CNN will only be as good had the worst element in the chain. In this case, data set shall not be discarded, and some tweaking had to be performed to gather the best samples. Handling large amount of data in particular when it comes to Hyperspectral Imaging can be very demanding from a resource and time perspective.

Using spectral library from python, we developed an algorithm that goes through all bands of the spectrum for all images to try identifying and saving the less noisy. The results are a visualization algorithm that allows us to save the best possible samples while also being able to display them for debugging purposes, something that was found very challenging while using Google Colab. The Figure-6 shows an example of such output.

Figure 6 - Hyperspectral Image Visualization (Band 9 - Bad, Band 69 - Average, Band 162 - Good)

Another set of tests were conducted using Gray-Level Co-occurrence Matrix (GLCM). CNNs are specifically designed to work with raw image data, learning features directly from the spatial and spectral structures within the image. GLCM, on the other hand, generates statistical features that summarize textures. Integrating these two disparate types of data may require a thoughtful and complex design.

Hyperspectral images often have high dimensionality, and combining these with the additional features from GLCM might lead to an increased complexity in the input space. With our CNN tailored for certain hyperspectral image cameras, adaptation is needed so that a convolutional concatenation is possible and the GLCM feature can be considered. In the interest of having better than current results, a series of functions were created so that this concatenation was possible and could contribute as another input to the AI models.

With GLCM, as depicted in Figure-7, the additional 25X channels from each Hyperspectral Image are enriched with 4 additional features: Contrast, Energy, Dissimilarity and Correlation. These then create a tensor which is then concatenated with the Hyperspectral tensor for a unique set of features.

We successfully integrated Near InfraRed and VISual spectrums image samples and successfully combined these within the HS-CNN model. The success of this merge, although not fully explored, was more positive than expected with the model achieving in some cases very close results to the original one has displayed in Figure-8 which shows a comparison between our base HS-CNN model and the modified one with GLCM for the Specim FX10 Hyperspectral Camera.

Figure 8 - HS-CNN Model vs GLCM Enabled Model

SAR-based remote sensing for crop monitoring

SAR uses electromagnetic properties such as amplitude, phase, relative amplitude, and relative phase of radio waves reflected from the crops on which it is transmitting the signal. These EM properties can be mapped to a color map to identify the type, age, and health of the crops. The airplane carrying the SAR moves in a straight line at any point of time. Hence, consecutive time of transmission/reception of Figure 3. The SAR[1] principle of operation RF pulses translates into different footprints of the antenna beam on the ground. A coherent (synchronous) combination of the received signals allows the construction of a virtual antenna aperture that is much longer than the physical antenna length, giving rise to the term “synthetic aperture” and gives the radar the property of being an imaging radar. Figure 3 represents the SAR principle of operation.

Figure 3. The SAR[1] principle of operation

Different types of crops respond to different RF spectrums. For example, tall and high biomass crops like sugarcane, maize, jute, and millets respond better to a lower microwave spectrum (1-2 GHz) whereas rice and pulses (low to medium biomass crops) are responsive to a slightly higher spectrum (4-8 GHz).

Figure 4. SAR imagery for different crop types ^[2]

Further, crops at different ages reflect RF signals differently which gives an indication of crop ages (Figure 5).

Figure 5. SAR imagery in different life stages of a crop^[3]

GPR-based soil study

To complement SAR usage in modern agriculture, GPR, being in closer proximity to ground, is used to study soil contents in detail. GPR sends a radio frequency signal toward the ground and part of it penetrates the ground. When the signal experiences the differences in the media, part of the penetrated signal is reflected back toward the radar and carries the relative characteristics of the media (see Figure 4(a)) . Electromagnetic properties of the signal are then mapped to an image representation with respect to survey coordinates (see Figure 4(b)). Usually, GPRs are pulled over the ground to collect data. With advancement of drones, GPRs are now being mounted on drones to provide faster mapping of large farming lands.

(a)

(b) GPR being used for soil survey [4], 2

(b)

Figure 6. (a) Concept of GPR (b) GPR being used for soil survey [4], (c) Drone-borne GPR [5

(c)

Figure 6. (a) Concept of GPR^[4] (b) GPR being used for soil survey ^[4], (c) Drone-borne GPR ^[5]

Using the EM properties of reflected signals and corresponding mapping to images, different layers of soil and their spatial distribution can be estimated with centimeter accuracy. These characteristics may be due to the lithology of the soil or due to presence of moisture, pollutants, agrochemicals. An example is shown in the figure below.

Figure 7. GPR imagery of soil: (a) Initial condition, (b) With movement of agrochemical agents ^[6]

Microwave heating

Studies have indicated the effectiveness of microwaves in drying material with water content and this property is useful in drying crops faster as compared to the conventional convection method. The role of microwave heating in pest control dates back nearly 70 years where lethal exposure of insects to a 12 MHz electric field was demonstrated. Recent experiments show that insects infesting dry products are quickly affected by microwave energy as the water content in their body gets heated. An application of microwave with controlled transmitting power helps in germination of seeds. Studies have shown that the use of microwaves during drying can be safe in seeds of soybeans, corn, and beans, among others. Many studies recognize that drying using microwaves has great potential as it reduces the drying time considerably. Other applications of microwave heating include microwave-assisted bio-fuel extraction and microwave-assisted pyrolysis (to convert biomass to an intermediate liquid product that can be refined to drop in hydrocarbon biofuels, oxygenated fuel additives, and petrochemical replacements).

RF communications in agriculture

Majority of the RF sensors described above are mounted either on airborne platforms or on a ground vehicle. Drones are the latest, most popular, and inexpensive airborne platform as compared to others. Drones are required to be controlled remotely using wireless technologies. A robust and long- distance RF communications system needs to be established for controlling and operating the drones as desired.

Figure 8. Remote-controlled drone for agriculture

With the advancement of technology, agricultural vehicles are adopting autonomy in their operations. The key elements of success for this are RF communications equipment and network infrastructure for a connected vehicle.

Figure 9. Agriculture vehicle with autonomy

5G technology in agriculture

The advent of 5G technology will revolutionize global farming landscapes and will open up multiple ways to establish and grow precision farming. The figure below shows that every element in modern agriculture once connected to a high speed and high throughput 5G cellular network, works in tandem with the other to optimize resources and maximize yield. The imagery generated from SAR and GPR demand throughput for transferring them to a distant and central location/data cloud. Similarly, to control farming equipment remotely, a low latency communications network in inevitable.

Figure 10. Uses of 5G technology in agriculture

Future of Hyperautomation

Hyperautomation will continue to evolve and redefine industries. Here are a few trends that could shape its future:

Hyperautomation as-a-service

Cloud-based hyperautomation platforms will become more accessible, allowing organizations of all sizes to leverage automation as a service. This democratization of technology will drive innovation across sectors.

Human-automation collaboration

Rather than replacing humans entirely, hyperautomation will focus on enhancing human capabilities.

Industry-specific solutions

Hyperautomation will be tailored to meet the specific needs of different industries. We can expect specialized solutions in sectors like healthcare, manufacturing, telecom, energy, and utilities addressing industry- specific challenges and requirements.

Enhanced cognitive capabilities

Advances in AI, ML, and Gen AI will lead to even more sophisticated cognitive capabilities, enabling systems to handle complex decision-making and problem- solving tasks.

IoT integration

IoT will become more tightly integrated with hyperautomation. Sensors and data from connected devices will be used to optimize and automate processes in real time.

Cross-industry collaboration

Industries will increasingly collaborate and share best practices for hyperautomation implementation. This cross-pollination of ideas will accelerate innovation and adoption.

Regulatory frameworks

Governments and regulatory bodies will establish frameworks to address the ethical and legal implications of hyperautomation, ensuring a responsible and fair use of the technology.

In the future, we can expect to see even more changes in the way hyperautomation is used and implemented. Advances in IoT, blockchain, and quantum computing will open opportunities for hyperautomation to be applied in new domains and enable it to automate highly complex tasks and processes.

Conclusion

This study presented a comprehensive system for assessing fruit ripeness using hyperspectral imaging and a 2D Convolutional Neural Network (CNN) architecture. The development of this architecture, carried out on Google Colab with GPU support, involved the integration of an optimal HyperSpectral Image Selector and additional features such as the Gray-Level Co-occurrence Matrix (GLCM). The evaluation, conducted on a dataset comprising various fruits, demonstrated the effectiveness of the proposed approach.

The article highlights the significance of HyperSpectral Imaging (HSI) in capturing material properties across a broad electromagnetic spectrum, particularly in fruit ripeness measurement. The HyperSpectral Convolutional Neural Network (HS-CNN) model, designed for hyperspectral recordings with around 200 channels of wavelengths, showed robust performance across different fruit types. The research not only adapted the AI model to Google Colab but also introduced enhancements like the optimal HyperSpectral Image selector and GLCM integration.

The dataset, collected from specialized hyperspectral cameras, covered the ripening process of avocados, kiwis, and other fruits. The HS-CNN model, tested alongside other machine learning models, demonstrated superior performance, achieving satisfactory ripeness values for avocados and kiwis.

The article further discusses the model's settings, including augmentation features, and highlights the importance of data preprocessing, especially in handling large amounts of hyperspectral imaging data. The integration of GLCM, generating statistical features to summarize textures, was explored, showing promising results in combination with hyperspectral data.

In summary, the presented system offers a sophisticated solution for fruit ripeness assessment, leveraging hyperspectral imaging and a tailored CNN architecture using Google’s Colab platform.

Future Work

Hyperspectral imaging emerges as a transformative technology with versatile applications across industries. Hyperspectral imaging has been applied to a wide range of scientific investigations at various scales, such as remote sensing, pigment determination in biology, medicine, coastal ocean imaging, water analysis, agriculture, cultural heritage and archaeology, and more.[7]

Beyond traditional applications, such as agriculture and environmental monitoring, we envision leveraging this technology for Base Station Tower Inspection in wireless communication. By employing hyperspectral imaging, we can enhance the efficiency and accuracy of tower inspections, ensuring optimal performance and reducing downtime.

Moreover, considering the rising concerns about skin cancer, there is an opportunity to integrate hyperspectral imaging for Skin Cancer Lesions Detection. The technology's ability to analyze subtle spectral variations could contribute to early and accurate detection, aligning with the broader focus on health and well-being.

In the field of mineral exploration, hyperspectral imaging can play a pivotal role in identifying Mineral Spectral Signatures. This not only streamlines mining operations but also aligns with the increasing emphasis on sustainable and responsible resource management.

To further augment our capabilities, the integration of LiDAR technologies with hyperspectral imaging opens avenues for Object Detection and Tracking, Object Classification, and Environmental Monitoring. The combination of these technologies offer a comprehensive solution that aligns with the evolving needs of managed services.

By strategically incorporating hyperspectral imaging, we position ourselves at the forefront of innovation, providing tailored solutions that not only meet current requirements but also anticipate and address future challenges in diverse industries.

About the Authors

This research paper was a collaborative effort by a team of experts in hyperspectral imaging, as part of Action Learning Project.

Srinivas Rao Kudavelly: srinivasrao.kudavelly@cyient.com

Marco Carvalho: marco.carvalho@cyient.com

Guilherme Pereira: guilherme.pereira@cyient.com

Flavio Bras: flavio.bras@cyient.com

Ashok Sethuraman: ashok.sethuraman@cyient.com

Sateesh Bheemarasetty: sateesh.bheemarasetty@cyient.com

Murty Darbha: murty.darbha@cyient.com

For more information on this Whitepaper, please feel free to get in touch with Srinivas Rao Kudavelly at srinivasrao.kudavelly@cyient.com

shutterstock_1976153234-1

References

Varga L., Makowski, J., Zell A., “Measuring the Ripeness of Fruit with Hyperspectral Imaging and Deep Learning”, IJCNN 2021, 2021

Arbeck, “Mono, Multi and Hyperspectral Cube and corresponding Spectral Signatures,” 2013. [Online]. Available: https://bit.ly/38lJTY0

Varga, Leon Amadeus and Messmer, Martin and Benbarka, Nuri and Zell, Andreas, “Wavelength-aware 2D Convolutions for Hyperspectral Imaging”, 2022

Hannah Frank and Leon Amadeus Varga and Andreas Zell, “Hyperspectral Benchmark: Bridging the Gap between HSI Applications through Comprehensive Dataset and Pretraining”, 2023, [Online] Available: https://arxiv.org/abs/2309.11122

N. Cristianini and J. Shawe-Taylor, “An Introduction to Support Vector Machines and Other Kernel-based Learning Methods”. Cambridge University Press, 3 2000

Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri Yuval Kluger, “Spectralnet: Spectral Clustering Using Deep Neural Networks”, ICLR, 2018

Cyient Blog : Hyperspectral imaging unlocking new realms of knowledge

About Cyient

Cyient (Estd: 1991, NSE: CYIENT) is a leading global engineering and technology solutions company. We are a Design, Build, and Maintain partner for leading organizations worldwide. We leverage digital technologies, advanced analytics capabilities, and our domain knowledge and technical expertise, to solve complex business problems.

We partner with customers to operate as part of their extended team in ways that best suit their organization’s culture and requirements. Our industry focus includes aerospace and defense, healthcare, telecommunications, rail transportation, semiconductor, geospatial, industrial, and energy. We are committed to designing tomorrow together with our stakeholders and being a culturally inclusive, socially responsible, and environmentally sustainable organization.

For more information, please visit www.cyient.com