Research progress of image acquisition platform for crop classification and recognition based on CNN

doi:10.13733/j.jcam.issn.2095‑5553.2024.08.025

Abstract

Abstract: Accurate crop classification and recognition based on machine vision is the premise of agricultural automation and intelligent operation. Convolution neural network (CNN) is one of the most widely used algorithms in crop image classification and recognition. The complexity of crop phenotypic characteristics and growth environment determines the diversity of crop image acquisition platforms. Through the analysis of crop classification and recognition research based on CNN at home and abroad from 2020 to 2022, the image acquisition platforms can be divided into two categories such as general platform and self‑built platform, among which the general platform hardware products are mature and easy to deploy, but equipment selection and environment construction should be done well. The self‑built platform is divided into fixed and mobile ones, which can obtain experimental data efficiently, but the hardware integration is more complicated. The advantages and disadvantages of various platforms and their applicable scope are compared and analyzed in detail. The future trends of crop image acquisition platforms will include high‑throughput, high‑efficiency, automated universal image acquisition devices, multi‑modal data acquisition and fusion applications integrating a variety of sensors, intelligent cameras with built‑in computing processing, etc. The more refined image acquisition platforms will effectively support in‑depth research on crop phenotypes.

Key words: plant phenotype, machine learning, convolutional neural network, image acquisition, crop classification and identification

摘要： 基于机器视觉的作物精准分类识别是农业自动化、智能化作业的前提。在作物图像分类识别任务中，卷积神经网络（CNN）是当前应用最广泛的算法之一。作物表型特征及生长环境的复杂性，决定作物图像获取平台的多样性。通过分析2020—2022年国内外基于CNN的作物分类识别研究，图像获取平台可划分为通用平台和自建平台两大类：通用平台硬件产品成熟、部署方便，但要做好设备选型和环境搭建；自建平台分为固定式和移动式，能高效获取试验数据，但硬件集成较为复杂。详细对比分析各类平台的优缺点及适用范围。作物图像获取平台的未来趋势包括：高通量、高效率、自动化的通用图像获取装置，集成多种传感器的多模态数据采集与融合应用，自带运算处理的智能摄像头等，更精细化的图像获取平台将有效支撑作物表型的深入研究。

关键词: 作物表型, 机器学习, 卷积神经网络, 图像获取, 作物分类识别

CLC Number:

v. Research progress of image acquisition platform for crop classification and recognition based on CNN [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(8): 170-179.

张倩, 王明, 于峰, 陶震宇, 张辉, 李刚. 基于CNN的作物分类识别图像获取平台研究进展[J]. 中国农机化学报, 2024, 45(8): 170-179.

References

[ 1 ] Lecun Y， Bottou L， Bengio Y， et al. Gradient-based learning applied to document recognition [J]. Proceedings of IEEE， 1998， 86(11): 2278-2324.
[ 2 ] Kubera E， Kubik Komar A， Piotrowska Weryszko K， et al. Deep learning methods for improving pollen monitoring [J]. SENSORS， 2021， 21(10)， 3526.
[ 3 ] Sabanci K， Aslan M F， Ropelewska E， et al. A convolutional neural network‑based comparative study for pepper seed classification: Analysis of selected deep features with support vector machine [J]. Journal of Food Process Engineering， 2022， 45(6)， 13955.
[ 4 ] 孟志超，贺磊盈，杜小强，等. 基于Enhanced VGG16的油茶品种分类[J]. 农业工程学报， 2022， 38(10): 176-181.
Meng Zhichao， He Leiying， Du Xiaoqiang， et al. Classification of camellia oleifera based on enhanced VGG16 network [J]. Transactions of the Chinese Society of Agricultural Engineering， 2022， 38(10): 176-181.
[ 5 ] 宋怀波，王亚男，王云飞，等. 基于YOLOv5s的自然场景油茶果识别方法[J]. 农业机械学报， 2022， 53(7): 234-242.
Song Huaibo， Wang Ya nan， Wang Yunfei， et al. Camellia oleifera fruit dectection in natural scene based on YOLOv5s [J]. Transactions of the Chinese Society for Agricultural Machinery， 2022， 53(7): 234-242.
[ 6 ] 尚钰莹，张倩如，宋怀波. 基于YOLOv5s的深度学习在自然场景苹果花朵检测中的应用[J]. 农业工程学报， 2022， 38(9): 222-229.
Shang Yuying， Zhang Qianru， Song Huaibo. Application of deep learning based on YOLOv5s to apple flower detection in natural scenes [J]. Transactions of the Chinese Society of Agricultural Engineering， 2022， 38(9): 222-229.
[ 7 ] Selvam L， Kavitha P. Classification of ladies finger plant leaf using deep learning [J]. Journal of Ambient Intelligence and Humanized Computing， 2020.
[ 8 ] Zhou Zhongxian， Song Zhenzhen， Fu Longsheng， et al. Real‑time kiwifruit detection in orchard using deep learning on Android (TM) smartphones for yield estimation [J]. Computers and Electronics in Agriculture， 2020，179: 105856.
[ 9 ] Fu Longsheng， Feng Yali， Wu Jingzhu， et al. Fast and accurate detection of kiwifruit in orchard using improved YOLOv3-tiny model [J]. Precision Agriculture， 2021， 22(3): 754-776.
[10] Lu Wei， Du Rongting， Niu Pengshuai， et al. Soybean yield preharvest prediction based on bean pods and leaves image recognition using deep learning neural network combined with GRNN[J]. Frontiers in Plant Science， 2022， 12: 791256.
[11] Seo D， Cho B H， Kim K C. Development of monitoring robot system for tomato fruits in hydroponic greenhouses [J]. Agronomy‑basel， 2021， 11(11): 2211.
[12] Taner A， Oztekin Y B， Duran H. Performance analysis of deep learning CNN models for variety classification in hazelnut [J]. Sustainability， 2021， 13(12): 6527.
[13] Chen Jiqing， Wang Zhikui， Wu Jiahua， et al. An improved Yolov3 based on dual path network for cherry tomatoes detection [J]. Journal of Food Process Engineering， 2021， 44(10)： 13803.
[14] Liu Yixue， Su Jinya， Shen Lei， et al. Development of a mobile application for identification of grapevine cultivars via deep learning [J]. International Journal of Agricultural and Biological Engineering， 2021， 14(5): 172-179.
[15] Shi Xiao， Chai Xiujuan， Yang Chenxue， et al. Vision‑based apple quality grading with multi‑view spatial network [J]. Computers and Electronics in Agriculture， 2022， 195: 106793.
[16] Nasiri A， Taheri‑Garavand A， Fanourakis D， et al. Automated grapevine cultivar identification via leaf imaging and deep convolutional neural networks[J]. A Proof‑of‑Concept Study Employing Primary Iranian Varieties. Plants‑Basel， 2021， 10(8): 162.
[17] Han Jingye， Shi Liangsheng， Yang Qi， et al. Real‑time detection of rice phenology through convolutional neural network using handheld camera images [J]. Precision Agriculture， 2021， 22(1): 154-178.
[18] Wei Pengliang， Jiang Ting， Peng Huaiyue， et al. Coffee flower identification using binarization algorithm based on convolutional neural network for digital images [J]. Plant Phenomics， 2020: 6323965.
[19] 赵春江，文朝武，林森，等. 基于级联卷积神经网络的番茄花期识别检测方法[J]. 农业工程学报， 2020， 36(24): 143-152.
Zhao Chunjiang， Wen Chaowu， Lin Sen， et al. Tomato florescence recognition and detection method based on cascaded neural network [J]. Transactions of the Chinese Society of Agricultural Engineering， 2020， 36(24): 143-152.
[20] 成伟，张文爱，冯青春，等. 基于改进YOLOv3的温室番茄果实识别估产方法 [J]. 中国农机化学报， 2021， 42(4): 176-182.
Cheng Wei， Zhang Wenai， Feng Qingchun， et al. Method of greenhouse tomato fruit identification and yield estimation based on improved YOLOv3 [J]. Journal of Chinese Agricultural Mechanization， 2021， 42(4): 176-182.
[21] Feng Quanlong， Yang Jianyu， Liu Yiming， et al. Multi‑temporal unmanned aerial vehicle remote sensing for vegetable mapping using an attention:Based recurrent convolutional neural network [J]. Remote Sensing， 2020， 12(10)， 1668.
[22] Sivakumar A N V， Li J T， Scott S， et al. Comparison of object detection and patch‑based classification deep learning models on Mid‑to Late‑Season weed detection in UAV imagery [J]. Remote Sensing， 2020， 12(13): 2136.
[23] Bishwa B S， Hu Chengsong， Muthukumar V B. Evaluating cross‑applicability of weed detection models across different crops in similar production environments [J]. Frontiers in Plant Science， 2022， 13: 837726.
[24] Safonova A， Guirado E， Maglinets Y， et al. Olive tree biovolume from uav multi‑resolution image segmentation with mask R-CNN [J]. Sensors， 2021， 21(5): 1617.
[25] Trevisan R， Perez O， Schmitz N， et al. High‑throughput phenotyping of soybean maturity using time series UAV imagery and convolutional neural networks [J]. Remote Sensing， 2020， 12(21): 3617.
[26] Psiroukis V， Espejo‑Garcia B， Chitos A， et al. Assessment of different object detectors for the maturity level classification of broccoli crops using UAV imagery [J]. Remote Sensing， 2022， 14(3): 731.
[27] Moazzam S I， Khan U S， Qureshi W S， et al. A patch‑image based classification approach for detection of weeds in sugar beet crop [J]. IEEE Access， 2021， 9: 121698-121715.
[28] Oliveira G S， Junior J M， Polidoro C， et al. Convolutional neural networks to estimate dry matter yield in a guineagrass breeding program using UAV remote sensing [J]. Sensors， 2021， 21(12): 3971.
[29] Briechle S， Krzystek P， Vosselman， G. SilviNet A dual‑CNN approach for combined classification of tree species and standing dead trees from remote sensing data [J]. International Journal of Applied Earth Observation and Geoinformation， 2021， 98: 102292.
[30] Wang Lingyue， Liu Meiling， Liu Xiangnan， et al. Pretrained convolutional neural network for classifying rice‑cropping systems based on spatial and spectral trajectories of Sentinel-2 time series [J]. Journal of Applied Remote Sensing， 2020， 14(1): 14506.
[31] Li Zhengtao， Chen Guokun， Zhang Tianxu. A CNN‑transformer hybrid approach for crop classification using multitemporal multisensor images [J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing， 2020， 13: 847-858.
[32] Wang Shouyi， Xu Zhigang， Zhang Chengming， et al. Improved winter wheat spatial distribution extraction using a convolutional neural network and partly connected conditional random field [J]. Remote Sensing， 2020， 12(5): 821.
[33] Yang Shuting， Gu Lingjia， Li Xiaofeng， et al. Crop classification method based on optimal feature selection and hybrid CNN-RF networks for multi‑temporal remote sensing imagery [J]. Remote Sensing， 2020， 12(19)： 3119.
[34] Ji Shunping， Zhang Zhili， Zhang Chi， et al. Learning discriminative spatiotemporal features for precise crop classification from multi‑temporal satellite images [J]. International Journal of Remote Sensing， 2020， 41(8): 3162-3174.
[35] Bhupendra， Kriz M， Ankur M， et al. Deep CNN‑based damage classification of milled rice grains using a high‑magnification image dataset [J]. Computers and Electronics in Agriculture， 2022， 195: 106811.
[36] Li Hao， Zhang Liu， Sun Heng， et al. Discrimination of unsound wheat kernels based on deep convolutional generative adversarial network and near‑infrared hyperspectral imaging technology [J]. Spectrochimica Acta Part A‑molecular and Biomolecular Spectroscopy， 2022， 268: 120722.
[37] Han Yifei， Liu Zhaojing， Kourosh K， et al. Quality estimation of nuts using deep learning classification of hyperspectral imagery [J]. Computers and Electronics in Agriculture， 2021， 180: 105868.
[38] Jin Baichuan， Zhang Chu， Jia Liangquan， et al. Identification of rice seed varieties based on near‑infrared hyperspectral imaging technology combined with deep learning [J]. ACS Omega， 2022.
[39] Singh T， Garg N M， Iyengar S R S. Nondestructive identification of barley seeds variety using near‑infrared hyperspectral imaging coupled with convolutional neural network [J]. Journal of Food Process Engineering， 2021， 44(10): 13821.
[40] Unlersen M F， Sonmez M E， Aslan M F， et al. CNN‑SVM hybrid model for varietal classification of wheat based on bulk samples [J]. European Food Research and Technology， 2022， 248(8): 2043-2052.
[41] Jahanbakhshi A， Momeny M， Mahmoudi M， et al. Classification of sour lemons based on apparent defects using stochastic pooling mechanism in deep convolutional neural networks [J]. Scientia Horticulturae， 2020， 263: 109133.
[42] Hsieh K W， Huang B Y， Hsiao K Z， et al. Fruit maturity and location identification of beef tomato using R‑CNN and binocular imaging technology [J]. Journal of Food Measurement and Characterization， 2021， 15(6): 5170-5180.
[43] 翟长远，付豪，郑康，等. 基于深度学习的大田甘蓝在线识别模型建立与试验[J]. 农业机械学报，2022， 53(4): 293-303.
Zhai Changyuan， Fu Hao， Zheng Kang， et al. Establishment and experimental verification of deep learning model for on‑line recognition of field cabbage [J]. Transactions of the Chinese Society for Agricultural Machinery， 2022， 53(4): 293-303.
[44] Wang Zhipeng， Jin Luoyi， Wang Shuai， et al. Apple stem/calyx real‑time recognition using YOLO‑v5 algorithm for fruit automatic loading system [J]. Postharvest Biology and Technology， 2022， 185: 111808.
[45] Xi R， Hou J， Lou W. Potato bud detection with improved Faster R-CNN[J]. Transactions of the ASABE， 2020， 63(3): 557-569.
[46] Xie Weijun， Wei Shuo， Zheng Zhaohui， et al. A CNN‑based lightweight ensemble model for detecting defective carrots [J]. Biosystems Engineering， 2021， 1208: 287-299.
[47] Jayakumari R， Nidamanuri R R， Ramiya A M. Object‑level classification of vegetable crops in 3D LiDAR point cloud using deep learning convolutional neural networks [J]. Precision Agriculture， 2021， 22(5): 1617-1633.
[48] Tian Li， Wang Chun， Li Hailiang， et al. Recognition method of corn and rice crop growth state based on computer image processing technology [J]. Journal of Food Quality， 2022， 2844757.
[49] Wang Shanshan， Zhang Wenyi， Wang Xingsong， et al. Recognition of rice seedling rows based on row vector grid classification [J]. Computers and Electronics in Agriculture， 2021， 190: 106454.
[50] Rahim U F， Utsumi T， Mineno H. Deep learning‑based accurate grapevine inflorescence and flower quantification in unstructured vineyard images acquired using a mobile sensing platform [J]. Computers and Electronics in Agriculture， 2022， 198.
[51] Blok P M， Evert F K， Tielen A P M， et al. The effect of data augmentation and network simplification on the image:Based detection of broccoli heads with Mask R-CNN [J]. Journal of Field Robotics， 2021， 38(1): 85-104.
[52] Vayssade J A， Jones G， Gee C， et al. Pixelwise instance segmentation of leaves in dense foliage [J]. Computers and Electronics in Agriculture， 2022， 195.
[53] Massah J， Vakilian K Asefpour， Shabanian M， et al. Design， development， and performance evaluation of a robot for yield estimation of kiwifruit [J]. Computers and Electronics in Agriculture， 2021， 185: 106132.
[54] 王辉，韩娜娜，吕程序，等. 基于Mask R-CNN的单株柑橘树冠识别与分割[J]. 农业机械学报， 2021， 52(5): 169-174.
Wang Hui， Han Nana， Lü Chengxu， et al. Recognition and segmentation of individual citrus tree crown based on mask R-CNN [J]. Transactions of the Chinese Society for Agricultural Machinery， 2021， 52(5): 169-174.
[55] Lin P， Lee W S， Chen Y M， et al. A deep‑level region‑based visual representation architecture for detecting strawberry flowers in an outdoor field [J]. Precision Agriculture， 2020， 21(2): 387-402.
[56] Fu Longsheng， Majeed Y， Zhang Xin， et al. Faster R-CNN‑based apple detection in dense‑foliage fruiting‑wall trees using RGB and depth features for robotic harvesting [J]. Biosystems Engineering， 2020， 197: 245-256.
[57] Su D， Kong H， Qiao Y L， et al. Data augmentation for deep learning based semantic segmentation and crop‑weed classification in agricultural robotics [J]. Computers and Electronics in Agriculture， 2021， 190: 106418.
[58] Kartal S， Choudhary S， Masner J， et al. Machine learning‑based plant detection algorithms to automate counting tasks using 3D canopy scans [J]. Sensors， 2021， 21(23).
[59] Potena C， Nardi D， Pretto A. Fast and accurate crop and weed identification with summarized train sets for precision agriculture [J]. Cham: Springer International Publishing， 2017： 105-121.
[60] Li Can， Lin Jiaquan， Li Boyang， et al. Partition harvesting of a column‑comb litchi harvester based on 3D clustering [J]. Computers and Electronics in Agricultur， 2022， 197: 106975.
[61] 刘芳，刘玉坤，林森，等. 基于改进型YOLO的复杂环境下番茄果实快速识别方法[J]. 农业机械学报， 2020， 51(6): 229-237.
Liu Fang， Liu Yukun， Lin Sen， et al. Fast Recognition method for tomatoes under complex environments based on improved YOLO [J]. Transactions of the Chinese Society for Agricultural Machinery， 2020， 51(6): 229-237.
[62] Gene‑Mola J， Sanz‑Cortiella R， Rosell‑Polo J R， et al. Fruit detection and 3D location using instance segmentation neural networks and structure‑from‑motion photogrammetry [J]. Computers and Electronics in Agriculture， 2020， 169：105165.

[1]	Chen Yuxin, , Liu Zhangxin, , Liu Xinyi, , Liu Tao, , Sun Chengming, , . Remote sensing classification and extraction of winter wheat in Yangzhou based on machine learning algorithm [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(8): 154-161.
[2]	v. Study on identification of pear leaf disease based on hyperspectral imaging technology [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(8): 162-169.
[3]	Tang Keji, Sun Wenlei, Yang Yang, Kong Delong. Trend analysis and early warning of engine service status of corn harvester [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(7): 160-165.
[4]	Huang Yiqi, Lu Linfei, Shen Hao, , Wang Fukuan, , Qiao Xi, . Research on invading insect recognition based on convolutional neural network [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(7): 222-227.
[5]	Guo Hongjie, Ma Rui, Wang Jia, Zhao Wei, Ma Dexin. Research on apple leaf disease image recognition based on convolutional neural network [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(5): 239-245.
[6]	Lin Yanxiang, Shen Yin, Li Guanglin. Research on wheat impurity identification based on improved InceptionV3 algorithm [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(4): 108-116.
[7]	Ma Zhiyan, , Li Hui, Yang Guangyou, . Research on key technologies of intelligent tea picking machine based on YOLOv3 algorithm [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(4): 199-204.
[8]	Zhang Huimeng, He Chao, Xu Jiawen, Luo Xin, Rong Jian, Liu Xueyuan. Detection algorithm of macadamia nut at havesting stage based on SCGYOLOv5n [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(4): 214-221.
[9]	Ji Yuanhao, Xu Jinpu, Yan Beibei, Xue Junlong. A method for detecting quality and defects in raw coffee beans based on improved ResNet50 model [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(4): 237-243.
[10]	Chen Luwei, , Zeng Jin, , Yuan Quanchun, , Pan Jian, , Yao Fengteng, , Lü Xiaolan, . Research progress on monitoring nitrogen content of fruit trees by UAV remote sensing [J]. Journal of Chinese Agricultural Mechanization, 2024, 45(2): 235-243.
[11]	Liang Wanjie, Cao Jing, Sun Chuanliang, Cao Hongxin, Zhang Wenyu, . Research and development of crop diseases intelligent recognition system based on deep learning [J]. Journal of Chinese Agricultural Mechanization, 2023, 44(9): 169-175.
[12]	Yang Bo, He Jinping, Zhang Lina. Identification and detection of rice leaf diseases by YOLOv5 neural network based on improved SPP-x [J]. Journal of Chinese Agricultural Mechanization, 2023, 44(9): 190-197.
[13]	Wang Jian, Liang Xingjian, Lei Gang. Image recognition of rice diseases and insect pests based on deep residual network and transfer learning [J]. Journal of Chinese Agricultural Mechanization, 2023, 44(9): 198-204.
[14]	Wang Chuntao, , , , Liang Weijian, Guo Qingwen, Zhong Hao, Gan Yu, Xiao Deqin, , . Review on computervisionbased detection of agricultural pests [J]. Journal of Chinese Agricultural Mechanization, 2023, 44(7): 207-213.
[15]	Li Ping, Ma Yukun, Li Yancui, Feng Jike, Zhao Mingfu. Study on wheat seed variety identification based on transfer learning [J]. Journal of Chinese Agricultural Mechanization, 2023, 44(7): 220-228.