消费级RGB-D相机在农业领域应用现状与发展趋势

doi:10.13733/j.jcam.issn.20955553.2022.04.029

摘要/Abstract

摘要： 三维重建技术在农业领域的应用加速农业信息化进程，消费级RGB-D相机是农业领域中物体三维信息采集的重要工具之一。这种相机最初应用在人机交互的游戏领域，因其较好的性能和低廉的价格逐渐应用在农业领域。为明确该类相机在农业领域的发展方向，首先对其工作原理进行介绍并分类；其次阐述其在农业领域的应用现状，包括作物表型参数提取、果实识别与定位、畜牧监测等；然后分析目前此类相机应用时存在的问题，主要包括室外应用易受强光干扰、分辨率不高、不适合远距离测量这几个方面；接着针对以上问题给出多传感器融合及算法优化两个改进方向；最后对其在农业领域的应用进行总结，从加强设施化农业中的应用、提高自动化程度、提升鲁棒性及应用于农业机器人等方面进行展望，以期其能更好地应用在农业生产领域。

关键词: 三维重建, 消费级RGB-D相机, 信息采集, 农牧监测

Abstract: The application of 3D reconstruction technology in the agricultural field accelerates the process of agricultural informatization. Consumer RGB-D camera is one of the important tools for 3D object information collection in the agricultural field. The consumer RGB-D camera was initially used in humancomputer interaction games and gradually applied in the agricultural due to its good performance and low price. In order to clarify the development direction of such camera, this paper firstly introduced and classified its working principle. Secondly, its application in agriculture was reviewed, including crop parameter extraction, fruit identification and location, animal husbandry monitoring, and so on. Then, the problems existing in the application of such cameras at present were analyzed, including the outdoor application was easy to be disturbed by strong light, the resolution was not high, and it was not suitable for remote measurement. Then, aiming at the above problems, two improvement directions of multisensor fusion and algorithm optimization were given. Finally, it summarized its application in the field of agriculture and looks forward to strengthening the application in facility agriculture, improving the degree of automation, improving robustness, and applying to agricultural robots in order to better apply it in the field of agricultural production.

Key words: threedimensional reconstruction, consumer RGB-D camera, information collection, agriculture and animal husbandry monitoring

中图分类号:

S126

彭孝东, 时磊, 何静, 漆海霞, 兰玉彬, . 消费级RGB-D相机在农业领域应用现状与发展趋势[J]. 中国农机化学报, 2022, 43(4): 206-215.

Peng Xiaodong, Shi Lei, He Jing, Qi Haixia, Lan Yubin. . Application status and development trend of consumer RGB-D camera in the agricultural field[J]. Journal of Chinese Agricultural Mechanization, 2022, 43(4): 206-215.

参考文献

［1］ Xiaojian J, Xiaojun L, Yongchao T, et al. Design and implementation of remote sensing imagebased crop growth monitoring system ［J］. Transactions of the Chinese Society of Agricultural Engineering, 2010, 26（3）: 156-162.

［2］ Su B, Noguchi N. Discrimination of land use patterns in remote sensing image data using minimum distance algorithm and watershed algorithm ［J］. Engineering in Agriculture Environment & Food, 2013, 6（2）: 48-53.

［3］ Meyer P. Segmentation and symbolic description for a classification of agricultural areas with multispectral scanner data ［J］. IEEE transactions on geoscience and remote sensing, 1992, 30（4）: 673-679.

［4］郑志雄, 齐龙, 马旭, 等. 基于高光谱成像技术的水稻叶瘟病病害程度分级方法［J］. 农业工程学报, 2013, 29（19）: 138-144.

Zheng Zhixiong, Qi long, Ma Xu, et al. Grading method of rice leaf blast using hyperspectral imaging technology ［J］. Transactions of the Chinese Society of Agricultural Engineering, 2013, 29（19）: 138-144

［5］章林忠, 丁玲玲, 蔡雪珍, 等. 基于近红外高光谱图像技术的栗果品质无损检测［J］. 安徽农业大学学报, 2019, 46（1）: 160-166.

Zhang Linzhong, Ding Lingling, Cai Xuezhen, et al. Nondestructive detection of Chinese chestnut （Castanea mollissima） nut qualities based on nearinfrared hyperspectral imaging techniques ［J］. Journal of Anhui Agricultural University, 2019, 46（1）: 160-166

［6］伍恒, 李明周, 张航, 等. 基于高光谱技术的哈密瓜果实物候期判别模型［J］. 江苏农业科学, 2019, 47（22）: 258-264.

［7］方伟, 冯慧, 杨万能, 等. 表型检测中用于小麦株型研究的快速三维重建方法［J］. 中国农业科技导报, 2016, 18（2）: 95-101.

Fang Wei, Feng Hui, Yang Wanneng, et al. A fast 3D reconstruction for wheat plant architecture studies in phenotyping ［J］. Journal of Agricultural Science and Technology, 2016, 18（2）: 95-101

［8］袁红照, 李勇, 何方. 三维点云数据获取技术［J］. 安阳师范学院学报, 2009（2）: 75-79.

Yuan Hongzhao, Li Yong, He Fang. Three dimensions point cloud data acquisition techniques ［J］. Journal of Anyang Normal University, 2009（2）: 75-79

［9］陈晨, 刘涛, 武威, 等. 三维传感器Kinect在农业领域的应用与发展［J］. 江苏农业科学, 2018, 46（8）: 11-14.

［10］ Condotta I, BrownBrandl T M, Pitla S K, et al. Evaluation of lowcost depth cameras for agricultural applications［J］. Computers and Electronics in Agriculture, 2020, 173: 105394.

［11］王旒军, 陈家斌, 余欢, 等. RGB-D SLAM综述［J］. 导航定位与授时, 2017, 4（6）: 9-18.

Wang Liujun, Chen Jiabin, Yu Huan, et al. An overview of RGB-D SLAM ［J］. Navigation Positioning and Timing, 2017, 4（6）: 9-18

［12］张露, 郑顺义, 杨锦锦. 基于Kinect的点云精度分析［J］. 郑州师范教育, 2015（2）: 51-54, 96.

［13］ HalmetschlagerFunek G, Suchi M, Kampel M, et al. An empirical evaluation of ten depth cameras: bias, precision, lateral noise, different lighting conditions and materials, and multiple sensor setups in indoor environments ［J］. IEEE Robotics & Automation Magazine, 2018, 26（1）: 67-77.

［14］余涛. Kinect应用开发实战［M］. 北京: 机械工业出版社, 2013.

［15］张广军, 王红, 赵慧洁, 等. 结构光三维视觉系统研究［J］. 航空学报, 1999, 20（4）: 78-80.

［16］王宝光, 贺忠海, 陈林才, 等. 结构光传感器模型及特性分析［J］. 光学学报, 2002, 22（4）: 481-484.

Wang Baoguang, He Zhonghai, Chen Lincai, et al. Model and performance analysis of structured light sensor ［J］. Acta Optica Sinica, 2002, 22（4）: 481-484.

［17］侯飞, 韩丰泽, 李国栋, 等. 基于飞行时间的三维成像研究进展和发展趋势［J］. 导航与控制, 2018, 17（5）: 1-7, 48.

Hou Fei, Han Fengze, Li Guodong, et al. Research progress and development trend of three dimensionalimaging cameras based on light flighttime technique ［J］. Navigation and Control, 2018, 17（5）: 1-7, 48

［18］ Hansard M, Lee S, Choi O, et al. Timeofflight cameras: principles, methods and applications ［M］. Springer Science & Business Media, 2012.

［19］ Endres F, Hess J, Sturm J, et al. 3D mapping with an RGB-D camera ［J］. IEEE Transactions on Robotics, 2014, 30（1）: 177-187.

［20］ JA GonzálezFraga, Kober V , DiazRamirez V H , et al. Accurate generation of the 3D map of environment with a RGB-D camera ［C］. Applications of Digital Image Processing XL. Society of PhotoOptical Instrumentation Engineers （SPIE） Conference Series, 2017.

［21］ Park Y, Lepetit V, Woo W. Textureless object tracking with online training using an RGB-D camera ［C］. IEEE International Symposium on Mixed & Augmented Reality. IEEE, 2011.

［22］ Wang K, Zhang G, Xia S. Templateless nonrigid reconstruction and motion tracking with a single RGB-D camera ［J］. IEEE Transactions on Image Processing, 2017, 26（12）: 5966-5979.

［23］ Lee Y H, Medioni G. RGB-D camera based navigation for the visually impaired ［C］. Proceedings of the RSS, 2011.

［24］ Lee Y H, Medioni G. RGB-D camera based wearable navigation system for the visually impaired ［J］. Computer Vision and Image Understanding, 2016, 149: 3-20.

［25］ Zhang L, Liu F, Tang J. Realtime system for driver fatigue detection by RGB-D camera ［J］. ACM Transactions on Intelligent Systems and Technology （TIST）, 2015, 6（2）: 1-17.

［26］ Chéné Y, Rousseau D, Lucidarme P, et al. On the use of depth camera for 3D phenotyping of entire plants ［J］. Computers and Electronics in Agriculture, 2012, 82: 122-127.

［27］ Wang W, Li C. Size estimation of sweet onions using consumergrade RGBdepth sensor ［J］. Journal of Food Engineering, 2014, 142: 153-162.

［28］ VázquezArellano M, Paraforos D S, Reiser D, et al. Determination of stem position and height of reconstructed maize plants using a timeofflight camera ［J］. Computers and Electronics in Agriculture, 2018, 154: 276-288.

［29］ Dionisio A, D José, FQ César, et al. An approach to the use of depth cameras for weed volume estimation ［J］. Sensors, 2016, 16（7）: 972.

［30］刘振宇, 丁宇祺. 自然环境中被遮挡果实的识别方法研究［J］. 计算机应用研究, 2020, 37（S2）: 333-335, 339.

Liu Zhenyu, Ding Yuqi. Research on recognition methods of shaded fruits in natural environment ［J］. Application Research of Computers, 2020, 37（S2）: 333-335, 339

［31］ Malik M H, Qiu R, Gao Y, et al. Tomato segmentation and localization method based on RGB-D camera ［J］. International Agricultural Engineering Journal, 2020, 28（4）: 278-287.

［32］ Nguyen T T, Vandevoorde K, Wouters N, et al. Detection of red and bicoloured apples on tree with an RGB-D camera ［J］. Biosystems Engineering, 2016: 33-44.

［33］ Seo J, Sa J, Choi Y, et al. A YOLObased separation of touchingpigs for smart pig farm applications ［C］. 2019 21st International Conference on Advanced Communication Technology （ICACT）. 2019.

［34］ Kuzuhara Y, Kawamura K, Yoshitoshi R, et al. A preliminarily study for predicting body weight and milk properties in lactating Holstein cows using a threedimensional camera system ［J］. Computers and Electronics in Agriculture, 2015, 111:186-193.

［35］ Jonguk L, Long J, Daihee P, et al. Automatic recognition of aggressive behavior in pigs using a kinect depth sensor ［J］. Sensors, 2016, 16（5）: 631.

［36］ Hu Y, Wang L, Xiang L, et al. Automatic nondestructive growth measurement of leafy vegetables based on kinect ［J］. Sensors, 2018, 18（3）: 806.

［37］ Sun G, Wang X. Threedimensional point cloud reconstruction and morphology measurement method for greenhouse plants based on the kinect sensor selfcalibration ［J］. Agronomy, 2019, 9（10）: 596.

［38］王可, 戈振扬, 郭浩, 等. 基于Xtion传感器的玉米果穗三维形态指标测量系统［J］. 传感器与微系统, 2015, 34（04）: 62-65.

Wang Ke, Ge Zhenyang, Guo Hao, et al. 3D shape indexes measurement system of maize earbased on Xtion sensor ［J］. Transducer and Microsystem Technologies, 2015, 34（4）: 62-65.

［39］刘慧, 徐慧, 沈跃, 等. Kinect传感器的植株冠层三维数据测量［J］. 农业现代化研究, 2016, 37（5）: 995-1000.

Liu Hui, Xu Hui, Shen Yue, et al. Measurement and 3Dreconstruction of plants based on Kinect sensor ［J］. Research of Agricultural Modernization, 2016, 37（5）: 995-1000.

［40］劳彩莲, 杨瀚, 李鹏, 等. 基于消费级深度相机的玉米植株三维重建［J］. 农业机械学报, 2019, 50（7）: 222-228.

Lao Cailian, Yang Han, Li Peng, et al. 3D reconstruction of maize plants based on consumer depth camera ［J］. Transactions of the Chinese Society for Agricultural Machinery, 2019, 50（7）: 222-228.

［41］杨斯, 高万林, 米家奇, 等. 基于RGB-D相机的蔬菜苗群体株高测量方法［J］. 农业机械学报, 2019, 50（S1）: 128-135.

Yang Si, Gao Wanlin, Mi Jiaqi, et al. Method for measurement of vegetable seedlings height based on RGB-D camera ［J］. Transactions of the Chinese Society for Agricultural Machinery, 2019, 50（S1）: 128-135.

［42］刘慧, 刘加林, 沈跃, 等. 基于融合深度信息的自动喷雾车全局非线性轨迹优化方法［J］. 农业机械学报, 2019, 50（5）: 33-42.

Liu Hui, Liu Jialin, Shen Yue, et al. Global nonlinear trajectory optimization method based on fusion of depth information for automatic spraying vehicle ［J］. Transactions of the Chinese Society for Agricultural Machinery, 2019, 50（5）: 33-42.

［43］麦春艳, 郑立华, 孙红, 等. 基于RGB-D相机的果树三维重构与果实识别定位［J］. 农业机械学报, 2015, 46（S1）: 35-40.

Mai Chunyan, Zheng Lihua, Sun Hong, et al. Research on 3D reconstruction of fruit tree and fruit recognition and location method based on RGB-D camera ［J］. Transactions of the Chinese Society for Agricultural Machinery, 2015, 46（S1）: 35-40.

［44］麦春艳, 郑立华, 肖昌一, 等. 一种基于信息融合苹果果实世界定位方法［J］. 东北农业大学学报, 2016, 47（1）: 74-80.

Mai Chunyan, Zheng Lihua, Xiao Changyi, et al. A world positioning method for apple fruits based on information fusion ［J］. Journal of Northeast Agricultural University, 2016, 47（1）: 74-80

［45］余承超. 机器人采摘葡萄的手眼协调技术研究［D］. 镇江: 江苏大学, 2019.

Yu Chengchao. Research on hand eye coordination technology of robot picking grapes ［D］. Zhenjiang: Jiangsu University, 2019.

［46］ Wang K, Guo H, Ma Q, et al. A portable and automatic Xtionbased measurement system for pig body size ［J］. Computers and Electronics in Agriculture, 2018, 148.

［47］郭浩, 张胜利, 马钦, 等. 基于点云采集设备的奶牛体尺指标测量［J］. 农业工程学报, 2014, 30（5）: 116-122.

Guo Hao, Zhang Shengli, Ma Qin, et al. Cow body measurement based on Xtion ［J］. Transactions of the Chinese Society of Agricultural Engineering, 2014, 30（5）: 116-122.

［48］王可, 郭浩, 刘威林, 等. 基于点云旋转归一化的猪体体尺测点提取方法［J］. 农业工程学报, 2017, 33（S1）: 253-259.

Wang Ke, Guo Hao, Liu Weilin, et al. Extraction method of pig body size measurement points based on rotation normalization of point cloud ［J］. Transactions of the Chinese Society of Agricultural Engineering, 2017, 33（S1）: 253-259.

［49］王可, 郭浩, 马钦, 等. 基于单视角点云镜像的猪只体尺测量方法［J］. 农业机械学报, 2018, 49（3）: 187-195.

Wang Ke, Guo Hao, Ma Qin, et al. Pig body size measurement method based on mirror reflection of single angle point cloud ［J］. Transactions of the Chinese Society for Agricultural Machinery, 2018, 49（3）: 187-195.

［50］刘波, 朱伟兴, 杨建军, 等. 基于深度图像和生猪骨架端点分析的生猪步频特征提取［J］. 农业工程学报, 2014, 30（10）: 131-137.

Liu Bo, Zhu Weixing, Yang Jianjun, et al. Extracting of pig gait frequency feature based on depth image and pig skeleton endpoints analysis ［J］. Transactions of the Chinese Society of Agricultural Engineering, 2014, 30（10）: 131-137

［51］施宏, 沈明霞, 刘龙申, 等. 基于Kinect的哺乳期母猪姿态识别算法的研究［J］. 南京农业大学学报, 2019, 42（1）: 177-183.

Shi Hong, Shen Mingxia, Liu Longshen, et al. Study on recognition method of lactating sowsposture based on Kinect ［J］. Journal of Nanjing Agricultural University, 2019, 42（1）: 177-183

［52］ Kuan Y W, Ee N O, Wei L S. Comparative study of Intel R200, Kinect v2, and primesense RGB-D sensors performance outdoors ［J］. IEEE Sensors Journal, 2019, 19（99）: 8741-8750.

［53］左超, 张晓磊, 胡岩, 等. 3D真的来了吗？——三维结构光传感器漫谈［J］. 红外与激光工程, 2020, 49（3）: 9-53.

Zuo Chao, Zhang Xiaolei, Hu Yan, et al. Has 3D finally come of age?——An introduction to 3D structuredlight sensor ［J］. Infrared and Laser Engineering, 2020, 49（3）: 9-53

［54］ Zanuttigh P, Marin G, Mutto C D, et al. Operating principles of structured light depth cameras ［M］. Springer International Publishing, 2016.

［55］ Oggier T, Lehmann M, Kaufmann R, et al. An allsolidstate optical range camera for 3D realtime imaging with subcentimeter depth resolution （SwissRanger）［J］. International Society for Optics and Photonics, 2004.

［56］丁津津, 张旭东, 高隽, 等. 基于TOF技术的3D相机应用研究综述［C］. 中国仪器仪表学会第十二届青年学术会议论文集, 2010: 76-79.

［57］ Gokturk S B, Tomasi C. 3D head tracking based on recognition and interpolation using a timeofflight depth sensor ［C］. IEEE Computer Society Conference on Computer Vision & Pattern Recognition. IEEE, 2004.

［58］徐胜勇, 卢昆, 潘礼礼, 等. 基于RGB-D相机的油菜分枝三维重构与角果识别定位［J］. 农业机械学报, 2019, 50（2）: 21-27.

Xu Shengyong, Lu Kun, pan Lili, et al. 3D reconstruction of rape branch and pod recognition based on RGB-D camera ［J］. Transactions of the Chinese Society for Agricultural Machinery, 2019, 50（2）: 21-27.

［59］张恒, 徐万红, 张禹. 激光雷达与深度相机融合的SLAM技术研究［J］. 机械工程师, 2020（7）: 22-24, 29.

Zhang Heng, Xu Wanhong, Zhang Yu. Research on SLAM technology of lidar and depth camera fusion ［J］. Mechanical Engineer, 2020（7）: 22-24, 29.

［60］ Mu L, Yao P, Zheng Y, et al. Research on SLAM algorithm of mobile robot based on the fusion of 2D LiDAR and depth camera ［J］. IEEE Access, 2020, 8: 157628-157642.

［61］ Song H, Choi W, Kim H. Robust visionbased relativelocalization approach using an RGBdepth camera and LiDAR sensor fusion ［J］. IEEE Transactions on Industrial Electronics, 2016, 63（6）: 3725-3736.

［62］苏本跃, 储小玉, 盛敏. 同态滤波下RGB-D点云配准方法［J］. 中国图象图形学报, 2019, 24（11）: 1985-1997.

Su Benyue, Chu Xiaoyu, Sheng min. RGB-D point cloud registration method via homomorphic filtering ［J］. Journal of Image and Graphics, 2019, 24（11）: 1985-1997.

［63］ Zhang K, Bi W, Zhang X, et al. A new kmeans clustering algorithm for point cloud ［J］. International Journal of Hybrid Information Technology, 2015, 8（9）: 157-170.

［64］潘波. 基于深度图修补的植物重建算法研究［D］. 重庆: 重庆大学, 2019.

Pan Bo. Research on plant reconstruction method based on depth map repair ［D］. Chongqing: Chongqing University, 2019.

[1]	殷悦;张慧春;郑加强;. 基于双目立体视觉的植物三维重建系统[J]. 中国农机化学报, 2021, 42(3): 129-135.
[2]	冯耀宁, 裴亮, 陈晓, 陈小兵, 刘燕, 陈彬. 变量施药关键技术综述变量施药关键技术综述[J]. 中国农机化学报, 2021, 42(12): 65-71.
[3]	朱丹;陈学东;张学俭;李锋;张建华;. 基于物联网的设施农业温室远程监控系统研究[J]. 中国农机化学报, 2020, 41(5): 176-181.
[4]	胡坤钰;李亚萍;. 螺旋输送试验台测试系统设计与试验[J]. 中国农机化学报, 2019, 40(2): 130-134.