基于Kinect相机的多视角RGB-D信息融合的甜椒姿态估计研究

doi:10.13733/j.jcam.issn.2095-5553.2023.10.023

摘要/Abstract

摘要： 针对在复杂环境中由于存在遮挡导致果蔬姿态计算不准甚至无法识别这一问题，提出一种基于Kinect相机的甜椒姿态估计研究方法，通过融合多视角下的甜椒点云信息，结合PCA算法估计出甜椒在空间中的三维姿态。基于Kinect相机的多视角RGB-D信息融合的甜椒姿态估计研究方法是将两间隔为60°的视角下Kinect相机采集甜椒的点云信息融合，并在ICP精配准过程中引入对应点对局部法向量夹角与平均曲率阈值双约束项提高点云配准精度，以解决两视角点云配准效果差、重叠度不高从而造成的由PCA算法估计点云姿态偏差大的技术问题。试验结果表明在无遮挡情况下，本文算法的误差均值为5.15°，在双视角中其中某一视角有遮挡情况下的综合均值误差为5.67°，双视角均有遮挡情况下综合平均误差为7.86°，满足实际作业时的技术要求。本研究不仅适用于甜椒的空间三维姿态识别，同时为其他智能采摘机器人姿态识别系统搭建提供技术参考。

关键词: 甜椒, 机器视觉, 姿态识别, 点云配准, PCA算法, 颜色阈值

Abstract: Aiming at the problem of inaccurate calculation or even unrecognition of fruit and vegetable poses due to the presence of occlusion in complex environments, a research method for estimating the poses of bell peppers based on Kinect camera is proposed to estimate the threedimensional poses of bell peppers in space by fusing the point cloud information of bell peppers from multiple viewpoints combined with the PCA algorithm. This method is based on fusing the point cloud information of bell peppers collected by Kinect camera from two viewpoints separated by 60°, and introducing the dual constraints of the corresponding point pairs of local normal vector angle and mean curvature threshold to improve the accuracy of the point cloud alignment during the ICP fine alignment process, in order to solve the technical problem of poor point cloud alignment and low overlap between the two viewpoints, which results in the large deviation of the point cloud attitude estimated by the PCA algorithm. The experimental results show that the mean value of the error of the algorithm in this paper is 5.15° in the case of no occlusion, the integrated mean error in the case of occlusion in one of the dualview perspectives is 5.67°, and the integrated mean error in the case of occlusion in both perspectives is 7.86°, which meets the technical requirements of the actual operation. This study is not only applicable to the spatial threedimensional pose recognition of bell peppers, but also provides technical references for the construction of other intelligent picking robot attitude recognition systems.

Key words: sweet pepper, machine vision, pose estimation, point cloud registration, PCA algorithm, color threshold

中图分类号:

牟向伟, 孙国奇, 陈林涛, 于新业, 彭柱菁, 伍倩. 基于Kinect相机的多视角RGB-D信息融合的甜椒姿态估计研究[J]. 中国农机化学报, 2023, 44(10): 159-167.

Mou Xiangwei, Sun Guoqi, Chen Lintao, Yu Xinye, Peng Zhujing, Wu Qian. Research on sweet pepper pose estimation based on multi view RGB-D information fusion of Kinect camera[J]. Journal of Chinese Agricultural Mechanization, 2023, 44(10): 159-167.

参考文献

［1］　蒋先平, 徐灿, 刘丰溥, 等. 果蔬采摘机器人系统的应用与发展［J］. 现代农业装备, 2021, 42(2): 2-8, 13.
Jiang Xianping, Xu Can, Liu Fengpu, et al. Application and development of fruit and vegetable picking robot system ［J］. Modern Agricultural Equipments, 2021, 42(2): 2-8, 13.
［2］　Kondo N, Nishitsuji Y, Ling P P, et al. Visual feedback guided robotic cherry tomato harvesting ［J］. Transactions of the ASAE, 1996, 39(6): 2331-2338.
［3］　熊俊涛, 林睿, 刘振, 等. 夜间自然环境下荔枝采摘机器人识别技术［J］. 农业机械学报, 2017, 48(11): 28-34.
Xiong Juntao, Lin Rui, Liu Zhen, et al. Visual technology of picking robot to detect litchi at nighttime under natural environment ［J］. Transactions of the Chinese Society for Agricultural Machinery, 2017, 48(11): 28-34.
［4］　Slaughter D C, Harrell R C. Discriminating fruit for robotic harvest using color in natural outdoor scenes ［J］. Transactions of the ASAE, 1989, 32(2): 757-763.
［5］　Billingsley J. Mechatronics and machine vision ［M］. Research Studies Press, 2000.
［6］　赵辉, 乔艳军, 王红君, 等. 基于改进YOLOv3的果园复杂环境下苹果果实识别［J］. 农业工程学报, 2021, 37(16): 127-135.
Zhao Hui, Qiao Yanjun, Wang Hongjun, et al. Apple fruit recognition in complex orchard environment based on improved YOLOv3 ［J］. Transactions of the Chinese Society of Agricultural Engineering, 2021, 37(16): 127-135.
［7］　闫建伟, 赵源, 张乐伟, 等. 改进FasterRCNN自然环境下识别刺梨果实［J］. 农业工程学报, 2019, 35(18): 143-150.
Yan Jianwei, Zhao Yuan, Zhang Lewei, et al. Recognition of Rosa roxbunghii in natural environment based on improved Faster RCNN ［J］. Transactions of the Chinese Society of Agricultural Engineering, 2019, 35(18): 143-150.
［8］　金玉成, 刘继展, 徐朱洁, 等. 农业机器人技术发展现状与趋势［J］. 农业工程技术, 2021, 41(33): 87.
［9］　张哲, 王毅, 付舜, 等. 基于咬合型末端执行器的柑橘采摘机器人采摘姿态研究［J］. 中国农业科技导报, 2018, 20(5): 75-82.
Zhang Zhe, Wang Yi, Fu Shun, et al. Study on the harvesting posture of the citrus harvesting robot based on occlusal endeffector ［J］. Journal of Agricultural Science and Technology, 2018, 20(5): 75-82.
［10］　宁政通, 罗陆锋, 廖嘉欣, 等. 基于深度学习的葡萄果梗识别与最优采摘定位［J］. 农业工程学报, 2021, 37(9): 222-229.
Ning Zhengtong, Luo Lufeng, Liao Jiaxin, et al. Recognition and the optimal picking point location of grape stems based on deep learning ［J］. Transactions of the Chinese Society of Agricultural Engineering, 2021, 37(9): 222-229.
［11］　Eizentals P, Oka K. 3D pose estimation of green pepper fruit for automated harvesting ［J］. Computers and Electronics in Agriculture, 2016, 128: 127-140.
［12］　谢忠红, 徐莹, 姬长英, 等. 基于计算机视觉的苹果生长姿态估算多方法融合［J］. 农业机械学报, 2011, 42(11): 154-157, 153.
Xie Zhonghong, Xu Ying, Ji Changying, et al. Estimation method of apple growing attitude based on computer vision ［J］. Transactions of the Chinese Society for Agricultural Machinery, 2011, 42(11): 154-157, 153.
［13］　董坦坦, 姬长英, 周俊, 等. 成熟番茄的图像识别及其位姿的获取研究［J］. 江西农业学报, 2009, 21(8): 152-155.
Dong Tantan, Ji Changying, Zhou Jun, et al. Recognition of ripe tomato image and obtaining of its pose ［J］. Acta Agriculturae Jiangxi, 2009, 21(8): 152-155.
［14］　Yin W, Wen H, Ning Z, et al. Fruit detection and pose estimation for grape clusterharvesting robot using binocular imagery based on deep neural networks ［J］. Frontiers in Robotics and AI, 2021: 163.
［15］　Lin G, Tang Y, Zou X, et al. Guava detection and pose estimation using a lowcost RGB-D sensor in the field ［J］. Sensors, 2019, 19(2): 428.
［16］　Li H, Zhu Q, Huang M, et al. Pose estimation of sweet pepper through symmetry axis detection ［J］. Sensors, 2018, 18(9): 3083.
［17］　Bac C W, Roorda T, Reshef R, et al. Analysis of a motion planning problem for sweetpepper harvesting in a dense obstacle environment ［J］. Biosystems Engineering, 2016, 146: 85-97.
［18］　Besl P J, McKay N D. Method for registration of 3-D shapes ［C］. Sensor fusion IV: control paradigms and data structures. Spie, 1992, 1611: 586-606.
［19］　宋涛, 曹利波, 赵明富, 等. 三维点云中关键点的配准与优化算法［J］. 激光与光电子学进展, 2021, 58(4): 375-383.
Song Tao, Cao Libo, Zhao Mingfu, et al. Registration and optimization algorithm of key points in threedimensional point cloud ［J］. Laser & Optoelectronics Progress, 2021, 58(4): 375-383.
［20］　荆路, 武斌, 方锡禄. 基于SIFT特征点结合ICP的点云配准方法［J］. 激光与红外, 2021, 51(7): 944-950.
Jing Lu, Wu Bin, Fang Xilu. Point cloud registration method based on the SIFT feature points combined with ICP algorithm ［J］. Laser & Infrared, 2021, 51(7): 944-950.
［21］　刘玉珍, 张强, 林森. 一种改进的基于快速点特征直方图的ICP点云配准算法［J］. 激光与光电子学进展, 2021, 58(6): 275-282.
Liu Yuzhen, Zhang Qiang, Lin Sen. Improved ICP point cloud registration algorithm based on fast point feature histogram ［J］. Laser & Optoelectronics Progress, 2021, 58(6): 275-282.
［22］　张晗, 康国华, 张琪, 等. 基于改进SAC-IA算法的激光点云粗配准［J］. 航天控制, 2019, 37(5): 67-74.
Zhang Han, Kang Guohua, Zhang Qi, et al. Laser point cloud coarse registration based on improved SAC-IA algorithm ［J］. Aerospace Control, 2019, 37(5): 67-74.
［23］　Wang Y, Chen Z, Wang J, et al. ImprovedPCA algorithm using artificial sparsity in dynamic MRI ［J］. Computational and Mathematical Methods in Medicine, 2017.
（上接第158页）
［10］　郭蓓佳, 籍颖, 锡建中, 等. 基于图像处理的蛋鸡体重估测方法研究［J］. 中国家禽, 2021, 43(11): 68-73.
Guo Beijia, Ji Ying, Xi Jianzhong, et al. Study on estimation model of layer weight based on image processing ［J］. China Poultry, 2021, 43(11): 68-73.
［11］　赵守耀, 陆辉山, 王福杰, 等. 基于轮廓特征的单只蛋鸡行为识别方法［J］. 中国农机化学报, 2022, 43(2): 143-147, 181.
Zhao Shouyao, Lu Huishan, Wang Fujie, et al. Recognition method of single layer behavior based on contour feature ［J］. Journal of Chinese Agricultural Mechanization, 2022, 43(2): 143-147, 181.
［12］　肖海俊. 彩色图像色彩聚类算法研究［J］. 湖北广播电视大学学报, 2012, 32(2): 158-160.Xiao Haijun. Research on color clustering algorithm of color images ［J］. Journal of Hubei Radio & Television University, 2012, 32(2): 158-160.
［13］　居继光, 黎寿丰, 居勇, 等. 邵伯鸡屠宰性能测定及肌肉品质分析［J］. 中国家禽, 2005(S1): 18-20.Ju Jiguang, Li Shoufeng, Ju Yong, et al. Study on slaughter performance and muscle quality of Shaobo chicks ［J］. China Poultry, 2005(S1): 18-20.
［14］　顿绍坤, 魏海平, 孙明柱. RGB颜色空间新的色差公式［J］. 科学技术与工程, 2011, 11(8): 1833-1836.
Dun Shaokun, Wei Haiping, Sun Mingzhu. A new distance color difference formula in RGB color space ［J］. Science Technology and Engineering, 2011, 11(8): 1833-1836.
［15］　沈志伟. 解读从RGB图到CMYK图的“变身”之法［J］. 印刷技术, 2011(3): 47-50.
［16］　刘柏江, 姜明新. 基于SIFT特征的图像匹配算法［J］. 信息系统工程, 2011(5): 34-36, 41.
［17］　戴雯惠, 樊凌. 基于改进透视变换的畸变图像校正方法研究［J］. 信息通信, 2020(11): 63-65.
［18］　曾兆滨, 郑华, 蔡坚勇, 等. 基于多项式回归的四带图像偏色校正算法［J］. 计算机系统应用, 2018, 27(4): 209-214.
Zeng Zhaobin, Zheng Hua, Cai Jianyong, et al. Color correction algorithm for fourband image based on polynomial regression ［J］. Computer Systems & Applications, 2018, 27(4): 209-214.
［19］　陈晓文, 郑华, 蔡坚勇, 等. 优化多项式回归的四带图像偏色校正算法［J］. 计算机系统应用, 2020, 29(3): 223-227.
Chen Xiaowen, Zheng Hua, Cai Jianyong, et al. Fourband image color cast correction algorithms based on optimized polynomial regression ［J］. Computer Systems & Applications, 2020, 29(3): 223-227.
［20］　Milan Sonka, Vaclav Hlavac, Roger Boyle著. 艾海舟, 苏延超译. 图像处理、分析与机器视觉(第三版)［M］. 北京: 清华大学出版社, 2011.
［21］　泮樟胜, 叶连宝. 浅谈利用PhotoShop精确计算图形面积［J］. 绿色科技, 2012(8): 261-263.

[1]	王焱清, , 汤旸, 杨光友, . 面向机器人柑橘采摘的控制系统设计与试验[J]. 中国农机化学报, 2023, 44(9): 146-153.
[2]	侯义锋, , 钱俊, , 王梁, 贺杰, 闭耀锋. 砂糖橘采摘机器人分拣机构设计[J]. 中国农机化学报, 2023, 44(9): 183-189.
[3]	王大明, 何逸霏, 李华英, 苟于江, 何辉波. 柑橘采摘机器人图像识别算法研究[J]. 中国农机化学报, 2023, 44(9): 222-226.
[4]	姜来, 王文娣, 霍晓静, 王辉, 唐娟, 李丽华, (. 死鸡识别机器人系统设计与试验[J]. 中国农机化学报, 2023, 44(8): 81-87.
[5]	刘浩, 孟令峰, 王松峰, 刘自畅, 杜海娜, 孙福山. 基于机器视觉的烤烟鲜烟成熟度判别模型优选[J]. 中国农机化学报, 2023, 44(8): 118-124.
[6]	舒卓, 陈立平, 陈梅香, 张瑞瑞, 郭新宇, 温维亮. 单目多视角图像的美国白蛾三维模型重建系统与试验[J]. 中国农机化学报, 2023, 44(8): 168-173.
[7]	刘雯雯, 孙裕晶, 姜树辉, 姜鹏. 遮挡情况下的行人检测方法研究[J]. 中国农机化学报, 2023, 44(7): 179-186.
[8]	周红标, 朱亚文, 刘晓洋, 张国良, 李边豪. 基于机器视觉的水稻印刷播种机成种率检测[J]. 中国农机化学报, 2023, 44(7): 194-199.
[9]	罗山, 侯俊涛, 郑彬. 基于机器视觉的石榴品质自动分级方法[J]. 中国农机化学报, 2023, 44(3): 117-122.
[10]	吴俊鹏, 黄光文, 李君, . 水果病害视觉与光谱检测技术研究现状及展望[J]. 中国农机化学报, 2023, 44(2): 112-118.
[11]	李志臣, 凌秀军, 李鸿秋. 基于机器视觉的槟榔分级方法研究[J]. 中国农机化学报, 2023, 44(1): 137-141.
[12]	耿丽杰, 顾健, 别晓婷, 冉维旭, 兰玉彬, . 基于Scan Context与NDT-ICP相融合的果园建图方法研究[J]. 中国农机化学报, 2022, 43(7): 44-50.
[13]	王珊, 薛新宇, 郭祥雨. 基于机器视觉的油菜菌核病分级检测研究[J]. 中国农机化学报, 2022, 43(5): 47-53.
[14]	刘媛杰, 张泽亮, 张洪洲, 李勇, 李伟强. 基于机器视觉技术的棉籽破损检测[J]. 中国农机化学报, 2022, 43(5): 71-76.
[15]	郑如新, 孙青云, 肖国栋. 基于机器视觉的金银花图像识别处理算法研究[J]. 中国农机化学报, 2022, 43(4): 153-159.