Enhance pick-and-place performance using multimodal interaction in operation environment
ISSN: 0143-991X
Article publication date: 28 March 2023
Issue publication date: 6 June 2023
Abstract
Purpose
Currently, the vision and depth information obtained from the eye-to-hand RGB-D camera can apply to the reconstruction of the three-dimensional (3D) environment for a robotic operation workspace. The reconstructed 3D space contributes to a symmetrical and equal observation view for robots and humans, which can be considered a digital twin (DT) environment. The purpose of this study is to enhance the robot skill in the physical workspace, although the artificial intelligence (AI) technique has high performance of the robotic operation in the known environments.
Design/methodology/approach
A multimodal interaction framework is proposed in DT operation environments.
Findings
A fast image-based target segmentation technique is combined in the 3D reconstruction of the robotic operation environment from the eye-to-hand camera, thus expediting the 3D DT environment generation without accuracy loss. A multimodal interaction interface is integrated into the DT environment.
Originality/value
The users are supported to operate the virtual objects in the DT environment using speech, mouse and keyboard simultaneously. The humans’ operations in 3D DT virtual space are recorded, and cues are provided for the robot’s operations in practice.
Keywords
Acknowledgements
This work is supported by NSFC-Shenzhen Robotics Research Center Project with No. U2013207.
Citation
Guo, X. and Chen, Y. (2023), "Enhance pick-and-place performance using multimodal interaction in operation environment", Industrial Robot, Vol. 50 No. 4, pp. 699-705. https://doi.org/10.1108/IR-10-2022-0260
Publisher
:Emerald Publishing Limited
Copyright © 2023, Emerald Publishing Limited