This is a gesture: AI for industry has been taught to understand hand movements even with dirty gloves.

Computer scientists have developed an AI complex for controlling industrial devices and computers using gestures. The system uses a regular webcam and is able to detect movements even with protective gloves, in low light conditions and against a background of visual noise. The introduction of technology can simplify the management of equipment in production, in workshops, during rescue operations in rubble, as well as when humans interact with robots as part of a single team. At the same time, experts recommend using such an interface in conjunction with voice commands and other control channels to increase reliability.

How AI recognizes gestures

The development by scientists from Novgorod State University is based on artificial intelligence algorithms that make it possible to recognize hand movements made even with gloves, in low light and against a noisy background. This makes it possible to use the system in workshops, mines, tunnels or in emergency zones during rescue operations, when high dust levels, limited visibility and the need to use protective equipment make traditional control systems ineffective.

— At many industrial facilities, staff work in overalls, and access to control panels is often limited. At the same time, technical solutions that allow contactless control of devices usually require additional equipment and have limitations on operating conditions. Available alternatives are computer vision systems with standard video cameras. But their resistance to interference is insufficient," Igor Kulakov, one of the developers and a senior lecturer at the Department of Information Technology and Systems at NovSU, told Izvestia.

According to him, the specialists used an open library of computer vision in the proposed solution. It was improved by adding smoothing of finger tremors, retrained the neural network on images of gloved hands and adjusted the thresholds for triggering gestures for industrial conditions. For example, numerical values of the distances between the fingers were set so that the program could distinguish intentional gestures from random movements.

To give a command to the computer, you need to bring your hand up to the camera. The program recognizes the tip of the index finger, and the cursor repeats the movement. To click with the left or right key, the operator needs to connect the thumb with the index or middle finger, respectively. Merging with the nameless one enables dragging mode, and pulling to the side enables scrolling. All gestures are performed on the weight, without touching the manipulators or the screen, the scientist said.

The wisdom of translation: An AI navigator will help doctors and foreign patients understand each other

The system, which has no analogues in Russia, knows medical terminology and reads documents in 10 languages.

Where development will be in demand

— The developers did not set out the task of creating a gesture recognition system from scratch — such algorithms exist. The goal was to make the software package work stably where ordinary computers go blind. For example, when an operator is working with dirty gloves, the light is blinking, and machines and moving objects are in the background. We have shown that with the help of development, ordinary cameras and free libraries can perfectly cope with these disturbances," commented Igor Kulakov.

According to him, the accuracy of the system is 90-95%, and the reaction speed is less than 0.1 seconds. This is comparable to the performance of expensive foreign controllers, but at a significantly lower cost of the solution. A comparison with well-known foreign analogues has shown that the development provides a similar level of accuracy at a lower cost. At the same time, no specialized equipment is required for its use — a standard camera is sufficient.

Such technologies are in demand in automated industries, where, for example, equipment needs to be reconfigured, and the presence of a person nearby is impossible or dangerous, the scientist added. The development will also reduce the risk of electric shock to workers and wear on equipment. In addition, the system can be integrated into robot control systems so that humans and machines can work together.

— Such systems can be used in various tasks where a person is not in a clean area, but directly interacts with infrastructure facilities. However, 90-95% confidence is not enough for most areas. And it is not clear whether it will be possible to achieve one hundred percent results using the presented technology," Evgeny Dudorov, a representative of the board of the Consortium of Robotics and Intelligent Control Systems, commented to Izvestia.

He explained that when working with industrial facilities and complex machinery, accurate execution of commands is especially important, since incorrect interpretation of gestures can lead to accidents, material damage and risks to people. The reliability of such systems can be improved through combined solutions, for example, by supplementing gesture control with voice commands. According to the expert, voice commands are more unambiguous and are better recognized by AI algorithms.

Additionally, gestures require training and standardization, as different people perform them with variations. At the same time, voice control is easier to implement and is already actively used in robotics, the expert noted.

— The advantage of the system is that there is no need for external devices, a video camera is enough. At the same time, the complex can work in a noisy environment where the voice interface is useless. One of the disadvantages is that the development is not suitable for people with disabilities. The program also requires careful adjustment to environmental conditions — white balance, dust, fog, lighting, and others," said Roman Meshcheryakov, chief Researcher at the Trapeznikov Institute of Management Problems.

According to him, sign language is one of the ways to control technology. Now, multimodal interfaces are becoming widespread — with simultaneous control by voice, gestures, gaze, etc. These areas will develop both in the form of conventional external devices and, in the future— implantable chips.

— The advantage of development is in saving resources. Scientists have taken a mature open library and adapted it to their needs. Gesture recognition today is not a science, but engineering. The algorithmic basis for this has long been open, the difference between a laboratory prototype and a device on the shop floor is in the technical details," explained Kirill Golovan, a programmer and developer of a benchmark for testing gesture recognition programs using electromyography.

The limitations of these systems are in the "nature" of video sensors, he noted. They need direct hand visibility and a stable frame, which is not always available. For example, inertial sensors on the wrist or electromyograph bracelets, devices that read bioelectric muscle signals and convert them into digital commands, can improve the reliability of recognition.

According to the expert, such systems are in demand in medicine. For example, a surgeon wearing sterile gloves cannot touch a screen or keyboard, but he needs to view CT scans and manipulate 3D models of organs. Contactless gesture control allows you to solve this problem without violating sterility.

In addition, such technologies can empower people with limited mobility, for whom traditional input devices are becoming a barrier. Another promising area is interaction with ATMs and information terminals. According to experts, this creates a significant potential market for the use of such solutions.