Research Projects
Instance Segmentation and Software Design for Real-Time Tool Detection and Human-Computer Interaction
This project focuses on the application of instance segmentation and robust software design in the realm of smart manufacturing and human-computer interaction (HCI). Leveraging advanced computer vision techniques, specifically You-Only-Look-Once (YOLO)v5 and Mask Region-Based Convolutional Neural Networks (Mask R-CNN), the system achieves real-time tool detection and interactive HCI using eye gaze recognition. The software interface, developed with PyQt and Python, integrates these models to provide dynamic visual assistance, enhancing productivity and accuracy in manufacturing processes.
A Gaze-Driven Manufacturing Assembly Assistant System
This project presents an innovative gaze-driven assembly assistant system tailored for human-centered smart manufacturing. By integrating assembly step recognition using CNN and LSTM networks, repetitive action counting with Transformer networks, and real-time eye gaze estimation, the system provides dynamic visual assistance to enhance quality and productivity in manufacturing processes. The developed software interface offers contextual tool and part displays, detailed procedural instructions, and interactive guidance based on worker gaze, demonstrating significant improvements in assembly accuracy and operational efficiency.
Web-Based App for Machine Learning Education
This project focuses on developing a web-based platform for storing machine learning-ready datasets and enabling real-time machine learning education. The system integrates IoT and sensor networks for data collection and interactive learning environments, offering secure data handling and optimized user experiences.
Implementing Eye Movement Tracking for UAV Navigation
This project explores a novel approach to controlling unmanned aerial vehicles (UAVs) through electrooculography (EOG) based eye movement tracking. By developing an interface that translates eye movements into UAV navigation commands, the system integrates biometric technology into UAV control, offering precise and efficient maneuvering capabilities. The research demonstrates the potential of EOG in enhancing human-robot interaction, particularly benefiting individuals with mobility or communication challenges.
Repetitive Action Counting Through Joint Angle Analysis and Video Transformer Techniques
This project introduces an advanced method for repetitive action counting by integrating joint angle analysis with pose landmarks using Transformer networks. Addressing common challenges such as camera viewpoint variability, over-counting, under-counting, and sub-action differentiation, the proposed approach achieves superior performance on the RepCount dataset. By leveraging both skeletal data and joint angles, the system enhances the accuracy and robustness of action repetition detection, making significant strides in applications like fitness tracking, rehabilitation, and manufacturing operation monitoring.
Fine-Grained Activity Classification in Assembly Based on Multi-Visual Modalities
This project focuses on the development of a fine-grained activity classification system in assembly environments using multi-visual modalities. By leveraging scene-level and temporal-level features through a two-stage neural network architecture, the system achieves high accuracy in recognizing and predicting continuous assembly activities. The creation of a specialized dataset and the integration of RGB and hand skeleton frames enhance the robustness and precision of activity recognition, contributing to improved productivity and quality control in smart manufacturing.
Real-Time Multi-Modal Human–Robot Collaboration Using Gestures and Speech
This project explores the development of a real-time, multi-modal Human–Robot Collaboration (HRC) system that leverages both gestures and speech for seamless interaction between humans and industrial robots. By integrating convolutional neural networks and advanced speech recognition algorithms, the system achieves high accuracy in gesture and speech recognition, enhancing the efficiency and intuitiveness of human-robot collaborations in manufacturing environments.
Robotic Cognitive Rehabilitation System for Mild Cognitive Impairment
This project presents the design and development of a Robotic Cognitive Rehabilitation System aimed at assisting patients with Mild Cognitive Impairment (MCI). By leveraging computer vision and advanced machine learning algorithms, the system facilitates immersive and effective cognitive rehabilitation through automated target acquisition and real-time human-robot collaboration.
