selected publications

1. IJCIM

   Explainable Human Activity Recognition Based on Probabilistic Spatial Partitions for Symbiotic Workplaces

   Tuli, Tadele Belay, and Manns, Martin

   International Journal of Computer Integrated Manufacturing Feb 2023

   Abstract Bibtex DOI

   In recent years, smart workplaces that adapt to human activity and motion behavior have been proposed for cognitive production systems. In this respect, methods for identifying the feelings and activities of human workers are being investigated to improve the cognitive capability of smart machines such as robots in shared working spaces. Recognizing human activities and predicting the possible next sequence of operations may simplify robot programming and improve collaboration efficiency. However, human activity recognition still requires explainable models that are versatile, robust, and interpretative. Therefore, recognizing and analyzing human action details using continuous probability density estimates in different workplace layouts is essential. Three scenarios are considered: a standalone, a one-piece flow U-form, and a human-robot hybrid workplace. This work presents a novel approach to human activity recognition based on a probabilistic spatial partition (HAROPP). Its performance is compared to the geometric-bounded activity recognition method. Results show that spatial partitions based on probabilistic density contain 20% fewer data frames and 10% more spatial areas than the geometric bounding box. The approach, on average, detects human activities correctly for 81% of the cases for a pre-known workplace layout. HAROPP has scalability and applicability potential for cognitive workplaces with a digital twin in the loop for pushing the cognitive capabilities of machine systems and realizing human-centered environments.

   @article{tuli_explainable_2023,
     abbr = {IJCIM},
     title = {Explainable Human Activity Recognition Based on Probabilistic Spatial Partitions for Symbiotic Workplaces},
     author = {Tuli, Tadele Belay and Manns, Martin},
     year = {2023},
     month = feb,
     journal = {International Journal of Computer Integrated Manufacturing},
     volume = {0},
     number = {0},
     pages = {1--18},
     publisher = {{Taylor \& Francis}},
     issn = {0951-192X},
     doi = {10.1080/0951192X.2023.2177742},
     urldate = {2023-03-27},
     keywords = {continuous probabilistic model,Human activity recognition,human motion capture,human-robot collaboration,workplace layout},
     bibtex_show = true,
     selected = true
   }

2. SORO

   Telepresence Mobile Robots Design and Control for Social Interaction

   Tuli, Tadele Belay, Terefe, Tesfaye Olana, and Rashid, Md Mamun Ur

   International Journal of Social Robotics Jul 2020

   Abstract Bibtex DOI

   Human–robot interaction has extended its application horizon to simplify how human beings interact with each other through a remotely controlled telepresence robot. The fast growth of communication technologies such as 4G and 5G has elevated the potential to establish stable audio–video-data transmission. However, human–robot physical interactions are still challenging regarding maneuverability, controllability, stability, drive layout, and autonomy. Hence, this paper presents a systematic design and control approach based on the customer’s needs and expectations of telepresence mobile robots for social interactions. A system model and controller design are developed using the Lagrangian method and linear quadratic regulator, respectively, for different scenarios such as flat surface, inclined surface, and yaw (steering). The robot system is capable of traveling uphill (30\{^{}circ }\\∘) and has a variable height (600–1200 mm). The robot is advantageous in developing countries to fill the skill gaps as well as for sharing knowledge and expertise using a virtual and mobile physical presence.

   @article{tuli_telepresence_2020,
     abbr = {SORO},
     title = {Telepresence {Mobile} {Robots} {Design} and {Control} for {Social} {Interaction}},
     issn = {1875-4805},
     url = {https://doi.org/10.1007/s12369-020-00676-3},
     doi = {10.1007/s12369-020-00676-3},
     language = {en},
     urldate = {2020-09-14},
     journal = {International Journal of Social Robotics},
     author = {Tuli, Tadele Belay and Terefe, Tesfaye Olana and Rashid, Md Mamun Ur},
     month = jul,
     year = {2020},
     bibtex_show = true,
     selected = true
   }

3. CIRP CMS

   Identifying human intention during assembly operations using wearable motion capturing systems including eye focus

   Manns, Martin, Tuli, Tadele Belay, and Schreiber, Florian

   Procedia CIRP Jan 2021

   Abstract Bibtex DOI

   Simulating human motion behavior in assembly operations helps to create efficient collaboration plans for humans and robots. However, identifying human intention may require high quality human motion capture data in order to discriminate micro-actions and human attention. In this regard, a human motion capture setup that combines various systems such as joint body, finger, and eye trackers is proposed in combination with a methodology of identifying the intention of human operators as well as for predicting sequences of activities. The approach may lead to safer human-robot collaboration.

   @article{manns_identifying_2021,
     series = {54th {CIRP} {CMS} 2021 - {Towards} {Digitalized} {Manufacturing} 4.0},
     title = {Identifying human intention during assembly operations using wearable motion capturing systems including eye focus},
     volume = {104},
     issn = {2212-8271},
     url = {https://www.sciencedirect.com/science/article/pii/S2212827121010532},
     doi = {10.1016/j.procir.2021.11.155},
     language = {en},
     urldate = {2021-11-29},
     journal = {Procedia CIRP},
     author = {Manns, Martin and Tuli, Tadele Belay and Schreiber, Florian},
     month = jan,
     year = {2021},
     keywords = {human-robot collaboration, Operator 4.0, Eye tracker, human motion capture},
     pages = {924--929},
     bibtex_show = true,
     abbr = {CIRP CMS},
     selected = true
   }

4. IEEE ETFA

   Knowledge-Based Digital Twin for Predicting Interactions in Human-Robot Collaboration

   Tuli, Tadele Belay, Kohl, Linus, Chala, Sisay Adugna, Manns, Martin, and Ansari, Fazel

   In 2021 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA ) Sep 2021

   Abstract Bibtex DOI

   Semantic representation of motions in a human-robot collaborative environment is essential for agile design and development of digital twins (DT) towards ensuring efficient collaboration between humans and robots in hybrid work systems, e.g., in assembly operations. Dividing activities into actions helps to further conceptualize motion models for predicting what a human intends to do in a hybrid work system. However, it is not straightforward to identify human intentions in collaborative operations for robots to understand and collaborate. This paper presents a concept for semantic representation of human actions and intention prediction using a flexible task ontology interface in the semantic data hub stored in a domain knowledge base. This semantic data hub enables the construction of a DT with corresponding reasoning and simulation algorithms. Furthermore, a knowledge-based DT concept is used to analyze and verify the presented use-case of Human-Robot Collaboration in assembly operations. The preliminary evaluation showed a promising reduction of time for assembly tasks, which identifies the potential to i) improve efficiency reflected by reducing costs and errors and ultimately ii) assist human workers in improving decision making. Thus the contribution of the current work involves a marriage of machine learning, robotics, and ontology engineering into DT to improve human-robot interaction and productivity in a collaborative production environment.

   @inproceedings{tuli_knowledge-based_2021,
     title = {Knowledge-{Based} {Digital} {Twin} for {Predicting} {Interactions} in {Human}-{Robot} {Collaboration}},
     doi = {10.1109/ETFA45728.2021.9613342},
     booktitle = {2021 26th {IEEE} {International} {Conference} on {Emerging} {Technologies} and {Factory} {Automation} ({ETFA} )},
     author = {Tuli, Tadele Belay and Kohl, Linus and Chala, Sisay Adugna and Manns, Martin and Ansari, Fazel},
     month = sep,
     year = {2021},
     keywords = {Collaboration, digital twin, Digital twin, human action models, human-robot interaction, Knowledge based systems, machine learning, Ontologies, ontology, Predictive models, Productivity, Semantics},
     pages = {1--8},
     abbr = {IEEE ETFA},
     bibtex_show = true,
     selected = true,
     file = {2021130209.pdf}
   }

5. CIRP CARV Best paper award

   Understanding Shared Autonomy of Collaborative Humans Using Motion Capture System for Simulating Team Assembly

   Tuli, Tadele Belay, Manns, Martin, and Jonek, Michael

   In Towards Sustainable Customization: Bridging Smart Products and Manufacturing Systems 2022

   Abstract Bibtex PDF DOI

   In virtual production planning, simulating human motions helps to improve process planning and interaction efficiency. However, simulating multiple humans sharing tasks in a shared workplace requires understanding how human workers interact and share autonomy. In this regard, an Inertial Measurement Unit based motion capture is employed for understanding shifting roles and learning effects. Parameters such as total time, distance, and acceleration variances in repetition are considered for modeling collaborative motion interactions. The results distinguish motion patterns versus the undertaken interactions. This work may serve as an initial input to model interaction schemes and recognize human actions behavior during team assembly. Furthermore, the concept can be extended toward a human-robot shared autonomy.

   @inproceedings{tuli_understanding_2022,
     abbr = {CIRP CARV},
     award = {Best paper award},
     address = {Cham},
     series = {Lecture {Notes} in {Mechanical} {Engineering}},
     title = {Understanding {Shared} {Autonomy} of {Collaborative} {Humans} {Using} {Motion} {Capture} {System} for {Simulating} {Team} {Assembly}},
     isbn = {978-3-030-90700-6},
     doi = {10.1007/978-3-030-90700-6_59},
     language = {en},
     booktitle = {Towards {Sustainable} {Customization}: {Bridging} {Smart} {Products} and {Manufacturing} {Systems}},
     publisher = {Springer International Publishing},
     author = {Tuli, Tadele Belay and Manns, Martin and Jonek, Michael},
     editor = {Andersen, Ann-Louise and Andersen, Rasmus and Brunoe, Thomas Ditlev and Larsen, Maria Stoettrup Schioenning and Nielsen, Kjeld and Napoleone, Alessia and Kjeldgaard, Stefan},
     year = {2022},
     keywords = {Human motion capture, Shared autonomy, Team interaction, Manual assembly, Role shifting},
     pages = {527--534},
     bibtex_show = true,
     selected = true,
     pdf = {shared_autonomy_final.pdf}
   }

6. CIRP CARV Best Young Fellow Paper Award

   A Motion Capture-Based Approach to Human Work Analysis for Industrial Assembly Workstations

   Jonek, Michael, Tuli, Tadele Belay, and Manns, Martin

   In Production Processes and Product Evolution in the Age of Disruption 2023

   Abstract DOI

   In industry, manual work is becoming increasingly important despite high labor costs due to the trend towards smaller productionProduction volumes and a higher number of variants. In order to identify optimization potential and increase productivity, there is a strong need to analyze and understand manual processes. Especially in Small and Medium-sized Enterprises (SME), which have limited resources for classic process time analysis, these are rarely available. In this work, a method for automatic generation of process time analyses of manual assemblyManual assembly processes is presented. It employs an industrial human activity recognitionHuman activity recognition system. Human motion data is captured in the industrial environment for manual assemblyManual assembly operation. It is post processed using a spatial partitioning approach. The study shows about 76% of the manual operations in the proposed use case scenario are automatically detected and the remaining are hardly identified. It is shown that process time analysis can be carried out without expert knowledge and without significant manual effort and provides knowledge about the process, which can be used to identify optimization potentials to increase productivity.

7. CIRP CMS

   Latent Space Based Collaborative Motion Modeling from Motion Capture Data for Human Robot Collaboration

   Tuli, Tadele Belay, Henkel, Martin, and Manns, Martin

   Procedia CIRP Jan 2022

   Abstract Bibtex DOI

   Collaborative assembly operation is one of the current challenges regarding human robot collaboration (HRC). In this context, it is still unclear how robots and humans should behave in handling an object and anticipating mutual care. In many cases, the modeling of collaborative behaviors shows difficulties, which can be addressesd by simplifying motion modeling techniques. In the current work, we propose a latent space approach that combines functional principal component analysis to derive low dimensional features with Gaussian mixture models to generate high-likelihood motion behavior estimates. This approach may increase agility in task planning and reduce programming difficulties in HRC.

   @article{tuli_latent_2022,
     abbr = {CIRP CMS},
     series = {Leading manufacturing systems transformation – {Proceedings} of the 55th {CIRP} {Conference} on {Manufacturing} {Systems} 2022},
     title = {Latent {Space} {Based} {Collaborative} {Motion} {Modeling} from {Motion} {Capture} {Data} for {Human} {Robot} {Collaboration}},
     volume = {107},
     issn = {2212-8271},
     url = {https://www.sciencedirect.com/science/article/pii/S2212827122004127},
     doi = {10.1016/j.procir.2022.05.128},
     language = {en},
     urldate = {2022-05-27},
     journal = {Procedia CIRP},
     author = {Tuli, Tadele Belay and Henkel, Martin and Manns, Martin},
     month = jan,
     year = {2022},
     keywords = {motion capture, principal component analysis, human robot collaboration, Gaussian mixture models, latent space, motion modeling},
     pages = {1180--1185},
     bibtex_show = true,
     selected = true
   }

8. JIST

   Human motion quality and accuracy measuring method for human–robot physical interactions

   Tuli, Tadele Belay, Manns, Martin, and Zeller, Sebastian

   Intelligent Service Robotics Jul 2022

   Abstract Bibtex DOI

   In human–robot collaboration (HRC), human motion capture can be considered an enabler for switching autonomy between humans and robots to create efficient and safe operations. For this purpose, wearable motion tracking systems such as IMU and lighthouse-based systems have been used to transfer human joint motions into robot controller models. Due to reasons such as global positioning, drift, and occlusion, in some situations, e.g., HRC, both systems have been combined. However, it is still not clear if the motion quality (e.g., smoothness, naturalness, and spatial accuracy) is sufficient when the human operator is in the loop. This article presents a novel approach for measuring human motion quality and accuracy in HRC. The human motion capture has been implemented in a laboratory environment with a repetition of forty-cycle operations. Human motion, specifically of the wrist, is guided by the robot tool center point (TCP), which is predefined for generating circular and square motions. Compared to the robot TCP motion considered baseline, the hand wrist motion deviates up to 3 cm. The approach is valuable for understanding the quality of human motion behaviors and can be scaled up for various applications involving human and robot shared workplaces.

   @article{tuli_human_2022,
     abbr = {JIST},
     title = {Human motion quality and accuracy measuring method for human–robot physical interactions},
     issn = {1861-2776, 1861-2784},
     url = {https://link.springer.com/10.1007/s11370-022-00432-8},
     doi = {10.1007/s11370-022-00432-8},
     language = {en},
     urldate = {2022-08-04},
     journal = {Intelligent Service Robotics},
     author = {Tuli, Tadele Belay and Manns, Martin and Zeller, Sebastian},
     month = jul,
     year = {2022},
     bibtex_show = true,
     selected = true
   }