Department of Industrial and Materials Science CHALMERS UNIVERSITY OF TECHNOLOGY Gothenburg, Sweden 2020 Human Motion Analysis based on Human Pose Estimation Exploring the potential of AviX to import data directly from pose estimation and a way of working with ergonomics Master’s Thesis in Production Engineering RINKKHESH VENKATESULU SHASHANK VENKATESH KOUNDINYA MASTER’S THESIS 2020 Human Motion Analysis based on Human Pose Estimation Exploring the potential of AviX to import data directly from pose estimation and a way of working with ergonomics RINKKHESH VENKATESULU SHASHANK VENKATESH KOUNDINYA Department of Industrial and Materials Science Division of Production Systems CHALMERS UNIVERSITY OF TECHNOLOGY Gothenburg, Sweden 2020 iii Human Motion Analysis based on Human Pose Estimation Exploring the potential of AviX to import data directly from pose estimation and a way of working with ergonomics RINKKHESH VENKATESULU, SHASHANK VENKATESH KOUNDINYA © RINKKHESH VENKATESULU, SHASHANK VENKATESH KOUNDINYA, 2020. Supervisors: Roland Örtengren, Department of Industrial and Materials Science Oskar Hård, Virtual Manufacturing Sweden AB Examiner: Cecilia Berlin, Department of Industrial and Materials Science Master’s Thesis 2020 Department of Industrial and Materials Science Division of Production Systems Chalmers University of Technology SE-412 96 Gothenburg Telephone +46 31 772 1000 Cover: Operator performing the work task and the AviX software with ergonomic scoring Printed by Chalmers Reproservice Gothenburg, Sweden 2020 iv Human Motion Analysis based on Human Pose Estimation Exploring the potential of AviX to import data directly from pose estimation and a way of working with ergonomics RINKKHESH VENKATESULU SHASHANK VENKATESH KOUNDINYA Department of Industrial and Materials Science Chalmers University of Technology Abstract In today’s scenario, the means of performing ergonomic assessment are mostly limited to obtaining the required information through manual observation. While this is good for assessing an already existing workplace, more accurate information can be obtained with the use of current tracking technology and valuable time can be saved in comparison with the manual way of observation. This thesis attempts to automate the process of filling in a part of an ergonomic assessment in the AviX software from posture data. The work done also delves into the method of obtaining this posture data automatically from the use of appropriate systems. An exploration into the existing ergonomic standards pertaining to the scope of the thesis is also made. The different techniques to obtain movement data are discussed. Each technique is analyzed to find out the means of extracting the required posture data. The method to make it possible in real-time is described as well. An explanation of the working of the AviX software in storing information pertaining to the created tasks is given. The process of automatically modifying the task information in the software is described. A suitable experiment to test the entire procedure is also created and explained. A study of the person’s posture during the experiment is discussed. The future possibilities with this method of obtaining movement data are also addressed and explained in this thesis. Suggestions on how to combine our thesis work with the work done in others are also given. Keywords: Real-time, Pose estimation, Motion tracking, Ergonomic assessment, AviX automation, VR usage, REBA, RAMP. vi Acknowledgements This thesis was carried out at Virtual Manufacturing Sweden AB as part of the final project of the Master’s program in Production Engineering at Chalmers University of Technology. We would like to thank our supervisor at Virtual Manufacturing Sweden AB, Oskar Hård, for his invaluable support throughout this project. Special thanks to Johanna Sigvardsson, Torbjörn Danielsson, Eric Gustafsson, Johanna Dahlvik and Daniel Kron for their contribution. We would also like to thank all our colleagues and peers at Virtual Manufacturing Sweden AB whom we met over the course of this thesis project. We send thanks to Oskar Ljung & Erik Emanuelsson at Solme AB for their crucial contribution. A huge and special thanks goes to our supervisor at Chalmers University of Technology, Roland Örtengren, for his guidance and support in making this thesis, and especially this report, a success. We truly appreciate the time and effort you put in with us during this thesis and we are honoured to have worked with you. We also thank our examiner, Cecilia Berlin, for her support and assistance not only during this thesis, but also over the course of the entire Master’s program. We would like to specially thank our parents for their love and constant support and dedication. You are our moral pillars and we are forever indebted to you. Finally, we would like to send warm thanks to all the healthcare workers, doctors, hospitals, and all the people involved in the fight against the Covid-19 pandemic. Thank you for your dedication in safeguarding the health and wellbeing of the people. Rinkkhesh Venkatesulu & Shashank Venkatesh Koundinya, Gothenburg, September 2020 viii Contents List of Figures x 1 Introduction 1 1.1 Background 1 1.2 Problem Definition 1 1.3 Purpose 2 1.4 Aim 2 1.5 Limitations 2 2 Frame of Reference 3 2.1 Ergonomic Standards 3 2.1.1 Volvo Group Ergonomic Memorandum 3 2.1.2 Volvo Group Ergonomic Standard 4 2.2 Ergonomic Tools 4 2.2.1 Rapid Entire Body Assessment (REBA) 5 2.2.2 RAMP 6 2.3 AviX 8 2.4 Pose Estimation 8 2.4.1 Tensorflow - Openpose estimation 9 2.4.2 Xsens MVN analyse 11 2.4.3 HTC Vive with Unity 12 3 Methodology 13 3.1 Data Collection 13 3.2 Experimental Setup 13 3.3 Pose Estimation Data Computation 15 3.3.1 Xsens MVN Analyse 15 ix 3.3.2 HTC Vive System in combination with Unity 17 3.4 Accuracy Testing 20 3.5 Storage of Collected Data 20 3.6 Ergonomic Evaluation 21 3.6.1 REBA 21 3.6.2 RAMP 22 3.7 Data Transfer to AviX 26 4 Results 30 4.1 Real-Time Pose Estimation 30 4.2 Data Transfer to AviX 34 4.3 Ergonomic Evaluation 36 4.3.1 REBA 36 4.3.2 RAMP 37 5 Discussion 40 5.1 Obtained Results 40 5.2 Future Work 41 6 Conclusion 44 Bibliography 45 x List of Figures 2.1 Openpose joint numbers 10 3.1 Created work area 14 3.2 Xsens body segment tracker positions 16 3.3 Tracker positions 18 3.4 Step computation with trackers 18 3.5 Alternate step computation with trackers 19 3.6 Bend angle computation with trackers 20 3.7 REBA assessment sheet 22 3.8 RAMP I assessment sheet 24 3.9 RAMP II assessment sheet 25 3.10 Sample tree layout 27 3.11 Exported Json file 27 3.12 Modified Json file for import 28 3.13 AviX user interface after importing the Json file 29 4.1 Real-time Pose Estimation 30 4.2 Bend angle variation 31 4.3 Steps made 32 4.4 Step Rate graph 32 4.5 Variation of bend angles and steps made within one second 33 4.6 Result after AviX import 36 4.7 REBA assessment of assembly process 37 4.8 RAMP I assessment of assembly process 38 4.9 RAMP II assessment of assembly process 39 1 1 Introduction 1.1. Background Virtual Manufacturing Sweden AB is a company that focuses on supplying lean-based production and logistics development services. The services include station design, work instructions, value stream mapping (VSM), design of assembly lines with associated standardized work methods and more. Virtual Manufacturing makes use of the AviX software, developed by the company Solme AB, to handle the work done in the Assembly and Production Flow department. This thesis is concerned with the transfer and interpretation of posture data to be used in ergonomic evaluation, and also the standard of ergonomics within the company it is working with today and its use of ergonomic data. The main reason for Virtual Manufacturing AB commissioning this thesis is to understand how to make an effective use of the REBA ergonomic assessment tool present in the latest version of AviX which is yet to be released. Also, they need suggestions on a way for working with ergonomics for their projects within the company. 1.2. Problem Definition There are many ways of determining human movements to study and understand the motion of a person. Out of these existing methods, a suitable one should be selected so as to fetch the data quickly and in a simple manner. The method used to determine the human motion itself should not act as a hindrance to any movements taken by the person. Moreover, companies might not have access to all of the tools existing in the market to perform this study. So, consideration has to be given to what is available at hand and currently being used and to those methods that require minimum investment. After this information is obtained, one should be able to store it in a software which is used for production study. AviX is one such production tool that is used to study the processes that take place in a production line and optimize them to yield better results. An investigation is needed on how the data can be transferred to AviX software such that the posture data along with the number of steps taken in the processes can be automatically generated without the need for entering them manually. A second consideration is to establish a method for the company to work with ergonomics in their future projects. At present, the company does not have an existing method to work with ergonomics. So, a study has to be done to find the most suitable method for the company to use based on their customers’ requirements. Hence, by using this method the company will be able to conduct ergonomic analysis and find ways to improve the existing conditions of the workers. This method of working with ergonomics should generally be applicable to suit most 2 working situations because of the various possible environments. 1.3. Purpose The need to develop a methodology to make it possible to analyse human motion by estimating the pose of a person is to establish a simple method that can be used for many situations in a factory. Using this method, the data regarding workers’ movement can be found and computed to study the ergonomics of any workplace. To make this data easily accessible, it is required to be put into a commonly used software that handles information related to production. AviX is one such commonly used tool and importing data into this software will make it much easier to study production. A method of working with ergonomics is needed to be referred to when analysing the ergonomics of any workplace. Hence, it is important to establish a method that is suited to fit the majority of working conditions. 1.4. Aim The aim is to find a method to transform the data concerning human movements into Avix software and to also make suggestions for a way of working with ergonomic evaluation. From this project, the expected outcomes are as stated below: ● A methodology for making use of the AviX software to store posture data, and give information on harmful poses obtained from the pose estimation ● A way of working with ergonomics to show Virtual Manufacturing Sweden AB how posture data can be used to analyse the work done in their Assembly and Production Flow department. 1.5. Limitations The data collected during the pose estimation phase of the project is strictly focused on human movements for ergonomic evaluation. Not all features of the human body are taken into consideration in this project for the pose estimation. Head, Arms, Hands, Shoulder, Facial expressions, fingers and toes were not focused on while estimating the human pose. The data transfer is done only to AviX, and not to any other software. 3 2 Frame of Reference Ergonomic standards in industries are practices that improve and promote the productivity of the workers, along with maintaining their health and well-being (Ergonomic Standards, 2020). These are standardised procedures that allow the company to keep a track of the physical and mental stresses that are exerted on a worker and restrict them under certain limits that the company has set. Companies prioritize good ergonomic practices because they are beneficial in reducing the financial expenses that the company incurs due to worker absence and their ill- health. Now, ergonomic evaluation is another part that is vital to this thesis. Ergonomic evaluation is the assessment of the work tasks performed by an operator, along with the tools used and the environment the tasks are set in (Ergonomic Assessment, 2020). Ergonomic evaluation allows for the identification of work tasks that are harmful for the operators and to take remedial action. There are many ergonomic evaluation tools that are available, depending on the kind of work that needs to be focused on. 2.1. Ergonomic Standards As the access to ergonomic standards of various companies is limited to the public, we were provided only with the ergonomic standards that our thesis company had access to. These ergonomic standards are listed below: 2.1.1. Volvo Group Ergonomic Memorandum Volvo Group has developed an ergonomic memorandum to promote a “stimulating and healthy work environments for its employees” (Brault et al., 2013). The ideal work environment, according to this memorandum, is that it must be adapted to be accessible by a wide variety of employees, it must be adapted to repetitive tasks and frequent handling of parts, the equipment present must be easy to operate, it must be designed to increase the efficiency of work, and finally it should not pose any physical health risks to the employees. Although this memorandum is not an analysis tool for ergonomics, this can be used to compare the current workstation design with that of the ideal work environment explained in this book. The risk factors along with their risk scores and the recommended values are a good way to design a new workstation or re-design an already existing workstation with room for improvements. 4 The different aspects of a work environment covered in this memorandum are: Workstation accessibility, Anthropometric data, Dimensions of workstation, Posture, Stress, Workstation usage frequency, Movements, Material-handling aids, and Logistics. The purpose of this memorandum is to help Volvo Group in improving the quality of work of the employees, along with the overall quality of the product. This also improves the health and safety of the employees, thereby avoiding any unnecessary costs and losses to the company. 2.1.2. Volvo Group Ergonomic Standard The version of Volvo Group Standard that is referred to in this thesis was developed in March 2009 (Volvo Group, 2009). The document involves the scope of the ergonomic standard and its fields of application. It includes working postures and how the working environment affects a human body. The machines and tools present in the working environment and the kind of effects the load of these tools have on the human body are explained in-depth. The standard also includes details on how the sequence of work tasks along with work pace affect the ergonomics of employees. This document interprets the term “ergonomics” as the ergonomics to prevent the occurrence of Musculoskeletal Disorders (MSDs). It also stresses the fact that work environments should be designed to stimulate personal development of the employees. This standard describes that to achieve personal development, the operators need to be able to rotate through jobs/work tasks. In this document, any factor of a work task that is said to be strenuous on the body if carried out for a long time has to be avoided. Here, a long period of time is if the length of the work task either takes up more than 2 hours in an 8 hour workday, or is done frequently like more than a 100 times in a workday. The most optimal work posture is when the loads are acting on the body when it is in a “neutral” position. Work postures include bending of the back, twisting, lifting of the upper arm, position of the wrists, angle between body and upper arm, etc. The standard prohibits any kind of activity that requires the operator to squat, kneel or work on one leg for extended periods of time. Work movements include walking, pushing/pulling, climbing stairs and/or inclined surfaces, etc. A good working movement is one which allows for the muscles of the body to be relaxed and the joints to move freely. Small recovery breaks are important for the muscles to relax from the strains of work tasks. The standard also prohibits any kind of rapid movements of the body, repetitive work tasks, or highly monotonous work. 2.2. Ergonomic Tools There are many ergonomic assessment tools that are available to perform an ergonomic analysis. One tool that has an appropriate structure to make use of posture data is the REBA 5 ergonomic evaluation tool, making it a good choice to pursue. The company, Solme AB, has also decided to include this tool in the latest release of the AviX software. Each task entered into AviX can be attached with a REBA analysis and this makes it easy to analyse every task and make improvements accordingly. During our study of ergonomic evaluation methods, we came across the RAMP ergonomic evaluation tool which also makes use of posture data to a certain degree. It is a free ergonomic assessment tool developed at KTH Royal Institute of Technology, Sweden. This tool takes in a lot more input data when compared to REBA, and makes a holistic assessment of the work that is done. Both of these ergonomic tools that were considered for analysing the ergonomic aspects of the human operator are described below: 2.2.1. Rapid Entire Body Assessment (REBA) Rapid Entire Body Assessment, as the name suggests, is an ergonomic analysis tool that is simple to use to “rapidly” assess a task or an activity to check for risks of musculoskeletal disorders (Hignett & Mcatamney, 2000). A wide range of activities/tasks and postures can be evaluated for risks using REBA (A Step-by-Step Guide to the REBA Assessment Tool, 2017). A major advantage of using REBA is that it divides the body into different sections that can be analysed individually with respect to the posture taken by that particular body section within a work task. This makes the evaluated result more holistic and accurate. It allows for an ergonomist or a production planner to improve a work task based on a part of the posture that seems to be harmful. The final result is the REBA score obtained along with an associated level of urgency. A low score signifies that a particular task is good with no considerable risks, while a high score is an indication that urgent action must be taken to improve that particular work task. The REBA score is from a scale of 1 - 15. The assessment is split into two sections: A and B. In section A, the assessment is started by analysing the positions of the neck, legs and trunk. There are additional score points if there is any twisting or bending of these parts involved, and also if there is any load lifting involved. In section B, analysis of the arm and wrist is carried out. The positions of both the upper arm and the lower arm are analysed. A coupling score is added to the combined score of the arm position. The coupling score is for the hold or grip that the operator has during a particular activity. If the hold position is poor, points are added to the score. Scores obtained in sections A and B are then compared on table C. A corresponding score is obtained. An activity score is added to the score obtained from table C, that signifies if a particular activity or posture is repetitive. This is the final REBA score. The main limitation of the REBA method is that it does not consider time, neither the duration of the work task nor the time the operators get in between work tasks. It can analyse only one person at a point in time. But the advantages far outweigh the limitations, because of the simple understanding and use of this method. It does not require extensive skill or knowledge 6 of any particular software or tool. The harmful risks of MSDs are identified and the respective work tasks can be prioritized to be improved. 2.2.2. RAMP Risk Assessment and Management tool for manual handling Proactively, or RAMP as it is abbreviated, is an ergonomic risk management tool developed at KTH, i.e. the Royal Institute of Technology in Stockholm, Sweden (Rose & Lind, 2017). One of the main reasons we chose to do a literature study on RAMP is due to its holistic approach towards ergonomic analysis. It considers both the upper and the lower extremities of the body. The analysis can be done by people with basic knowledge of ergonomics. The results are comprehensive and can be understood by common laymen. But most of all, it is a standard tool (or method) that can be accessed easily and is free to use. The main purpose behind the development of this tool is to make a comprehensive study on the physical ergonomic risks that occur during the manual handling of parts, especially in the logistics, transport and manufacturing sectors. The main sequence of the risk assessment is: identification of risks, analysis of the risks, action to eliminate or mitigate those risks, and finally a follow-up plan to keep those risks in check by making sure to stick to the action plan. There are four modules involved in RAMP: 1. Two assessment modules a. RAMP I b. RAMP II 2. Results module 3. Action module RAMP I is an initial risk identification and assessment method. It involves a checklist to assess the instances of potential risks in the following areas of work tasks involved in manual handling: a) Postures, b) Work movements and repetitiveness, c) Lifting work, d) Pushing and pulling work, e) Influencing factors, f) Reports on physical work, and g) Perceived physical discomfort. The assessment is presented with three different risk levels, namely: High risk, Investigate further, and Low risk. ● High risk activities are ones that have a high probability to cause MSDs. Improvement measures are to be given highest priority. It is colour-coded red. ● Investigated further are activities that need to be analysed more in-depth. RAMP II is recommended for this. It is colour-coded grey. ● Low risk activities are ones that pose no immediate threat of MSDs. Improvement measures can be developed, but not prioritized. It is colour-coded green. 7 RAMP II is a more in-depth analysis and assessment method. It allows for a deeper analysis of the work tasks based on the same checklist that RAMP I utilizes. The assessment is again presented in three different risk levels, namely: High risk, Risk and Low risk. ● High risk activities are those that pose heightened risk of employees developing MSDs. Improvement measures must be given immediate priority. It is colour-coded red. ● Risk activities are those that pose risk to certain employees of developing MSDs. Improvement measures must be undertaken. It is colour-coded yellow. ● Low risk activities are those that pose no risk of MSDs, apart from those employees whose physical capacity is low. It is colour-coded green. The Results module is designed to act as an effective means to convey the results of the risk assessment. This usually can include any one or all of four levels of scope: 1. Single workstation or job 2. A department 3. A site 4. A company This can also be done in various levels of detail. For example, a high level of detail with reports of all risk factors being generated, or one where only the risk factors are presented with their respective risk levels and are colour-coded. The Action module is a method designed and generated to help support the change in work tasks that have been assessed to be ergonomically harmful. The action module is made up of three parts: I. An action module that helps by providing support to develop the actions or suggestions in the areas of Employees, Organisation, Technology & Design, Vision & Strategies, and Environment. II. Action suggestions are presented for the factors that are colour-coded red or yellow, based on the above mentioned areas. III. Finally, a template is presented for the preparation of an Action plan. In this plan, information regarding the planned action steps to improve the work tasks, the responsibilities that are required to be completed, time schedules for the activities, etc. are included. In RAMP I, the questions in each of the seven areas of work are simple yes/no questions. There are certain conditions that determine if the answer is either yes or no. In the results sheet, all these questions are presented as work factors and are colour-coded to show if they pose a 8 threat of MSDs or not. At the end of the sheet, a summary of the results is presented with the number of activities in each colour/risk category. In RAMP II, a more extensive analysis is carried out. This enables the ergonomists or the evaluators to be more precise and thorough while developing their improvement plan of the risk activities. The factors are more in-depth in the same seven areas of work tasks. The posture of the body and the external factors affecting it are analysed to a very high extent. The questions in this module are not completely of the yes/no type. The questions involve more educated and precise answers with risk scores, body twisting, force exerted while pushing or pulling, etc. 2.3. AviX AviX is a computer software that is used to analyse the work done in the workplace and optimize them to maximise the productivity (https://www.avix.eu/). This software has six modules that can be used to analyse the production processes. They are given below as follows: ● Method & Time Study ● Resource Balance ● FMEA (Failure Mode and Effects Analysis) ● SMED (Single-Minute Exchange of Dies) ● DFX (Design for Excellence) ● ERGO (Ergonomics) By making use of all these six modules it is possible to find out ways in which the current work scenario can be improved. In addition to the analysis part, AviX also has the ability to generate work instructions and reports that act as a reference for the operators performing the task. The latest version of the software yet to be released will have the REBA ergonomic assessment built into its ergo module. All tasks that take place in the production process are first created and entered into the software in the form of a tree layout. There can be many workstations or operators who are assigned certain tasks. So instructions for that particular workstation or operator can be generated. One can also add the parts and tools used in the production process into the software giving more details on the work that is done. In this way this software provides all the features necessary for analysing a production process. These features make it easy to understand the current situations in the production process and improve upon them. Virtual Manufacturing Sweden AB uses this software in their production related projects to perform analysis and make improvements. 2.4. Pose Estimation Pose estimation is the concept of tracking human movements and converting them into useful information. This information can be analysed to find out if the workplace or work is ergonomically good. The main posture data that is concerned with our thesis is the Step Count and Back Bend Angles for which pose estimation serves as a really useful tool. The Back Bend 9 Angle data is used to fill out a part of the already existing ergonomic evaluation tool built into AviX which is REBA. The same data from the pose estimation stage can be used to fill out a part of the RAMP assessment concerning the bending of back. There are different ways by which the pose of a human can be estimated with the tracking of the joints. Such ways of tracking dealt with in this project are: ● Tensorflow - Openpose estimation ● Xsens MVN analyse ● HTC Vive with Unity The applications of pose estimation in general are as follows: ● Study of human activities to find out the movement of a person involved in an activity ● In movies and video games to give motion data to a virtually created character ● In industrial applications to convey the required path information to the robot or machine 2.4.1. Tensorflow - Openpose estimation Human pose estimation is the process of fixing a set of coordinates on the joints of the human body and establishing the connection between them in order to retrieve the actual posture of the person as seen in the captured data (Raj, 2019). Here, each coordinate represents a joint and the connections between these coordinates represent the limbs of the human body. In this way, a virtual human skeleton is created over the actual person in the image which tracks their movement. Each joint, after being detected, is associated with a 2D vector which determines the position along with its orientation in the image (Cao et al., 2017). These 2D vectors are determined at pixel level and are collectively known as Part Affinity Fields (PAFs). By making use of these PAFs, it is ensured that the connection is only between the joints belonging to the same person. Hence, this prevents wrong estimation by connecting the joints belonging to two different people. This estimation can be done in real time as well from an existing video or image file. There are two approaches towards estimation of the human pose, namely: ● Top-Down ● Bottom-Up Top-Down approach This approach starts by detecting individual people in the image or video file (Raj, 2019). Each person in the file is considered separately and then the algorithms run to identify the joints and associate the coordinates on them, after which these coordinates are connected and the pose of the person is determined. A bounding box is created for each person detected and further operations to determine the body parts and their connections are performed considering just the image within these bounding boxes. This method has the risk of not establishing the pose of a person if the initial detection step misses out a person in the image. 10 Bottom-Up approach In this approach, the joints of all the people in the image are first found and associated with coordinates (Raj, 2019). After this, the connection between these coordinates is determined by algorithms to establish their connection appropriately which results in the determination of the human pose as seen in the file. This estimated connection is done by making use of the Part Affinity Fields as mentioned before to avoid the errors in representing a wrong human skeleton (Cao et al., 2017). Confidence maps are used to locate the required joints or parts of the human body in the image as they detect the probability of the presence of a particular body part in the pixel. The tensorflow pose estimation method makes use of the movement data concerning the joints of the human body as shown in Figure 2.1. This method does not require any trackers placed on the human body. The person can perform the work task freely and is not restricted by any external devices as in the other methods. This method makes use of python scripts to create a virtual human manikin that follows the motion of the person being captured. This script contains the operations to be performed in order to draw the virtual skeleton over the human. Here, each joint of the body has a number given to it. By modifying the python script, one can get the coordinate values of the required joints. These output values are generated on an excel sheet. From this, a python script developed by us is able to utilize the data concerning the coordinate positions of the joints and give the required count of steps and bends. Figure 2.1: Openpose joint numbers Source: https://github.com/CMU-Perceptual-Computing- Lab/openpose/blob/master/doc/output.md [accessed 15 May, 2020] 11 To find the steps the joint numbers 10 and 13 are taken into consideration as they correspond to the right foot and left foot respectively. Whenever the relative distance between the feet exceeds 50 cm, a step is counted. This will happen continuously to the end of the work task. In this way, the steps made by the operator can be registered. To find the bends made by the operator, the points taken into consideration are joint numbers 1, 8 and 11 which correspond to the neck, right hip and left hip respectively. Here, the midpoint of 8 and 11 is found to locate the actual hip position. The relative distance between the joint number 1 and the midpoint is first calculated horizontally and then vertically. The division of the horizontal distance by the vertical distance is first computed. Then the tan inverse of this computation is done to find out the angle of bending. Whenever this value exceeds a predetermined maximum bend allowance a bend is counted. In this way, the bends made by the operator can be registered. By using the two concepts stated above, it is possible to find the steps and bends made by the operator without the use of any external sensors and just capturing a person’s motion by means of an RGB camera. 2.4.2. Xsens MVN analyse The Xsens system makes use of inertial sensors to track the movements of the human. Here, the sensors are attached to the various segments of the human body to track the movements. For this method, there is no external camera used to record the person performing the movements. The movements are tracked by making use of inertial sensors which transmit the data to a USB hub connected to a computer. In this way, the motion is tracked and transferred to a humanoid manikin within the software according to the body measurements of the person under study. From this information, the movement of the entire human body is tracked and it can be analysed further. Inertial sensors According to What are Inertial Sensors? (2019), inertial sensors, also known as accelerometers, are components that make use of a spring mass system and capacitors to compute the various movements. There is a spring mass system connected to a set of plates in a capacitor like setup. Whenever there is a movement, the mass within the system moves and the plates are also deflected as a result. Now due to this deflection, the capacitance is varied and this information is used to compute the acceleration. One system consisting of a spring mass system and a capacitor exists for each direction namely x, y and z. The capacitance variation in each of these directions is then combined and analysed together thereby, giving the final position of the concerned body part after the movement. Hence, the data regarding movement is transmitted from each sensor to the USB hub connected to a computer and then this data is combined to transfer the movement of the person to the manikin in the software resembling the body measurements of the person under study. 12 Output The output obtained from this system is a series of x, y and z coordinates for each body segment. This data can be exported to an Excel sheet where further computation can be done to analyse the motion data, therefore making it easy to track and ensure ergonomics in the workplace. 2.4.3. HTC Vive with Unity This system makes use of trackers to interface with objects in virtual reality. This makes it possible for a person to work in a virtually created workstation and analyse the work being done early on. In this method, base stations are used to scan the environment in order to find the trackers. These trackers can be placed on the human body and the relative motions can be studied to understand the movement and ergonomics of the worker. Tracking In the beginning, one of the base stations flashes Infrared light which lights up the controllers and trackers (Buckley, 2015). Then a plane horizontal scan is done which locates the position of the tracker in the horizontal plane followed by another flash and a vertical scan to find the position in the vertical plane. From the intersection of these two planes a light ray is drawn from the base station to infinity through the tracked position. This entire procedure is done by the other base station as well. The point where the light rays from both the base stations intersect denotes the position of the tracker at that instance. This entire cycle is done many times per second. In this way, the HTC Vive makes tracking of objects possible in real time. Unity Unity is a game development software that can be used for the purpose of creating interactable game objects for virtual reality applications as well. For our purpose, this software can be used to create virtual workstations with interactable objects that resemble a real workstation with various tools and components. This software makes it possible to display the relative position of the trackers which can be analysed further to track the human movements. Hence using the HTC Vive system in combination with the Unity software yields the data required for human motion analysis. 13 3 Methodology The different methods for estimating the pose of a person are described below. The way in which the desired data concerning steps and bend angles are obtained and stored is also explained. Here, the process of using the output of the pose estimation phase to fill out a part of the ergonomic assessment in the AviX software automatically is covered. Information regarding the file that AviX makes use to hold the scores of REBA assessment is given. The sample work task that was created by us in order to test the methods is also dealt with in this chapter. This sample work task is evaluated by using RAMP and REBA. The method of evaluation using these evaluation tools is discussed briefly. 3.1. Data Collection One aspect of this thesis was to understand how AviX could be automatically fed with the data from the pose estimation results. To get information on how the software stores information regarding the user created tasks, we set up an interview with O. Ljung & E. Emanuelsson (Personal interview, March 12, 2020) at Solme AB, the company that created AviX. As a precursor to ergonomic evaluation, we wanted to get an understanding as to what constitutes good ergonomics within a workplace. For this purpose, Virtual Manufacturing AB shared with us the Memorandum and the Standard of Volvo Group. This information was the basis on which we carried out the developmental work. 3.2. Experimental Setup In order to test the working of the methods used to handle information from the pose estimators and also to verify if the desired part of the ergonomic assessment concerning the back bend is being done correctly, an experiment was conducted. This experiment is a made up of a sample work task which was created by us in accordance with the type of work dealt with in the Assembly and Production Flow department projects at Virtual Manufacturing Sweden AB, solely for the purpose of testing. The created workspace is shown in Figure 3.1. 14 Figure 3.1: Created work area The various tasks associated with this assembly process are given below: ● Pick up plank ● Place plank on product ● Pick up welding tool ● Weld plank joints ● Return weld tool ● Pick bearing ● Place bearing into socket ● Pick up screwdriver ● Screw bearing-socket unit ● Return screwdriver ● Move product to finish area The time for the example work was 49 seconds. The trackers were placed on the neck, pelvis and feet body segments. After this, the experiment was carried out to test the real-time pose estimation of the person performing the assembly 15 tasks. The tasks were also created in AviX software and the dictionary file was exported and kept ready. In this way, the experimental setup was done and then it was tested with some runs to verify the results. 3.3. Pose Estimation Data Computation The methods by which the human pose can be estimated while performing this experiment are given below. A detailed description of how each system works in order to yield the corresponding information pertaining to the steps and back bends is also given. The tools used for estimating the pose of a person were: ● Xsens MVN analyse ● HTC Vive System in combination with Unity 3.3.1. Xsens MVN Analyse This system can be used to track the basic x, y and z coordinates of the different segments of the human body as shown in Figure 3.2. The output from this system can be exported to a spreadsheet which displays the coordinates for all the different segments of the human body. The two main body segments focused here are the pelvis and the neck. Using the positions of these two body parts, the basic information about the human motion is found out. 16 Figure 3.2: Xsens body segment tracker positions The Xsens output gives a series of x, y and z coordinates for different segments of the body. To compute the data from the output coordinates, a python script was created. This script acts like a third party software that is used to take the values from the output of the pose estimation as its input and perform calculations in order to find out the required information. This code developed by us has various functions like step-bend computation, displaying output on a spreadsheet in Microsoft Excel and modifying a file that is used by AviX software that holds data concerning its tasks. All these functions are made possible by importing various modules of python that perform the intended function. The python script can be found in Appendix A. To find out the number of steps that a person has taken, the x and y coordinates of the pelvis are studied at each position. These x and y coordinates represent the centre of the human pelvis. From there the distance travelled by this point is studied and compared to the distance required to be considered as a step. The created python script reads these values outputted from the Xsens system. Here the formula resets the consideration point to calculate the relative distance. From this computation the total number of times a step is taken by a person can be detected. This is a relatively simple method as it makes use of only one body segment to determine the number of steps an operator takes. 17 An alternate method to finding the number of steps taken by the operator is to consider the feet segments of the body instead of the pelvis. In this way, the computation takes place such that the relative distance is horizontally calculated between the left foot and the right foot. This relative distance is compared and if it is found to exceed 50cm, then a step is counted. No further step is counted till this relative distance value reaches less than 50cm. Once it goes less than this value, the program resets its consideration point and gets ready to check for the condition in order to find the next step. This is done continuous till all the values have been taken into consideration and computed. The alternate method is a much more stable way of getting the required information pertaining to the steps of a person. To find the bends and the bend angles that a person makes, two body parts have to be taken into consideration, namely, the pelvis and the neck. Using trigonometry, the relative position of these two segments are studied. At first the difference between their z coordinates is found out. Then the distance between the position of these two segments is studied in the XY plane. Then the XY plane distance is divided by the difference between the z coordinates. From this resulting value, the arctan operation is performed to get the angle value at that instance. This angle value denotes how much the human body bends while working. An angle limit point is set and the computation leads to a count each time the bend angle crosses this angle limit point. Hence the number of times that a person bends can be found out from this computation. In this way, the steps and bends made by an operator can be found out and stored and documented in Excel automatically. 3.3.2. HTC Vive System in combination with Unity Using the HTC Vive trackers, the position of the human worker can be tracked. Here, combining the system with Unity game development software allows yielding the corresponding x, y and z coordinates which can be further computed to get the required aspects of the human motion. The Unity software uses C# scripts to establish relations to the various objects in a scene. This C# script holds the functions that is to be performed on either the trackers or controllers. By performing such functions one can obtain information related to the movements of a player or in our case an operator. A C# script is created to get the x, y and z coordinates of the trackers and perform operations on them to yield the steps and the bending details of the worker in real time. One tracker is placed at the pelvis area, the second tracker is placed either near the upper chest or the back of the neck depending on the user’s preference and finally two trackers are placed one on each foot of the person as shown in Figure 3.3. 18 Figure 3.3: Tracker positions To find the steps taken by the worker, the distance moved by the tracker near the pelvis region is taken into consideration as shown in Figure 3.4. The distance moved by this tracker is compared with the preset value for a step. If the distance moved by this tracker exceeds the distance for a step, then a step is counted and displayed on the screen according to the C# script written. 19 Figure 3.4: Step computation with trackers An alternate method to find the number of steps taken by a person while performing the work tasks is to take the two trackers placed at each foot of the person instead of considering the tracker at the pelvis as shown in Figure 3.5. Here, the relative distance between the feet is taken into account to find if they are at least 50cm apart from each other. Using this condition, a step gets counted each time the distance between the feet exceeds 50cm. In this way, even if the person were to run or walk fast the steps would be counted accurately. Figure 3.5: Alternate step computation with trackers To find the number of times the user bends along with the bend angle, the initial position of the second tracker placed according to the users preference near the head is first offset to bring it in line with the position of the first tracker. From here the relative distance between the trackers is observed. This observed relative distance is then divided by the height difference between the trackers. Then the tan inverse function is used to get the angle value of the bending motion. When this angle value crosses a preset limit, a bend is counted and is displayed. The trackers that are taken into consideration and the illustration of computing bends is shown in Figure 3.6. 20 Figure 3.6: Bend angle computation with trackers The script also writes this data automatically to an Excel sheet. In this way, the steps and bends made by an operator can be identified and stored and documented in Excel automatically. The C# script can be found in Appendix A. 3.4. Accuracy Testing The trackers are worn by the person and the person bends to a predetermined bend angle by manual observation from another person. The bend angle value that the tracker shows at that instance is noted. Then the person bends beyond the current value up to a certain point and comes back to the predetermined bend angle value. Now the value shown by the tracker is noted again. This is repeated for different bend angle values and the readings are noted. After all readings are noted, the maximum deviation is calculated from the Excel sheet to get an idea of how much the tracked angle can differ from the actual angle. 3.5. Storage of Collected Data The pose estimation data regarding the steps and back bends is collected by using the methods as mentioned in chapter 3.2. The computation is done by using the Python script developed by us in the case of Xsens system and C# script in the case of the HTC Vive System which makes use of the concepts described in the pose estimation data computation section of this chapter 21 to find the number of steps and the information regarding the bend of his or her back while performing the work task. This information regarding the number of steps and the back bending data of the operator is stored in an Excel sheet. This gives time specific information regarding the data obtained from the pose estimation technique. From this data, the total number of steps, the number of bends and the maximum bend angle of the operator during work can be seen. 3.6. Ergonomic Evaluation The ergonomic evaluation tools tested in this thesis work are RAMP and REBA. These two tools are used to evaluate the ergonomics of the work that is done in our experiment which is described in Section 3.1. The reason for us selecting the above two methods was that we found them very suitable for the work tasks in the projects undertaken by Virtual Manufacturing AB in their Assembly and Production flow department. Hence, these tools were selected to evaluate the ergonomics of the work being done. 3.6.1. REBA The main motivation behind choosing REBA for the ergonomic assessment was that AviX is a tool widely used by Virtual Manufacturing Sweden AB in their projects. The REBA assessment will be made available in the yet to be released latest version of the AviX software by Solme AB. This assessment is done to evaluate the extreme postures of the work done in our experiment. The part of the REBA assessment that is automatically filled is the trunk position, which is done by making use of the pose estimation phase. The posture of the operator performing the work task is observed and the remaining part of scoring, which is not filled automatically from the posture data, is done by us to find out the final REBA score. The REBA assessment sheet is shown in Figure 3.7. 22 Figure 3.7: REBA assessment sheet (Cornell University Ergonomics Web, 2000) 3.6.2. RAMP The RAMP tool is used to assess the work being done on a scoring basis. Here, a part of the assessment pertaining to the back posture can be automatically done by making use of the pose estimation phase. The assessment itself is divided into two parts namely RAMP I and RAMP II. The classification of tasks into different areas is generated after the assessment is done. In this way, one can know how many tasks require immediate attention concerning ergonomics. The RAMP I assessment basically classifies the areas into red, grey and green. First the work task is observed, and the situations mentioned in the assessment sheet are evaluated to get the RAMP I assessment. The RAMP I assessment sheet is shown in Figure 3.8. Here, the second criteria under the heading 1.1 which is the back/ upper body – bent or twisted gets filled in automatically from the pose estimation results. At the bottom of the image, one can see the number of work tasks that come under the red, grey and green category. To investigate the grey areas further, it is required to perform the RAMP II assessment to obtain further detailed information concerning ergonomics. The RAMP II assessment, when done, reveals more information about the work tasks performed by the operator. Here, the assessment classifies the work tasks into red, yellow and 23 green areas. This assessment will show if the tasks, previously in the green area, end up in the yellow area and if there are more red areas that require attention. These changes happen depending on the criteria for the assessment. This assessment will reveal more information pertaining to the ergonomics of the work task. The RAMP II assessment sheet can be found in Figure 3.9. In this way, the RAMP assessment is done to find out the ergonomics of the observed workplace. 24 Figure 3.8: RAMP I assessment sheet (KTH Royal Institute of Technology, 2017) 25 Figure 3.9: RAMP II assessment sheet (KTH Royal Institute of Technology, 2017) 26 3.7. Data Transfer to AviX From the interview with O. Ljung & E. Emanuelsson (Personal interview, March 12, 2020), AviX has the ability to import data regarding the processes in a workstation by making use of a json (JavaScript Object Notation) file which holds the data corresponding to the production tasks involved in a factory. The form of data stored in a json file is basically a dictionary which maps the attributes of a process to its values. Each of these processes are given a unique identification by the AviX software. This identification has to stay the same for AviX to recognize which process parameter belongs to which process task. Hence the export ips file is an important step to be taken first. But in order to get the json file, the corresponding tree layout and the processes must be first created in AviX. Once the basic sequence of tasks is created, the entire data can be exported by using the IPS (Internal Patching System) function. This is done by right clicking on the workstation and selecting IPS > Export IPS file. This creates a json file in the specified location which contains the processes along with its parameters. In order to add the score for a part of the REBA assessment to this file, a python code is written to add this to the existing dictionary file. This score is obtained by making use of the results that were obtained from the previous pose estimation methods. In this way, a part of the REBA analysis data is automatically added to the dictionary file once the program is executed. After this is done, the newly modified json file is imported back into AviX and the ergonomic REBA analysis is shown along with its associated process. The AviX software automatically calculates the final REBA score depending on the input fed to the analysis. Therefore, by making use of this method it is possible to make AviX incorporate ergonomic REBA analysis automatically without the need for manual entry by the user. The first step for transferring data into AviX is to first create the sequence of tasks in the software. The second step is to export the created layout as an IPS file. The third step is to modify this exported file to include the ergonomic analysis parameters. The final step is to import this modified IPS file back into AviX, thereby automatically generating the ergonomic data for the processes. In this way, manual entry is eliminated, and the process is automated. The details shown in the Figure 3.10 are only for illustration purposes and they are not related to our experiment. The details pertaining to our experiment can be found in chapter 4.2 27 Figure 3.10: Sample tree layout AviX creates a unique reference ID for each of the tasks that are created in the layout. It is really important for the parameters to be referenced to their corresponding tasks along with these unique IDs. Once the tree is created, the workstation is exported as a json file by using the Export IPS file option. This creates a json file containing the information about the processes by assigning a unique ID to them as seen in Figure 3.11. Figure 3.11: Exported Json file From the results of the pose estimation, the ergonomic data is entered into the json file which is shown in Figure 3.12 by making use of a python code. After this addition of ergonomic data is done, the json file can be imported back into AviX by making use of the Import IPS file function. Once the file is imported, the ergonomic assessment can be seen in the AviX User Interface. 28 Figure 3.12: Modified Json file for import The python code developed by us contains the syntax that the software understands and reads the ergonomic scores from. In this way, the resulting data from the pose estimation is analysed to find out the ergonomic values which are then added to the json file in the particular format from which the AviX software can import the ergonomic assessment as shown in Figure 3.13. 29 Figure 3.13: AviX user interface after importing the Json file The trunk position value is selected by the code by referring to the corresponding bend value data in the Excel sheet. When the code computes the trunk position for a particular task, it refers to the timings of that particular task from the Excel sheet and takes the maximum value of the bend angle from that column within the user-defined timings for the task. Hence, the maximum bend angle information pertaining to the particular task is obtained and the trunk position is automatically selected during the assessment in the software. The reason for selecting the maximum value over the average value of the bend angle is because we needed to know the extreme positions for the particular task in order to find if corrections are required. Using the average value of the bend angle might hide the very short timed harmful postures. Therefore, to make a proper ergonomic assessment, the maximum bend angle of the back is required. 30 4 Results 4.1. Real-Time Pose Estimation The trackers were placed on the person performing the work task and the assembly process was started. The pose estimator was able to determine the number of steps taken and the bend angles of the back in real-time as shown in Figure 4.1. The information pertaining to the steps and bends are obtained simultaneously as the work tasks are being carried out by the operator. In this way, the tracking of human movements can be done in any given work scenario. The tracker positions are sampled 89 to 90 times per second. Therefore, we get more detailed info on the posture data throughout the process. Figure 4.1: Real-time Pose Estimation In the above Figure 4.1, the person performing the work tasks with the trackers on is shown on the right and the information regarding the person’s movements are shown on the left along with the number of steps and bend angles. This makes it possible to already identify key areas 31 that affect the ergonomics of the work-space. Hence, the real-time pose estimation data was obtained while performing the actual work itself. From this information, it becomes easy to monitor the posture data of the person performing the work task. The graph in Figure 4.2 shows how the bend angles vary over the course of the entire experiment and that the working time is 49 seconds. Here, the graph shows the time the bend angle of the back is in the different zones classified as red, yellow and green. When the bend angle is in the red zone, this indicates a high bend angle for the back which implies a risk and should not normally be accepted. When the bend angle Is in the yellow zone this can signify a risk and these situations should be further investigated. The time that the bend angle is in the green zone, means no risk and is acceptable. Here it is possible to sum up the time the bend angle is in the three zones and relate that to the total time of the work task. Other types of analyses are also possible (e.g. accumulated and average time length in each zone, bend angle histogram). In this way, one can understand the bending curve showing the bend angle values of the person performing our experiment. Figure 4.2: Bend angle variation In Figure 4.3 the steps taken by the person over time while performing the work task can be seen as a cumulative graph. Each step in the graph line is one step taken by the person at that corresponding time. 32 Figure 4.3: Steps made From Figure 4.4, it can be seen that the step rate for this particular work task is 0.53 steps per second. It also makes it possible to find out at which instance the steps occur throughout the process. Figure 4.4: Step Rate graph 33 An interesting observation that was made here is the fact that the bend angles can have a large variation, 30 degrees in this case, over the course of just one second as indicated in Figure 4.5. This shows the importance of monitoring the process in milliseconds. The step count is discrete, but the bend angles change continuously during that particular second. Figure 4.5: Variation of bend angles and steps made within one second The jerks or disturbances shown in the graph were due to the fact that we did not use proper tracker attachments due to their unavailability in the company. We used velcro straps which were not ideally designed for holding the trackers. The actual straps suited for the trackers are available on the market which will reduce these jerks or disturbances. There are other issues that also play a role in creating these disturbances such as the alignment of the trackers on the person and the tracking field boundary. The accuracy testing of the trackers was done by comparing the differences between manual observation and tracker values as shown in Table 4.1. 34 Table 4.1: Accuracy Deviation 4.2. Data Transfer to AviX When the real-time pose estimation is being done, the data pertaining to the steps and bends is being written simultaneously in an Excel sheet. This Excel sheet contains the data in an appropriate format which gives the step count and bend angle at each second of the pose estimation phase as shown in Table 4.2. The Bends column gives info on the number of times the person has exceeded back bending more than 15 degrees which is the recommended maximum bend angle in the Memorandum of Volvo Group. 35 Table 4.2: Steps, Bends and Bend angles with time While running the code, the times at which each of the tasks end are entered and the analysis begins. Based on these entered time values the ergonomic analysis takes place by taking the posture information at various times from the Excel file of the entire assembly process and individual REBA scores are created for each of the tasks. This is added to the modified dictionary file which AviX uses to store information about the tasks. After importing the modified file into AviX, each task is attached with its corresponding REBA score respectively as shown in Figure 4.6. 36 Figure 4.6: Result after AviX import In this way, the data transfer to AviX is made possible and automatic ergonomic assessment with high accuracy is ensured. 4.3. Ergonomic Evaluation The ergonomic evaluation of the entire task described in Section 3.1, considering the extreme position of the person performing the assembly process, was done using the two tools namely REBA and RAMP as mentioned in Section 3.5. Both of the evaluation methods were performed on the same assembly process. 4.3.1. REBA The results of the REBA ergonomic assessment are shown in Figure 4.7. 37 Figure 4.7: REBA assessment of worst posture in the assembly process The trunk position is automatically filled in by using the results from pose estimation. The other posture options apart from the trunk position are filled in manually. But with use of additional trackers and the same method for automatic data transfer, it should be possible to automate the remaining posture options as well. The final score of the REBA evaluation shows that there are medium risks in the process that require investigation and change. In this way, the REBA assessment can be used to determine if the process has minimal risks. 4.3.2. RAMP The RAMP analysis done shows the particular areas of the process to be addressed. This evaluation takes more information into account as input for the evaluation. The back-posture information in the RAMP assessment is readily obtained from pose estimation. But the other fields in the sheet are filled manually considering the operator’s work tasks for the entire day. The results of RAMP I are shown in Figure 4.8. 38 Figure 4.8: RAMP I assessment of assembly process From RAMP I, there was one red area pertaining to repetitive work. There were 7 grey areas which needed further investigation and hence RAMP II was also performed. The results of RAMP II are shown in Figure 4.9. 39 Figure 4.9: RAMP II assessment of assembly process Here, upon further investigation with RAMP II, it can be seen that there are actually two more red areas which need immediate attention than that indicated in RAMP I. This analysis gives details on where the risk areas are identified so that appropriate action can be taken. 40 5 Discussion The discussion regarding the results obtained and the possibilities that our method opens for the future is given below. 5.1 Obtained Results The data obtained from the pose estimation phase is a result of direct measurement, thereby giving values of posture data in real-time. The trackers used for the pose estimation phase create almost no load on the operator. In the beginning, we tested out the bend angles by noting down the value shown from the trackers and then manually measuring the angle of bend while the person retained the same bent position. The accuracy of the tracked bend angle was ± 2 degrees. One factor affecting the accuracy was the attachments used for holding the trackers in place due to the absence of the required resources. There are other factors also such as the alignment of the trackers with one another which can contribute to inaccuracies in the measured angle. Working in the boundary areas of the workspace can also yield inaccurate measurements if one of the trackers goes outside the tracking field. This pose estimation data does not need further analysis in terms of data acquisition and is readily converted into posture data by means of our python code. This saves a lot of time in production when one wants to assess the postures occurring at a workplace during a work task. Moreover, this data gets directly incorporated into AviX which saves a lot of time. Now, our method enables the automatic recording of body movement and posture of multiple tasks that are created in the AviX tree layout based on the pose estimation data. This is a great advantage compared to the conventional method where a person assesses each work task and enters the resulting scores manually. The important thing to be cognizant of for our method is that the film used in the AviX software should be the same one that was obtained during the real time pose estimation. Hence, it is necessary to film the operator performing the work with the trackers on. This small step enables getting the ergonomic score from the real time pose estimation results concerning posture data to be transferred to the AviX software easily. An ergonomic evaluation tool was also suggested in order to study the entire assembly process as a whole. The extreme positions of the operator were considered during the assessment. REBA is an entirely posture-based method and it was found that the REBA analysis gives quite a 41 good understanding of the postural risks at a workplace and a part of it can be automatically done by the use of our method. RAMP on the other hand also assesses posture, but it takes much more details into account and gives a holistic approach to ergonomic assessment of the workplace. The RAMP assessment is done manually but can benefit from the results of the pose estimation phase. Here, based on the results of the RAMP assessment, it is possible to pinpoint the exact area that needs to be addressed in order to improve the workplace ergonomics. Currently based on our literature study phase and the method used, we found REBA and RAMP ergonomic assessment tools to fit in well with our thesis work. It is up to the company to take into account the requirements of their customers and choose an appropriate ergonomic evaluation tool for their intended purpose. 5.2 Future Work The entire pose estimation is computer based, which is faster and more accurate than the manual observation in order to fill out the ergonomic assessment. With the incorporation of more trackers, the current pose estimation method can be extended to other parts of the body. Our idea is that additional trackers can be placed on the various joints of the body such as the wrists, knees, shoulders and even on the fingers by means of a tracking glove. One problem that we see while implementing this is worker comfort as the trackers used in HTC Vive are quite big for small joints of the body. If in the future there is a new design for these trackers, then it might make it comfortable to wear for the operator. Doing this can extend the information obtained during the pose estimation to other aspects such as twisting motion, wrist position and leg position. Also, with this information there is a possibility of making multiple ergonomic assessments with different ergonomic tools at the same time. The computer gives information about the posture in terms of quantitative values which are more precise than the traditional method of manual observation. Using this information one can evaluate the design and layout of the work area with respect to ergonomics and make the appropriate changes if required. By making use of our method, the documentation of posture data can be done with ease as the information can be simultaneously recorded in real time by the computer. This can help decision making to a large extent. If the tracking system is comfortable enough for the operator such that it can be worn throughout the day, then studies related to repetitiveness and duration can also be conducted. This will give a better result as the measurement is directly done by the use of the tracking system in comparison with the manual observation or entry. Moreover, this estimation is possible in real-time as the work is being done, eliminating the need for post processing. Using our method, it is possible to take the factors into consideration as described by two researchers: 42 An appropriate job exposure assessment should consider all tasks (including breaks) constituting the job. We thus suggest a thorough description of job exposure to comprise quantitative assessment of (1) 'task distribution', i.e. occurrence and duration of the different tasks included in the job; (2) 'task exposure', i.e. exposure of the different body parts due to each task. (Winkel, & Mathiassen, 1994, p. 983) Our method can also be extended to favour biomechanical calculations of the human body in the workplace. The measurement of actual forces that occur in the body cannot be measured directly. It is only possible to measure these forces indirectly by establishing certain relationships linking various other factors to the forces that might be generated in the body. The weight of the object being lifted and the posture of the person while lifting this weight can be taken into account in order to establish a relationship between these factors and the forces generated in the human body (Örtengren, 1989). An example would be to find the velocity at which a particular tracker is moving to calculate the acceleration. After this is done, the entire mass of the body part in motion can be multiplied with its acceleration in order to find out the force exerted on that body segment. So, it is very important to consider the weight of the body part that is being subjected to loading to get the accurate results. This can be done by having some sort of understanding regarding how the total weight of the body is distributed to these body parts, thereby giving the necessary data to carry out the computation of forces. With the data obtained from the real time pose estimation in our thesis work, one can get the global reference angle of the Torso in the low-back biomechanical model as mentioned in Chaffin et al. (2006). In the future, with more trackers, this can be extended to get the remaining exterior angles mentioned in the model. With this information, the method mentioned in the book can be used to measure the compression force acting in the spine to overcome the load that is being lifted. This value can then be compared and verified if it is within acceptable limits using the NIOSH Lifting equation (Waters, et al., 1993). In this way, it becomes easier to perform biomechanical calculations. The moments acting on the back can also be ascertained in a similar manner when the forces are known. Hence, by analysing the posture data of the operator using our method, one can obtain useful information automatically which makes it easy to perform advanced calculations in biomechanics. Hence, from the results of the kinematic analysis, it is possible to compute information in order to perform a kinetic analysis. The Intelligently Moving Manikin software, also known as IMMA, already has the RULA ergonomic assessment function which can be modified based on the user’s requirement (Fraunhofer Chalmers, n.d.). This is done by making changes to the ergonomic function of the software through json scripting or creating a new functionality using LUA scripting (Bogojevic, & Söderlund, 2020). By doing so, new joints or user-defined functions can be added to the 43 manikins and tracked for ergonomic assessment. The information obtained from our pose estimation can be used to give the movement data to the manikins in the IMMA software. This can be done by making use of the positional coordinates of the person’s tracked elements moving in the workspace and feeding this data as input to the corresponding joints of the manikin in the software. By doing so, the manikin will be able to replicate the exact motion of the actual operator performing the work. This helps in the evaluation of the workplace and its ergonomics. In this way, human motion can be defined at first and the workplace layout can be designed in accordance with the operator’s movement. If it is possible to get the positional coordinates of the manikin’s body segments in terms of x, y and z coordinates, our method of estimating the posture will work to make an ergonomic assessment. 44 6 Conclusion The work done in this master thesis provides a method to make effective use of the REBA ergonomic assessment feature in the AviX software, which will be included in the software’s latest version. AviX is a prominent tool that is being used for production planning and assessment. So, by automating a part of an ergonomic analysis within the software, a lot of time and effort is saved. Initially, a study was made regarding the different ways of obtaining the data pertaining to human movements. During this study, it was found that motion tracking could be the most suitable option to get the required information. Based on the results obtained, the following conclusions can be made: 1. It is possible to quantify the number of steps and the back-bending angle in real-time with the help of motion tracking. 2. The AviX software stores information pertaining to the created task in the form of a json script. 3. When this json script is modified, the modifications are reflected back in the software. 4. A part of the ergonomic assessment tool in AviX can be automated by using the results from the pose estimation phase to modify this json script. 5. The REBA and RAMP ergonomics assessment tools benefit a lot from the automatic posture data transfer and continuous motion tracking. The first aim of the thesis was to enable the new release of the AviX software to store posture data and give information on the harmful poses during the pose estimation, which has been achieved. The second aim of the thesis was to show Virtual Manufacturing Sweden AB how the obtained posture data could be used to carry out an ergonomic analysis of the work done during their projects, which has also been described in this report. Our work provides the basis for an automated approach to human motion analysis with quantifiable posture data values in a work scenario. We hope that our method of real-time pose estimation opens up multiple possibilities in the future for assessing the movements of a person. 45 Bibliography A Step-by-Step Guide Rapid Entire Body Assessment (REBA). (n.d.). Retrieved April 13, 2020, from http://ergo-plus.com/wp-content/uploads/REBA-A-Step-by-Step-Guide.pdf Bogojevic, L., & Söderlund, H. (2020). Simulations of ergonomic assembly: Ergonomic evaluation using the digital human modelling tool IMMA. (Master's thesis, Chalmers University of Technology, Gothenburg). Gothenburg: Chalmers Reproservice. Brault, J., Gaget, L., Genest, M. L., Lundqvist, M., Bertin, C. M., Roybin, C., . . . Viossat, E. (2013). Memorandum Ergonomic Main Recommedations (2013 ed.) (948916689 740229705 C. Grosvalet & 948916690 740229705 M. Perrat, Eds.). Volvo Group. Buckley, S. (2015, May 19). This Is How Valve's Amazing Lighthouse Tracking Technology Works. Retrieved March 9, 2020, from https://gizmodo.com/this-is-how-valve-s-amazing-lighthouse- tracking-technol-1705356768 Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017). Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). doi: 10.1109/cvpr.2017.143 Chaffin, D. B., Andersson, G., & Martin, B. J. (2006). Occupational biomechanics (4th ed.). Hoboken, New Jersey: Wiley-Interscience. Cornell University Ergonomics Web. (n.d.). REBA Worksheet. Retrieved April 13, 2020 from http://ergo.human.cornell.edu/ahREBA.html Ergonomic Assessment. Gale Encyclopedia of Nursing and Allied Health. . Retrieved July 06, 2020 from Encyclopedia.com: https://www.encyclopedia.com/medicine/encyclopedias- almanacs-transcripts-and-maps/ergonomic-assessment Ergonomics Standards. (n.d.). Retrieved July 06, 2020, from https://webstore.ansi.org/industry/ergonomics Fraunhofer Chalmers. (n.d.). Intelligently Moving Manikin. Retrieved September 29, 2020 from http://www.fcc.chalmers.se/focus-areas/engineering-industry/imma/ Hidalgo, G., Fujisaka, Y. OpenPose Demo – Output. Available from: https://github.com/CMU- Perceptual-Computing-Lab/openpose/blob/master/doc/output.md 46 Hignett, S., Mcatamney, L. Rapid entire body assessment (REBA). Appl. Ergon. 2000, 31, 201– 205. KTH Royal Institute of Technology. (2017). RAMP - Risk management Assessment tool for Manual handling Proactively. Retrieved April 13, 2020 from https://www.ramp.proj.kth.se/ramp-risk-management-assessment-tool-for-manual-handling- proactively-1.730128 Ljung, O., & Emanuelsson, E. (2020). Personal interview, March 12, 2020. Örtengren, R. (1989). Biomechanical aspects on body loading during work and sport. In C. Edling & D. Brune (Eds.), Occupational hazards in the health professions (pp. 237-249). Boca Raton, FL, Florida: CRC Press. Raj, B. (2019, May 1). An Overview of Human Pose Estimation with Deep Learning. Retrieved February 14, 2020, from https://medium.com/beyondminds/an-overview-of-human-pose- estimation-with-deep-learning-d49eb656739b Rose, L., & Lind, C. (2017). User Manual for the Risk Management Tool RAMP. KTH Royal Institute of Technology, Sweden. Volvo Group. (2009). Standard Volvo Group. Volvo Group. Waters, T. R., Putz-Anderson, V., Garg, A. & Fine, L. J. (1993) Revised NIOSH equation for the design and evaluation of manual lifting tasks. Ergonomics, 36(7): 749–77. What are Inertial Sensors? (2019, January 21). Retrieved March 9, 2020, from https://www.azosensors.com/article.aspx?ArticleID=1559 Winkel, J., & Mathiassen, S. E. (1994) Assessment of physical work load in epidemiologic studies: concepts, issues and operational considerations, Ergonomics, 37:6, 979-988, DOI: 10.1080/00140139408963711