ABSTRACT
Greater understanding of how highly skilled operators achieve high machine performance and productivity can inform the development of automation technology for construction machinery. Current human operator models, however, have limited fidelity and may not be useful for machinery automation. In addition, while physical modeling and simulation is widely employed in product development, current operator simulation models may be a limiting factor in assessing the performance envelope of virtual prototypes.
A virtual operator modelling approach for construction machinery was developed. Challenges to the development of human operator models include determining what cues and triggers human operators use, how human operators make decisions, and how to account for the diversity of human operator responses. Operator interviews were conducted to understand and build a framework of tasks, strategies, cues, and triggers that operators commonly use while controlling a machine through a repeating work cycle.
In particular, a set of operation data were collected during an excavator trenching operation and were analyzed to classify tasks and strategies. A rule base was derived from interview and data analyses. Common nomenclature was defined and is explained. Standard tasks were derived from operator interviews, which led to the development of task classification rules and algorithm. Task transitions were detected with fuzzy transition detection classifiers.
RELATED WORK
Preliminary studies of operator modeling found in autonomous vehicle research helps to develop an understanding of how a virtual operator model can be used to improve autonomous control systems. Data collection, task analysis and human behavior study techniques can be combined to gather, organize, and represent how human operators perform certain operations, supported by their strategies, situation awareness, knowledge, and decision making process.
- Operator Modeling Approaches
- Task Modeling Approaches
- Autonomous Control
MODELING METHODS
Figure 1. The virtual operator model interacts with the vehicle model through a well defined interface.
To develop a virtual operator model that replicates human operator behavior, operator interviews were first conducted to understand the approach of operators and to collect information about their behavior. The modeling structure described in Figure 1 describes the elements of the VOM.
MODELING RESULTS
Figure 2. Task timeline based on human operator interviews. Task start and end times were calculated based on analysis of videos of trenching operations.
The tasks and sub-tasks of the trenching operation were identified from the operator interviews and are summarized in a nominal task timeline (see Figure 2). Five main tasks were identified within the trenching operation: bucket filling, bucket lifting, swing to dump, dumping, and swing back to trench. The timing of the start and end of each task was estimated through review and analysis of video of a trenching operation for one of the participants.
It was noted in both interviews and video analysis that the tasks overlap. Task overlap was a consistent theme among all participants – one participant said that the more expert the operator, the more he or she can overlap tasks to increase efficiency and reduce cycle time. While the video analysis of timing provided a qualitative estimation of task overlap, vehicle data analysis (described later in this section) was used to obtain more precise estimates of task timing.
Figure 3. Data Characterization with Task Timing Overlaid on Traces of the Operator Inputs and Machine Configurations.
The durations of specific tasks were overlaid on traces of the machine cylinder extension lengths, swing speed, and operator inputs (see Figure 3). The topmost graph of Figure 3 shows the cylinder extension positions for the Boom, Arm, and Bucket. The remaining graphs show operator control inputs for Swing, Boom, Arm, and Bucket. All these signals were used to identify the five tasks:
Bucket Filling, Bucket Lifting, Swing to Dump, Dumping, and Swing to Trench within two work cycles. Rectanglar bars with different colors were used to show the start time point and end time point of tasks with task names on them. Timing information of tasks could be directly read from the diagram.
Figure 4. Transition Detection Results between Swing back to Trench and Bucket Filling.
Figure 4 represents the transition identification result between Swing back to Trench and Bucket Filling. The green line represents the fuzzy classification result for the transition detection. The blue line represents the ground truth task for Bucket Filling, which describes when Bucket Filling starts and ends. By comparison of the traces, the correct detection happens when the green line starts to rise slightly ahead of blue line, since the goal of the classifier is to predict a transition between tasks. If the green line raises later than the blue line, the transition is detected late.
Figure 5. State Sequence derived from Fuzzy Transition Detection Classifiers to Represent the Transition Time to each Task.
By combining of all identified transitions within the trenching operation, the sequence of tasks and current state of the operation can be represented. Figure 5 visualizes the sequence of the tasks with information about when each task starts, which can be considered as a state sequence model. An accurate task sequence is important timing and state information for control signal generation.
DISCUSSION AND CONCLUSION
Virtual operator modelling based on studies of human operator perception, decision making, and behaviors can be used to improve development of autonomous and robotics systems and improve evaluation methods in virtual environments. It has a direct benefit for automating construction activities. Initial work focused on construction vehicle operation; future work will expand the application area to agricultural operations, using similar approaches.
The main challenges of developing virtual operator models are understanding the ability of human operators to adapt, modeling operator perception, and modeling operator decision making. This paper introduced methods used to collect data and identify information needed to start modelling. A fuzzy logic task transition model has shown promising performance in correctly identifying transitions between tasks. Expert behavior has in part been identified as the ability of operators to overlap tasks within an operation. Future work will include developing methods for identifying overlapping tasks, modeling overlap, and generating control inputs.
Source: Iowa State University
Authors: Yu Du | Michael C. Dorneich | Brian L. Steward | Eric R. Anderson | Lawrence F. Kane
>> Latest Vehicle Automation Projects for Engineering Students