Abstract

This study examined the use of a simulation in a human resource management course. 215 students took part in the simulation as a graded component of the course and their performance was compared against exam performance. Individual differences such as cognitive ability, personality traits and affect were also examined for patterns of validity. Results include evidence of discriminant and convergent validity on the two performance measures and also that certain characteristics of students predict success on exams, but not simulation performance and vice versa. This indicates that simulations engage students in ways different from the lecture and examination method. Further, results suggest that this engagement may then lead to improving exam scores over the course of the semester.

Introduction

A simulation is a type of instructional method that approximates a real life situation with students’ decisions resulting in outcomes that mirror what would happen if the participant were on the job. Most instructors using them believe that simulations enrich the learning experience and include a grading component based on simulation performance (Anderson & Lawton, 1992). Simulations have long been popular in management education, especially in strategic management courses where studies indicate about 50 percent of professors employ a simulation (Tompson & Dass, 2000). However, simulations have been developed for use in many different business school courses. For example, Interpretive Simulations (http://www.interpretive.com/simulations/) currently has six simulations that can be used across an array of management courses, including human resource management. Smith and Golden (http://home.att.net/~simulations/) have developed eight simulations for use in management courses, including the one examined in this study, Human Resource Management (1994). Thus, simulations are readily available for use in teaching management courses. However, there is some debate on the validity and overall veracity of this practice. This study examines these notions in a human resource management context.

All instructional programs should promote efficient learning, long-term retention of knowledge, and the application of the knowledge of factual information to the job situation (Cascio, 2005). A growing body of research suggests that simulations can be very effective training tools (Bell, Kanar & Kozlowski, 2008). In designing instructional programs (such as university courses) to incorporate principles of learning developed over the past century, instructors of more applied courses have searched for different ways to structure courses in a manner that parallels the job situation. Simulations have grown in popularity through the years, with simulations typically capitalizing not only on the intense environment spurred by competition between the students, but also by the objective nature of the feedback provided by the simulation (Tompson & Dass, 2000). That said, a firm relationship between the levels of student learning and participation in a simulation has been difficult to assess.

The relationship between student learning and participation in a simulation is difficult to establish, particularly if one is looking for a connection between objective test scores and simulation performance. Since simulations allow students to see the impact of their decisions in an artificial, risk free environment, some believe that the stakes are not high enough to produce results mimicking a realistic situation. For example, Thorngate and Carroll (1987) found that the effects of luck may be attenuated but not eliminated with the implication being that learning and simulation performance are unrelated (Washbush & Gosen, 1998). On the other hand, Keys and Wolfe (1990) concluded that games are internally valid for a strategic management course and Wolfe and Roberts (1993) have argued that simulation games have external validity in predicting future career success of players (Washbush & Gosen, 1998). Still others have found no relationship between simulation play and exam scores, level of simulation performance and exam scores, and recency of simulation play and exam scores (Wellington & Faria, 1991). Nonetheless, even this finding, which stands in contrast to the bulk of research results, is less condemning of simulations than it may appear. Simply put, there are substantial indications that even students that perform poorly in the objective sense of a simulation may still learn a significant amount (Gosen & Washbush, 2004). Still, evidence of the validity of simulations being related to other graded aspects of the course and/or student learning would go a long way toward solidifying the status of simulations as an appropriate pedagogical technique.

To examine the validity of simulations, the specific kinds of learning goals to which simulations speak should be identified. For instance, Whiteley and Faria (1989) concluded that simulations are effective in improving quantitative skills, but not in improving theoretical knowledge. However, others suggest simulations may assist students in learning theory as it strengthens the connections between theory and practice, encouraging students to elaborate the theoretical information in a way that is useful to them (Proserpio & Gioia, 2007; Zantow, Knowlton & Sharp, 2005). Beyond theory, simulations are excellent tools to develop decision-making skills. As compared to typical multiple choice exams, simulations are a harbinger of applied skills. Furthermore, case analyses and problem solving exercises are similar to simulations in that they require students to engage in critical thinking (Seaton & Boyd, 2008), a common learning goal across business schools. Finally, Kachra and Schnietz (2008) outline a seven-step process to help guide instructors on changing their courses to focus on decision-making since such a learning goal is perfectly suited to simulations. Thus, it seems that there are ample reasons for the proper alignment of student learning goals with the outcomes of simulations. Still, the question lingers, why isn’t there stronger evidence of validity between simulation performance and exam scores?

Part of the issue may come from the context of most extant research—the strategic management course. These simulations are often conducted in groups, and group-level characteristics such as cohesion and formal decision-making procedures have been shown to be powerful predictors of simulation performance (Faria, 2001). But are these results artifacts of the group setting? We propose that individual differences across students are also likely to complicate, and partially explain, the relationship between simulation performance and exam scores. There is a strong possibility that personality is linked to performance on different types of assessments. For example, a person high in conscientiousness has the personal characteristics to study intensely for multiple choice exams but may not be hardwired as well for the uncertainty found in real life simulations. Extraversion has similar implications for students. Does the student high in extraversion have enough “alone time” for intense multiple choice exam preparations?

Mood is another individual characteristic that could have impact on various assessments differentially. Mood has been studied as a state (how you feel now) and also as a trait (how you feel overall). Both state and trait positive and negative affect, or mood, have been shown to correlate with many job-related and health-related outcomes (Erez & Isen, 2002; George, 1991) including affective commitment (Kluemper, Little, & DeGroot, 2009). Commitment has three dimensions with the strongest outcomes associated with the affective component (Allen & Meyer, 1990). Affective commitment occurs when one feels a strong pull resulting from how one feels toward the organization or task. It is an element required for one to follow through on a task that might not be pleasant and it is stronger in those who are high in positive affect (mood). Thus, the levels of trait mood could logically impact students’ approach to, and performance on, various course assessments if they have differentially weighted affective commitment components.

Thus, our purpose in this exploratory study is twofold. First, we want to examine the correlation between the objective results from an HR simulation with the traditional course evaluation tool of exams using multiple choice, or objective questions. Our second purpose is to examine the characteristics of students who perform well in both these course evaluation tools. Do aspects of personality or affect differentially predict performance as measured by the two evaluation tools?

Method

Simulation

Each participant takes the role of an HR professional in a moderate sized company making the types of operating decisions required in a typical HR department such as pay increases, hiring, whether to spend on programs such as training, affirmative action, and survey research and they must do this all within a budget (see Smith and Golden (1994) for more details). Importantly for this research, each student operates independently, acting as the HR manager for their own company. Hence, the team effects that often dominate simulation performance did not affect our participants. In each decision period, students must choose where to spend and what not to support. These decisions are recorded and then processed using a computer algorithm evaluating their decisions against their competition and against their budgets. In addition to these decisions, each period has a written incident that must be content analyzed toward making a choice from among several options. Competition comes from groups of 20 participants, each participant representing one company, organized into “industries.” Companies are competing for finite resources set up within the simulation and the output delivered by the simulation shows each participant how his or her individual company performed against the competition. As an additional performance evaluation, the authors ranked the outcomes for each company by decision, which were totaled to create an overall performance metric that is used as a proxy for simulation performance.

Participants

This study was conducted using 215 students in a third year undergraduate university course in HRM at a mid-sized university. As part of the course, students worked for 13 weeks on the HR simulation which was worth 30% of their final grade. A one-hour class period was provided each week to ask questions of the instructor and to finalize decisions before submitting them. Each student was expected to establish objectives, plan strategy, and then make decisions dictated by these plans. Additionally, students were surveyed at the end of the course for their reactions to the simulation.

Measures

Dependent variables. Two dependent variables are used in this study. The first is exam grades. As part of the course, each student took three typical, objective exams on course contents worth 30% of the final grade. The second dependent variable in this study was an objective measure of performance from the above described Human Resources Management simulation. In this study, there were 8 decision periods that counted and with each decision period representing one-quarter of a year, the study simulated a 2 year period for each participant’s company. The first 4 weeks of the study were used to train the students for the task they were about to undertake. At week 5, they began submitting decisions that were evaluated by the computer program and these evaluations were ranked against their competition, the other companies. At week 12, final competitive positions were derived that comprised a cumulative ranking of their performance. Aside from the obvious benefit of this measure being free from subjective rater bias, it also offers the benefit of spreading out performance scores and eliminating range restriction.

Independent Variables. The following independent variables were measured in Week 1 of the study.

1. Personality. Big Five personality traits were measured using Form S of the NEO Five-Factor Inventory (Costa & McCrae, 1992). Reliability estimates expressed by Cronbach’s alpha coefficients were: α = .80 for emotional stability, α = .66 for extroversion, α = .47 for openness to experience, α = .64 for agreeableness, and α = .67 for conscientiousness.

2. Mood. Measures of positive and negative mood, or affect, were gathered using the 20-item Job Affect Scale (Brief, Burke, George, Robinson, & Webster, 1988). Positive affect was measured by asking participants to rate themselves on 10 items such as enthusiastic, active, and relaxed using 5-point degree of agreement scales (α = .72). Negative affect was measured with 10 items such as distressed, fearful, and jittery and this scale had similar internal consistency (α = .79). For each item, respondents are asked to indicate how s/he felt at work during the project on a 5-point scale ranging from 1=not at all, to 5=very much for each of the mood adjectives.

3. Cognitive Ability. The Wonderlic Personnel Test (Form V) was used to measure cognitive ability. The reliability and validity of this test have been established in the literature. Reliability estimates range from .82 - .95 and studies predicting outcomes in employment settings such as job performance, productivity, and supervisory ratings have resulted in validity estimates ranging from .22 - .67 (Wonderlic Inc., 2002).

4. Reaction to Simulation. Three questions were written to gather information about participants’ reactions to the simulation and were measured using five-point scales. The questions asked if they enjoyed the simulation whether they performed well or not, if it challenged them, and if it was a fair representation of course material. Internal reliability was sufficiently high for this short scale (α = .70).

Results

Table 1 shows the correlations among all study variables, their means and standard deviations. Of particular importance are the correlations among the two dependent variables, simulation performance and exam performance, and the individual difference variables. Simulation performance is positively related to exam performance (r = .33, p < .05) supporting the notion that both measures are tapping course success dimensions. Simulation performance is also positively related to cognitive ability (r = .16, p < .05) and positive mood (r = .15, p < .05). Exam performance is positively related to conscientiousness (r = .20, p < .05) and negatively related to extraversion (r = -.24, p < .05). These statistically significant correlations provide impetus to examine the individual difference variables as predictors of both simulation and exam performance.

Regression analysis results are presented in Table 2. As the table shows, two separate regression equations support the correlational information reported above. For simulation performance, cognitive ability is the strongest predictor (β = .21, p < .05).with positive mood as the only other statistically significant predictor (β = .14, p < .05). By contrast, the strongest predictors of exam performance are personality variables: conscientiousness (β = .21, p < .05) and extraversion (β = -.20, p < .05). Cognitive ability also predicts exam grades significantly (β = .20, p < .05).

Not shown in the tables are the results of exit interview questions. Three questions were asked of students in an attempt to gather information about participants’ reactions to the simulation. Students were asked to provide rating on five point scales. The results show that they enjoyed the simulation whether they performed well or not (mean=4.2). They also felt it challenged them (mean=4.6) and was a fair representation of course material (mean=4.3). Student evaluations of instruction were higher for the semester with the simulation than the previous semester when the course was structured without the simulation (4.84/5, n=215 versus 4.57 /5, n=196).

Discussion

To our knowledge, this is the first study to examine the use of a simulation in an HR course. One of our goals was to use the simulation in conjunction with other tools in an effort to examine both discriminant and divergent validity while also improving the learning environment for students. To examine attainment of this goal, we hypothesized that the simulation would be deemed useful if it correlated with exam scores to some degree, but not perfectly. Our logic for this is that all course components should have a degree of convergent validity with exam scores, but since objective exams measure quite different skills and abilities than what one might expect on applied tests (including simulations), there should also be evidence of discriminant validity. This result is exactly what we found, and is consistent with findings regarding the use of simulations in the strategic management course. Had the exam scores and simulation scores not been correlated at all, there would be no evidence of the utility of the simulation. As we found, there was a moderate relationship between the results of both types of assessment, a finding that simulation users should be relieved to see. Simulations are related to traditional course outcomes, but their discriminant validity also appears to be quite high. As we discuss later, however, some post-hoc analysis of these results suggest the simulation may have considerable utility in OB/HR courses.

Our second goal was to examine the characteristics of students as compared to the different assessments. We hypothesized that certain personality traits, or other individual differences, should be correlated with different assessments. Correlational evidence shows that cognitive ability is related to success on both assessments in approximately equal fashion. This suggests that performance in both domains draws on a common set of cognitive abilities and thus reinforces the validity of using simulations. When we consider the personality and mood constructs, however, we see a more interesting pattern.

As Table 2 reveals, both extraversion and conscientiousness are the significant predictors of exam performance, but not simulation performance. The more extraverted students do worse on exams than less extraverted students. This could be due to the extravert having difficulty mastering course concepts from text books since this is typically a solitary process. The extravert might be more distracted by outside influences and have difficulty succeeding in study that requires discipline without being around others. Since extraversion does not predict simulation performance, students across the entire range of extraversion can succeed on the simulation.

Conscientiousness is also a predictor for exam, but not simulation, performance. It is this result that suggests an interesting aspect of using a simulation. Because conscientiousness indicates that the individual is self-disciplined and achievement oriented, the surprise is not that conscientiousness is related to exam performance, but that it is not related to simulation performance. This suggests there is something about the simulation that engages the less dedicated student than does the traditional motivation factor of getting a good grade on tests. Said plainly, it appears that simulations may engage students in HR courses that our traditional means of instruction have not.

Positive mood also predicts simulation performance. This implies that there is an affective component to simulation performance, an implication that makes sense given that many aspects of work imitated by the simulation require affective responses. Whereas students that are motivated to achieve, as indicated by their higher levels of conscientiousness, will plow through the material and do better on tests regardless of their view (positive or negative) of the material or testing methodology, the simulation again seems to engage students that are not motivated by the traditional approach.

On a post-hoc basis, we investigated whether the correlation between exam performance and simulation performance grew stronger over the semester. It did. The correlation between the simulation performance and first exam score was not significant (r= 0.13). For the third exam score it was 0.38 (p < .05). We suggest that there are at least two reasons for this occurrence. First, it is possible that the positive experience of the simulation caused the less conscientious students to become more interested in the coursework and study harder for tests at the end of the course. Second, recent research on simulations in strategic management courses suggests that generative learning is taking place, with the simulation promoting the student to organize, integrate, and elaborate on the textbook information as they perform the simulation (Zantow, Knowlton & Sharp, 2005). Either of these explanations is good news for human resource educators.

HR educators have long been frustrated by the disengagement of the typical undergraduate student when compared to other business courses (Burke & Moore, 2003). This disengagement is made all the more frustrating by the fact that job recruiters clearly indicate the value and importance of HR material (Rynes, Trank, Lawson & Ilies, 2003). Our findings indicate the use of a simulation may reduce the degree of disengagement and generate more learning. The improvement noted in course evaluations when a simulation was used is another hopeful sign. All educators, not just HR educators, are challenged by the different learning styles of students that have been raised with virtual technologies (Proserpio & Gioia, 2007). Proserpio and Gioia suggest we are no longer teaching a TV-generation, but rather a virtual generation. Among the many changes in pedagogy that this generational shift may require is an increased use of simulations in coursework. Students crave the immersion of the interactive experience that simulations can offer and are likely to learn more in courses that provide this experience. This seems a particularly important time to consider the use of simulations in our courses.

Conclusion

It is apparent from this study that simulations used in human resource management classes have discriminant validity distinct from that of traditional exams since they do not correlate very strongly. Furthermore, certain characteristics of students predict success on exams, but not simulation performance. For instance, none of the Big 5 personality factors predict simulation performance though some predict exam scores. Rather than being a problem, this is a positive finding in that it indicates simulations engage students in ways different from the lecture and examination method. Although much research remains to be done, our preliminary findings suggest that a simulation may engage certain students who become disengaged from the typical HR course.

Our findings also suggest that this engagement may then lead to improving exam scores over the course of the semester. Given that so many programs rely heavily on objective exams to estimate the amount of material learned, adding something like a simulation to the course that improves exam performance should be extremely valuable as we try to increase student knowledge of HR. It proves that simulations benefit the students in that they help to learn course material already covered in lectures. Though differences in learning styles (concrete versus abstract perceivers) certainly leads to success on tests directed at the strength of each learning style, the dynamic environment in which we live and work requires that students are able to adapt to multiple styles to remain relevant and successful. In summary, our results should hearten those that already use a simulation in their HR courses and encourage those that do not use a simulation to consider so doing.

References