Chapter 4 Comprehensive Exams

4.1 Qualifying Comprehensive Exam: Integrative Literature Review

The I/O comprehensive exam is a prospective analysis of the student’s readiness to embark upon their dissertation research. This will take the form of a publishable integrative, theoretical, or meta-analytic literature review on the student’s broad area of study. The topic should expand beyond but subsume the scope of their dissertation and should be comparable to papers published in Psychological Bulletin, Academy of Management Review, Annual Review of Organizational Psychology and Organizational Behavior, Journal of Management (biannual review), Psychological Review, and so forth. A full listing of outlets publishing literature reviews, tutorials on how to write reviews, and examples of reviews are provided in Appendix A; students are encouraged to peruse these materials for examples and guidance. The reviewed research topic must bridge multiple theories, concepts, controversies, or findings within a domain of work psychology, systematically organize and review published findings in the area, and identify the key challenges and future directions for the area. Note the quality of the paper must be very high with the exam evaluating the student’s ability to demonstrate:

• An understanding of main theories, findings, and perspectives in a specialized topic
• An integration and synthesis of empirical and theoretical knowledge for a chosen topic
• A critical analysis of prior research, and
• Implications for future scholarly and practical work

To be admitted to doctoral candidacy, students must complete both a written and oral exam on their literature review and its connection to past work to ensure familiarity and mastery of relevant theories, empirical findings, reviews, historical trajectories, concepts, measures, techniques, and implications to make a novel and meaningful contribution to the field. Mastery will be defined as understanding, analyzing, and integrating enough general and specialized material literature to cover two entire courses worth of content, which roughly translates into 40+ source materials. The exam will be offered once each year at the start of the Fall semester. This allows the entire summer of their second year for preparation and a make-up opportunity in early winter. Students may take any portion of the exam twice. Two failure attempts will result in dismissal from the program or advanced remediation (see Section 4.2.2 for elaboration).

4.1.1 Scope of Review

The topic should be broad enough that students are able to find enough published papers to support the review. However, it should be focused enough that the paper has depth and students are able to thoughtfully explore the methodological and theoretical issues raised by the topic. The paper should make a scientific contribution; that is, it should not be a simple description of all of the studies that have been conducted on the topic. Instead, it should move the area forward by (a) making a theoretical advancement by, for example, pointing out inconsistencies in the literature and providing suggestions for how future research in this area should resolve them, or providing a theoretical framework to guide future research, or bringing up questions that are important but have yet to be addressed, or (b) making a methodological contribution by, for example, conducting a meta-analysis or other quantitative analysis (e.g., examination of effect sizes) to provide statistical confirmation of an effect.

There is no guideline for number or age of references or studies to review because that will depend on the topic. Very generally, however, papers of this breadth and depth have at least a few dozen references and focus on at least a dozen core papers in the review. The review should focus on peer-reviewed empirical journal articles, but book chapters and other reviews are helpful in guiding a search for references and providing theoretical background. We also suggest that students do not pick a general topic for which there are already many reviews (e.g., what accounts for the sex difference in leadership? What predicts turnover?). Again, we want the paper to make a unique contribution and be publishable.

Here are a few examples of the sort of titles that would meet criteria for a publishable literature review:

1. The efficacy of online coaching for improved leadership behaviors

This is good because virtual learning is a hot topic yet the review is focused enough (leadership, virtual coaching) so as to not be redundant with general meta-analyses and reviews which have already compared the efficacy of face-to-face and online learning modalities for a range of training outcomes (affective, behavioral, cognitive). At the same time, it is sufficiently broad to cover a variety of literatures which students can knit together into general conclusions (leadership behaviors: transformative, servant, task, planning, conflict, etc.; online coaching: live, adaptive assessment, aps, weekly diaries, dynamic or interactive). Can also delve into different issues, such as whether certain leadership outcomes are more amenable to virtual coaching than others, do effects depend on modality or integration with a broader L&D strategy, and a taxonomy of different ways to coach virtually.

2. Distinguishing job satisfaction, organizational commitment, and employee engagement

This would be a theoretical expansion and review of recent meta-analyses, articles, and other papers attempting to clarify how commonly studied employee attitudes are distinct yet overlapping. It is current, controversial, and one could get into many issues: what is the historical origin of each term and are each conceptually distinct states? What are the advantages & disadvantages of aggregating versus distinguishing attitudes? Among the three which are most powerful for different outcomes (performance, citizenship, turnover) or do they share similar predictive power? How do you recommend using these to take research on attitude and motivational theory forward? This could be a meta-analysis but the challenge would be there are several related meta-analyses so the review would need to expand, augment, or build upon past research in some way.

3. Preventing team dissolution: antecedents, processes, and interventions

In this paper one could review the literature focused primarily on how to prevent a team from falling apart over time. The time part is crucial to distinguish from reviews which focus purely on team conflict (disagreement) to a broader consideration of teams disconnecting, disbanding, or even working against one another over time. It also extends beyond conflict to consider other drivers of team dissolution, such as stress, lack of alignment, poor design, and team composition. Finally, the review would be unique in emphasizing any interventions and mechanisms which help teams overcome conflict, stay resilient, maintain cohesion, or hit other processes that help teams stay together. Some of this work is being pursued for teams in extreme environments, such as astronauts on long-term space missions. This could even be a meta-analysis if there are sufficient number of studies looking at an outcome representing team dissolution. One could then provide an integration and analysis of why interventions don’t always work as expected for team dissolution and provide recommendations for moving forward.

4. A cross-cultural analysis of career development trajectories

This could be a broad review bringing together cultural psychology, developmental psychology, and vocational psychology into an integrative focus on how career attitudes, prestige, and aspirations change across the lifespan in different countries. You can ask whether western and eastern countries have different trends in attitudes towards their career over time? Why might that be? What are the cultural/historical/religious reasons that underlie differences in valence and change over time? Is this changing with China’s rapidly westernizing culture? What are the implications of this for international companies, understanding workforce policies, and the generalizability of theories about career development? There are lots of angles one could take with this.

4.2 Eligibility and Timing

Students are eligible for comprehensive exams when they have met the following requirements:

1. Completed 36-42 credits worth of core methods, I/O, and elective courses (end of year 2)
2. Completed I/O Research Seminar Readings and Defense (PSYC 691, 692) or I/O MA Thesis (PSYC 698)
3. Selected a formal program mentor for dissertation and research
4. Hold a GPA of 3.00 or higher

4.2.1 Timeline and Process

The comprehensive examination process begins in the spring after a student has defended their thesis. The I/O Doctoral Director is responsible for assembling an annual exam committee consisting of three or more I/O faculty members. The committee typically includes the Doctoral Director, student’s primary advisor, and one or more independent faculty. The committee will be constituted by the end of the Spring semester (June) and the exam itself will begin in the start of the Fall of the student’s third year (September). Failure on any given stage of the comprehensive exam (i.e., proposal, written review, oral defense) leads to a one-month revision period where the student may address deficiencies and resubmit their responses. A second failure at the current or future stages will result in immediate dismissal from the PhD program.

The general process is as follows:

1. Idea Develolpment: We strongly encourage students to consult with their supervisor for help in picking a comps paper topic, as he or she is the expert in your area. Ideally, this will begin some point during your second year with January being a good starting point to start reviewing ideas and articles. Once students have picked a topic, we also encourage consultation with members of the comprehensive committee to determine whether the topic meets the criteria for a comps paper and for any questions about the paper’s structure prior to drafting the proposal.

2. Proposal: To begin the comprehensive exam process, students must submit a 3-page proposal and example reading list (see Section 4.2.2 below) by May 31st to the I/O PhD Director and exam committee. The proposal and reading list outline intended theoretical and empirical scope of the intended literature review and should include the following sections:

   • Breadth of Review: Explain why the area is important for work psychology, why the review is needed, and how the review advances the field. Include:
     • Relevant theories or principles
     • At least two domains, ideas, controversies, methods, or topics to be discussed (e.g., leadership, development, motivation, statistical technique, person v. situation, dynamic criteria, wearable sensors, assessment method, income inequality, etc…)
     • General approach to how review will be conducted (qualitative, quantitative, how comprehensive, boundaries, narrative, integrative, meta-analytic, bibliometric, etc…)
   • Review method: Describe methodology for selecting articles and reviewing findings (see resources and tutorials in Appendix A)
   • Example articles: list several example articles to be reviewed, including any specified by student’s advisor

3. Conduct Review: During the summer and in consultation with their advisor, the student expands the comprehensive exam proposal into a formal systematic review (see tutorials and suggested readings in Appendix A, especially Torraco, 2016). The literature review is expected to demonstrate all the following: clear and concise writing, significance, logical or thematic organization, method of review, critical analysis of existing literature, synthesis of findings into new knowledge, exploration of emerging developments or topics, and foundations for future research. While length will vary depending on topic and scope, a typical review/meta-analysis is between 8,000 – 12,000 words. A bibliography of all reviewed works should also be included (not part of page limit).

4. Review Submission and Evaluation: The final draft of the review is due September 1st. This coincides with SIOP submission deadlines so the student may also submit an abbreviated version of the manuscript to be considered for the annual SIOP conference. In the spirit of journal reviews, the committee has three weeks to evaluate the product as “accepted”, “minor revision”, “major revision” or “rejection” (which amounts to a fail) along with written feedback. If “major revision”, then student has 1 month to address comments otherwise the submission is a fail.

5. Oral Defense: Doctoral candidates are expected to be able to communicate effectively and knowledgably both in writing and orally. Within a month of passing the written component (including with revisions), the student completes an oral defense of their literature review where they verbalize the significance of the review, address prior deficiencies, and connect the review to general findings from I/O psychology. Students are expected to be able to discuss key subject areas or fields of research and to clarify, summarize, or assert different positions or arguments taken in prior written work. The oral exam itself is to begin with a brief introduction by the PhD Director, with the exam protocol being outlined. The student is expected to present a brief overview of their written portion of the exam including revisions to the written product. Each member of the committee is given a chance to ask questions, but the format will often shift between relatively structured questioning and a more free-flowing discussion. The discussion will center on the student’s literature review but the scope of the exam is not necessarily limited to that material. The examiners will be looking to fill in any perceived gaps in the written work, and to more generally assess the student’s facility with the literature, empirical material, and broader discipline. The accept/reject decision is to be made by simple majority of the votes cast by members of the examination committee. In the event of a minor revision, the committee may require amendments, revisions or conditions for a passing grade. The committee must then determine the arrangements for ensuring the conditions are met. Passage of the oral comprehensive exam signifies the student has appropriate mastery of the field’s central material and is permitted into doctoral candidacy to conduct focused and independent dissertation work.

4.2.2 Timeline

Table 4.1: Timeline

January		Student will begin discussing comprehensive exam proposal with their advisor
Mid May-Early June		Comprehensive exam committee formed. Student submits 3 page comprehensive exam proposal and reading list which introduces the research area they plan to review. Inability to do so results in the first ‘fail’ of the comprehensive exam process.
June-August		Student conducts a systematic review of the literature based upon formulated goal, scope, and methods. References are assembled (40+ sources), analyzed, synthesized, and evaluated. General conclusions formulated.
September		Student submits final review in publishable form to the comprehensive exam committee by September 1st.
		Comprehensive exam committee members agree on their evaluation of the literature review using specific evaluator criteria.
Within 3 Weeks Evaluation of literature review	Accept/Minor Revision: Within a month, student moves on to the Oral Defense component of comprehensive exam.	Major Revisions: If one or more members provide major revisions (with comments), the student must resubmit within 1 month of receiving the Committee’s feedback. The committee will then have 2 weeks to determine whether the revision is a Accept or a Reject (which would count as their first Fail on the exam).	Reject: If the literature review is deemed incomplete, incoherent, or of limited value the student will receive written comments from the committee and has 1 month to rewrite their literature review (first Fail on the exam). The Committee would then have 3 weeks to determine whether the new submission is a Pass or a Fail (which would be the 2nd Fail, ending the exam and requiring the student’s withdrawal from the PhD program.)
Within 1 month of passing Written Component		The comprehensive exam meets for the student’s Oral Defense. The student is to summarize the scope and significance of their literature review, address gaps and revisions, and link their specialization to general I/O principles.
(October – November)	Accept: Student becomes PhD Candidate and proceeds to the Dissertation Project	Minor Revision: Any revisions, amendments or conditions for a passing grade	Reject: Student receives written feedback on the oral presentation through the supervisor and may repeat the Oral Defense a month later. If this is the second failure (or the student fails Orals twice), the student will be required to withdraw from the PhD program.

4.3 Grading

4.3.1 Literature Review Rubric

Grading criteria will be based on both included content and quality of reasoning. Content is based upon the Torraco (2016) checklist for integrative literature reviews which includes evidence of the following:

• Goal: Is the topic, scope, need, boundaries, and perspective of the review articulated?
• Organization: Is the review organized around logical flows of ideas with a coherent narrative and primary themes? May include diagrams or visual representation.
• Methods: Are the methods for the review sufficiently described, such as how articles identified, included, analyzed, and described? Is the process repeatable?
• Critical Analysis: Identify deficiencies, omissions, conflicts, strengths, and weaknesses of current literature? What is poorly represented or unadressed?
• Synthesis: Are new ideas, insights, or propositions provided? Might include research agendas, taxonomies, alternative concepts or models, meta-analytic conclusions, or meta-theory.
• Future: Does the review explore future of the topics for the field, such as trends, interdisciplinary developments, or new directions? Is there a foundation for future research or direction on which to build questions?

Quality is reflected in how well the student organizes, integrates, and evaluates previously published material (Bem, 1995; Cronin & George, 2020). As such, they clearly defined and clarify the problem; summarize previous work in the area; identify relationships, contradictions, gaps, and inconsistencies; and suggested the next step or steps in solving the problem. The cognitive complexity or ‘critical thinking’ exerted in the synthetic function of the review will be evaluated using a simplified variant of Bloom’s Taxonomy of cognitive reasoning to graduate-level writing (Granello, 2001).

Table 4.2: Application of Bloom’s Taxonomy to Evaluating the Integrated Literature Review

Area.of.Analysis	Skill	Content.and.organization
Knowledge: recall or recognize ideas	Repeats information from other sources	Listing of what others found with no summative comments; unable to capture main ideas in own words
Comprehension: Grasp meaning and interpretation	Summarizes main points of articles reviewed	Able to distinguish main ideas of reviewed articles, not just recitation
Application: Use material in new situations	Makes explicit links between source articles and current paper	Paper organized by source articles with direct and explicit link to current paper (e.g., “Therefore, findings from X support the review’s premise that…”)
Analysis: Break into component parts	Identifies patters or themes in review; uses data from others to support current paper’s ideas	Articles link to review topic by identification of specific themes, components, or relevant elements of articles
Synthesis: Parts combined into new whole	Integrates and combines ideas from source articles into new whole; draw together ideas into thematically written research	Paper organized thematically rather than by source articles; main ideas presented with source material supporting and questioning ideas discussed
Evaluation: Judge value of material for given purpose	Make distinctions about quality of source articles based on objectively defined criteria	Paper organized thematically while source articles are analyzed and critiqued based on strengths and limitations. When findings conflict, this is acknowledged with quality or methods of source articles discussed as part of contradiction.

Based upon both the content and quality of the review, each committee member will globally evaluate the review with commentary and render a final decision, just like journal reviewers would. The outcome may be “accepted”, “minor revision”, “major revision”, or “reject”. If the outcome is “accepted” or “minor revision”, the student proceeds to step 4. If the outcome is major revision, the student must revise the article according to the committee’s comments and achieve an outcome of “accepted” or “minor revision” within a single month; otherwise, the student receives one “fail” and has another month to redress before termination. Finally, if the paper is “reject” the student automatically receives one fail and must reformulate and rewrite substantial portions of their paper and similarly attain an outcome of “accepted” or “minor revision” in order to remain in the doctoral program.

4.3.2 Oral Exam Rubric

The oral exam builds on the written review by requiring students to summarize their contribution, explain revisions, and answer posed questions. The oral serves three purposes: (a) evaluate ability to communicate the scope and significance of specialization; (b) opportunity to explain modifications, expansions, and revisions based on written feedback; and (c) demonstrate breadth of knowledge by tying their review to broader findings, principles, and concepts from the field.

Regarding these aims, the following criteria apply to grading performance on the oral exam:

• Content Mastery: Students can verbally present, defend, and explain their proposed conceptualization and evaluation of the literature. Shows concise understanding of main perspectives; general themes integrating concepts and results; major gaps or limitations; and implication for future scholarly and practical work.

• Revisions: Can pinpoint and address earlier gaps in the goals, organization, methodology, critical analysis, synthesis, or future directions of their systematic review. Address reviewer (i.e., faculty) criticisms and suggestions.

• Disciplinary Knowledge: Ability to connect the scope of their review to broader theories, methods, findings, principles, and concepts from industrial and organizational topics. Understands how to bring relevant results of the review back to relevant course material.

Considering the above, each examiner independently evaluates the student’s oral performance overall as “pass”, “minor revision”, or “reject.” The student is asked to leave the room after which the examiners discuss and arrive at consensus on the oral defense decision. A “pass” designates advancement to doctoral designation. A “minor revision” stipulates conditions needed before passing onto doctoral candidacy. Finally, a “reject” automatically counts as one fail where the student must address major conceptual gaps, repeat their oral defense, and attain an outcome of “pass” or “minor revision.”

4.4 Program Dismissal Due to Failed Comprehensive Examination

If a student does not pass their comprehensive examination after all re-attempts have been exhausted, then the following will occur:

1. The student will be notified in writing that s/he is being dismissed from the PhD program at the end of the current semester due to a failure to pass the comprehensive examination. The student is still eligible to receive an MA degree assuming coursework and capstone have been completed. Notification of dismissal should occur as soon as possible. Either the faculty or the student may request a meeting to discuss the dismissal decision and notification.

2. The student may appeal his/her dismissal to the Dean of the College within five (5) business days of receiving notification of such dismissal.

3. The student may appeal a decision to uphold the dismissal by the Dean of the College to the Dean of the Graduate College and the Office of the Provost within five (5) business days of receiving notification of the Dean of the College’s decision.

4. A decision to uphold the dismissal by the Dean of the Graduate College and the Office of the Provost may be appealed to the President of the University within five (5) business days of receiving notification of the Dean of the Graduate College/Office of the Provost’s decision. The decision of the President is final.

References

Bem, D. J. (1995). Writing a review article for Psychological Bulletin. Psychological Bulletin, 118(2), 172.

Cronin, M. A., & George, E. (2020). The why and how of the integrative review. Organizational Research Methods, 1094428120935507.

Granello, D. H. (2001). Promoting cognitive complexity in graduate written work: Using Bloom’s taxonomy as a pedagogical tool to improve literature reviews. Counselor Education and Supervision, 40(4), 292–307.

Torraco, R. J. (2016). Writing integrative literature reviews: Using the past and present to explore the future. Human Resource Development Review, 15(4), 404–428.