Chapter 12: Peer Review

How to provide useful feedback.

This chapter accompanies our first major project.

Motivating scenarios: You want to be helpful in improving and evaluating the work of your peers.

Learning goals: By the end of this chapter you should be able to:

• Meaningfully and respetfuly review the work of a peer.

Here’s an edited version of the section:

LLMs and Peer Review
ChatGPT and other large language models (LLMs) are not substitutes for peer reviewers. They are tools, not peers. While LLMs may provide useful feedback—indeed, using ChatGPT to evaluate your paper can offer valuable insights—this is not their primary role. Additionally, relying too heavily on the same feedback from a computer program can hinder diversity of thought in research. For this reason, journals like G3 have established policies prohibiting the use of LLMs for peer review. Specifically, in their instructions to reviewers G3 states: “The use of Large Language Models (LLMs), such as ChatGPT, to write your review is prohibited. Uploading a manuscript or associated files or manuscript information to an LLM also violates the confidentiality agreement in place for authors and the journal.” I follow the same policy here.)

Peer Review: What’s the Point?

Peer review cannot completely prevent bad science, nor is it the savior of science itself. As an editor for numerous academic journals and a reader of thousands of peer-reviewed articles, I can confidently say that bad science does get past peer review, and it does so routinely. This may occur for a variety of reasons—reviewers may lack the necessary expertise, be time-constrained, or simply miss flaws in the research. Additionally, peer review is not primarily designed to detect scientific fraud, though in rare cases it may expose issues that raise suspicion.

So, what is the value of peer review? The purpose of peer review is to ensure several things:

• The research presented is consistent with the author’s claims and conclusions, verifying that the results support the interpretations.
• The research is communicated effectively, clearly, and honestly, making it accessible and understandable to others in the field.
• The methodology and findings are sufficiently detailed to allow others to reproduce and reanalyze the work, which is a cornerstone of scientific rigor.
• Reviewers offer constructive feedback, helping refine the research by suggesting alternative interpretations, approaches, or critiques. This ensures that the research is viewed from multiple perspectives, potentially strengthening the final version.

Thus, peer review remains a vital mechanism for improving the quality of research. By facilitating dialogue, offering critical feedback, and promoting transparency, it enhances the integrity of the scientific process, even if it can’t guarantee perfection.

Why more than onve reviewer?

As an associate editor, I typically seek multiple reviewers (usually three) to comment on a research paper. We do this because having multiple perspectives helps highlight different strengths and weaknesses in the work. Each reviewer brings their own expertise and focus, meaning they may catch different issues or offer unique insights that others might overlook.

For that reason, while it’s important to do your best during peer review, it’s also essential to recognize that the paper will be evaluated by other reviewers as well. You don’t need to address every single aspect alone—collaborative evaluation helps ensure a more thorough review.

How to conduct peer review.

All peer review will be somewhat different, because have different assignments, aim for different jounrals etc etc etc. For this project you should evaluate the work relative to the rubric I presented. Still there are commonalities about what makes a good peer review A Good Peer Review

• Clearly articulates the author’s goals: A good peer review begins by summarizing the main objectives of the paper, ensuring that the reviewer has understood the key questions or hypotheses the authors are addressing. This helps align feedback with the author’s intent and provides a foundation for constructive critique. This is also an opportunity for the reviewer to evaluate these goals, and suggest additional or alterantive goals that could be pursued.

• Highlights how the authors succeeded in achieving their goals: Start by acknowledging the strengths of the paper. A good review points out where the authors have effectively met their goals, whether through well-designed experiments, insightful analysis, or clear and logical writing. Positive feedback reinforces successful elements and encourages further refinement in these areas.

• Provides constructive feedback on areas for improvement: A peer review should offer specific suggestions for how the authors can enhance their work. This may include clarifying ambiguous points, suggesting additional analyses, etc. The aim is to provide actionable advice that helps the authors better achieve their goals.

• Highlights shortcomings or areas for improvement: In addition to praising strengths, a strong peer review also identifies the paper’s weaknesses or areas needing more attention. These could include issues such as methodological flaws, weak data support, gaps in logic, or incomplete analyses. Pointing out areas for improvement allows the authors to refine their arguments, improve the validity of their findings, and enhance the overall quality of the paper.

  • Technical shortcomings: Are there any flaws in the methods or analyses that compromise the validity of the results? The reviewer should identify these issues and suggest ways to address them.

  • Conceptual weaknesses: Does the paper have logical gaps or inconsistencies in its argument? Reviewers should point these out and recommend ways to tighten the reasoning or clarify the hypotheses and interpretations.

  • Presentation issues: Sometimes, the issue lies not in the content but in how it’s communicated. Reviewers should note whether the writing is unclear, the structure is disorganized, or figures and tables are difficult to interpret, offering specific suggestions for improvement.

• Maintains a balanced, respectful tone: Good peer reviews are thoughtful and respectful, acknowledging the effort that went into the work while offering constructive feedback. Harsh or overly critical language can detract from the value of the review and discourage productive revisions. DONT BE A JERK

• Addresses both content and presentation: In addition to evaluating the substance of the research, a good review also considers the clarity and organization of the writing. Clear communication is crucial for the paper to have the intended impact, so suggestions for improving structure, readability, and flow are also very helpful.

Bad Peer Review

Bad peer reviews come in many forms, but they all share a common flaw: they fail to take the time to seriously and respectfully engage with someone’s work. A bad review can be dismissive, superficial, overly focused on minor issues, or unnecessarily harsh. Bad peer reviews are just shitty - they can upset authors, make them solve fake problems, and take their attention away from the important stuff. Here is what I tend to find in bad peer reviews:

• Superficial feedback: A review that simply says “looks good” or “this needs work” without providing detailed feedback is not helpful to the author. A good peer review provides specific, constructive feedback, but a bad one offers vague, uninformative comments that leave the author guessing about how to improve their paper. This lack of engagement can prevent the author from making meaningful revisions.

• Overly focused on trivial details: Sometimes reviewers get bogged down in small, nitpicky details—such as minor grammatical issues or formatting inconsistencies—at the expense of addressing the core ideas of the paper. While attention to detail is important, a review that misses the bigger picture is not useful. If the reviewer only points out minor flaws without addressing more substantive issues related to the research design, argument, or conclusions, the review becomes unbalanced, losing sight of the forest for the trees.

• Unnecessarily harsh or dismissive: Reviews that are mean-spirited or overly critical can do more harm than good. A bad peer review might ridicule the author’s work, dismiss it outright, or use an unconstructive tone. This kind of feedback is unprofessional and unproductive, as it discourages the author from improving the paper and can damage the collegial nature of academic discourse. Peer review should be constructive, even when pointing out significant flaws.

• Failure to engage with the substance of the work: A bad review may fail to address the core research questions or key contributions of the paper. Instead, the reviewer might focus on peripheral issues, offering little in terms of substantive critique. This lack of depth means the review doesn’t help the author understand the strengths and weaknesses of their work or how to improve it.

• Unreasonably demanding or unrealistic expectations: Some reviews ask for unrealistic changes that are beyond the scope of the paper or the capacity of the study. For example, requesting entirely new experiments or additional data that would take years to collect is often impractical. A bad review may also expect perfection from the outset, forgetting that peer review is meant to help improve the paper, not demand a finished, flawless product at the initial submission stage.

• Rushed or inattentive reviews: A common characteristic of bad reviews is a lack of care. A reviewer may skim the paper, offer minimal feedback, or fail to understand the methodology or context of the research. This can result in irrelevant comments or an incomplete assessment of the paper, leaving the author without the guidance they need to improve their work.

• Another class of bad review attemps to wrestle control away from the authors Bad reviews try to reshape the work to fit the reviewer’s vision. Good reviews, on the other hand, help the author achieve their goals while suggesting potential alternative directions. Respecting the author’s work means honoring their vision.

In summary, bad peer reviews are those that lack respect, depth, and thoughtfulness. Whether it’s through insufficient feedback, nitpicking minor points, harsh language, or an inability to grasp the larger contributions of the work, bad reviews fail to provide meaningful assistance to the author. A good peer review requires time, effort, and a commitment to helping the author refine and improve their research.

Homework

Evaluate my example project within the context of this rubric.

Figure 1: The accompanying quiz link