How to peer review: skill-building in a grad classes

I teach how to peer-review in graduate class because I think it is a core skill for any professional.  I first demystify peer-reviewing and academic journals, and answer questions that all students have about these topics that they have heard about but rarely learn about using this:

Nicholos and Gordon EOS, 2011

I describe my personal experience as a manuscript submitter, reviewer and associate editor. And then I outline the structure and types of questions to ask during a peer review (both listed below), and challenge them with three, increasingly difficult steps to learn how to peer review:

• first, peer review already published papers (which is surprisingly hard since it is already well edited but this is useful as practice and since it is impersonal).
• Second, peer review an open access manuscript that is currently in review (i.e. HESSD  or other open access journal). These can be actually submitted to the journal or not.
• Third, they peer-review eachother`s term papers before final submission of paper to me as part of the grade.

At each step myself or a TA gives them feedback and evaluates their peer reviews.

Good structure for a peer-review

• Short summary (1-2 sentences) and general assessment of novelty/contribution. Give the author(s) a few compliments here….everyone likes to eat the good-bad-good sandwich rather than just the bad sandwich.
• Discuss major concerns or suggestions for authors. Aim for positive criticism here.
• Recommend course of action: reject, accept with major revisions or accept with minor revisions.
• Document minor concerns with explicit page and line numbers.

Good questions to ponder:
Contributions and Audience:
What are the important contributions of this paper?
Does the paper make a significant, new contribution to this research area?
Who is the intended audience?

Technical soundness:
Are the methods fully described?
Is the mathematical/theoretical development (if any) complete and accurate?
Is the approach, experimental design, review or statistical analysis appropriate?

Organization and Style:
Is the paper a description of an experiment or concept or a synthesis of previous work?
Is the paper well written and organized?
What is the hypothesis, objectives or goals put forth?Are all the tables and figures necessary?
Can the paper be shortened?

Evaluation:
Are the interpretations of data and results justified?
What are the major conclusions? Are they significant? Are they interesting? What remains answered?

Your reactions:
Did you gain something from the paper (be specific)?
How does the paper relate to other topics discussed in class?Are such questions and/or methods relevant to your own research?