Surviving the Path to Peer Review Success Joyce Loper and Mike Strauss

Surviving the Path to
Peer Review
Joyce Loper and Mike Strauss
Office of Scientific Quality Review
Why OSQR Review?
1998 Farm Bill
ARS research peerreviewed every 5 years
Most review panelists
external to ARS
Satisfactory review
before beginning
Stakeholder input
Action Plan
Objectives set
You are
4. Project
Plan prepared.
Annual progress
Research initiated
OSQR Review and
“Life after the PDRAM…”
FIRST: Review OSQR Handbook and Area/RL expectations
-Plan Drafted lead scientist
and project team
-Review by other colleagues
-Review by RL
Revised plan to Area Office
for approval
(some require proof of outsider review)
Approved Plan sent by Area to
Office of National Programs
Revision if
(through Area)
Validation by
National Program Leader
Validated plan returned
to Area
Due to OSQR
If needed,
plan revised
Red denotes established
Dates for completion.
Some Advice
•Set a time line
•Based on when the plan is due to the Area Office
•What does your Area require?
•Some want proof of review outside your group.
•If not you should still send the plan outside your
group for review.
•Schedule time for:
• Each member of the team to write
•Members to coordinate plans
•Lead scientist to compile a cohesive document
•Colleagues to review the plan
•RL to review the plan
•Revision of the plan following review
Who Oversees OSQR?
Joyce Loper, ARS Scientific Quality Review Officer (SQRO)
•Joyce approves each panel chair and panel member who
participates in OSQR
•Joyce is responsible for certifying that project plans have
completed review
Mike Strauss, OSQR Coordinator
•With two staff members, Mike schedules all of the panels,
contacts and trains all of the panel chairs and panel members,
and coordinates the review and certification process.
Both Mike and Joyce:
•Attend panel meetings
•Read your project plans
•Read the OSQR reviews of your project plans
•Read your responses to review
What is OSQR Review?
A Dialogue
And …
External Review
Scientific Review
Prospective Review
Peer Review
Quality Review
Who are the Reviewers?
Panelists are your colleagues.
They read your peer-reviewed papers.
Panelists are active scientists.
Most are academics (per the Farm Bill).
Panelist often know your work.
And are often familiar with your excellence.
Panelists take their task very seriously.
The devote many hours to each review.
They don’t want to give low scores!
Who are the Reviewers?
0% 2%
Former ARS
How is a Panel Selected?
OSQR Receives suggestions/nominations from
ONP, Areas, others.
All potential chairs are screened for COIs.
Coordinator Interviews potential candidates and
SQRO approves Chairs.
Chairs work with Coordinator to develop a
balanced, proposed list of panel members.
SQRO reviews and approves final list of
Reviewers NEED to know…
What is the problem?
Why is it important?
Where are you going with it?
How are you going to get there?
And how will you know you have arrived?
Don’t make them hunt for this!
Aggregated Plans
In some cases plans may be comprised of several
independent pieces. In such cases be aware of…
- Clarity…why are all these pieces here?
What links them?
State, however, that they are independent
- Consistency…each piece should be equivalent in detail
Someone needs to oversee the final product.
- Content (flow)…is it an “easy read?”
The general format is not rigid.
Consider blending background and approach for each section.
- “Consensus”
First parts (summary, need for research, objectives…need to present the
whole picture.
The Message…
By page 7 (Need for Research)
reviewers should know:
The subject of your research
Why it is important
What it will produce
How you are going to get there
The rest of the document will “flesh out” this but if
the reviewers don’t know it by this point, they
probably won’t get it easily from the rest of the
Review Products
• Action Class Score
• Consensus review comments
Project Review Criteria
Adequacy of Approach and Procedures
Probability of Successfully Accomplishing
the Project’s Objectives
Merit and Significance
Document Outline
Title and Investigators..………….page 1
Signature Page…………… 2
Table of Contents……….………….page 3
Project summary (250 words) 4
Objectives...………… 5
Need for research (1-2 p)
15 - 30 pages
Scientific Background (5-7 p)
Approach & Procedures (6-15 p)
Prior Accomplishments (2 p)
Literature Cited
Milestone Table (1-3 p)
Past Accomplishments of Project Team Members
Issues of Concern statements
Appendices (letters plus other material)
These are not boxes…they are guides to your narrative flow.
Project Plan Components
Project Summary – 250 words
Write this in active voice.
State the essential problem and why it is important.
What have you done to date (1-2 sentences)?
How will you address the issue?
Why is this important?
This is where you capture the interest of the
reviewer. Make it compelling!
Project Plan Components
Where are you going?
Express need scientifically AND in the context of NP Action
Be concise in statement of research purpose.
Discuss potential benefits and anticipated products.
Identify relevant customers and stakeholders.
Briefly note the principal methods you will utilize (e.g.,
…using microarray technologies we will elucidate…”
Build upon, don’t repeat, the overview!
Project Plan Components
Objectives: 1-2 pages
How does all this fit?
Why are all these pieces here?
How do they relate?
Are there closely allied plans that bear on this work?
A figure can help! Should reflect your plan and be a guide.
- Objectives and sub-objectives
- Personnel
- Outcomes
- Related projects
Your plan can include 4 total pages of figures. Use them!
Project Plan Components
Scientific Background: 5-7 pages
Why are you going there?
Highlight knowledge gaps.
Literature demonstrates understanding and gaps. Not an exhaustive.
Show a rationale for the objectives.
How will this fill knowledge gaps?
Limit to 1/3 of project plan length
Note similar projects within and outside ARS and how your past work
prepares for or leads to this (provide details in the Prior
Accomplishments section…but say enough to convince reviewers you
know the area).
Cite preliminary data from your projects, if available
Project Plan Components
Prior Accomplishments: 1-2 pages
What have you done before?
Highlighted briefly in the Background.
Name prior project terminated within two years
Major objectives and accomplishments
Prior project investigators
Impact of prior work (science, technology, users)
Pertinent publications
A table or chart of past data can be very helpful.
Project Plan Components
Approaches & Procedures: 6-12 pages
How are you going to get there?
Set out your Experimental design.
Formulate REAL, testable, hypotheses!
Describe approaches and methods any why they are appropriate.
Discuss advantages and limitations (important if methods are “risky”).
Illustrate how objectives can be achieved.
Who will do what, how, and when (including collaborators and SCAs!)
Describe nature and extent of collaborations, including SCAs
Letters in Appendix need to confirm what you say!
For SCAs, a copy of the agreement is sufficient.
Include management, evaluation and contingencies.
What is your path to success? How will you monitor it?
Project Plan Components
Goal or Hypothesis
Goal or Hypothesis
Dynamic over the project lifecycle
Project Plan Checklist
• Readability and narrative flow
• Connection between parts (a diagram)
• Appropriate roles for all
• Appropriate expertise on team or from collaborators
• Grammar/spelling/proofreading
• Appropriate detail in Approach
• Clear, proper, milestones
• Real contingencies
Does the plan instill confidence in this team’s abilities?
Some Recent (frequent) Criticisms
Real Hypotheses—Are they testable? NOT REQUIRED but
don’t use a general goal where a hypothesis is better!
Lack of connection--How/why do the parts of your plan relate?
Or if part does not, why is it there?
Uneven presentation—If different people wrote different parts,
it will show!
Context of plan—How does this fit with other similar work
within and outside ARS?
Statistically sound—Are replicates sufficient? How will you
analyze…don’t just throw out jargon (“Data will be analyzed
using ANOVA.”)…are replicates sufficient? When you’re
done, will you be able to know if you arrived? “But we’ve
always done it this way” is not sufficient.
…More frequent Criticisms
How will it get done?—Who does what? What other
resources are there? (postdocs, technicians,
students…include in human and physical resources)
Vagueness that prevents real analysis—If the
information is confidential say why you can’t tell
them but say enough to allow some level of analysis.
Risk without justification--Risk can be good but ONLY if
it’s apparent you are aware of the challenge and
have justified it.
Data accumulation without analysis—It’s not enough to
gather data, what will you do with it?
To keep in mind…
The reviewers need to see the logical
“thread” through your work.
Don’t make readers “search” for what you
are doing!
Be clear, accurate, and correct.
Don’t assume reviewers know you and
your work…(a poor plan may not be
saved even if they do!)
Some hints to success…
Proofread Your Plan
Ask a nontechnical person to read your plan
Ask someone who hasn’t seen it to read and proofread
your plan
Ask a highly critical colleague to read it thoroughly.
Are collaborations documented appropriately?
Check hypotheses…
Treat this the same care you would
a competitive proposal. The reviewers will!
1. Don’t avoid them if appropriate but don’t force
them if they are not.
2. Must be falsifiable and testable.
3. Not restatements of objectives.
4. May not be appropriate for work like breeding
or germplasm characterization…but explain
5. Seek review by a statistician.
Hypothesis Problems
Too complex. Statements with “and” and “or” make it difficult to accept
or reject.
Wiggle words. “May” or “might” or “could”. If answer is to “try harder” it’s
not a testable hypothesis.
Misdirected. Statements like “Discovering the mechanism behind X will
enable us to…….” test researchers not the experimental system itself.
Statements of the obvious. “Disease results from expression of genes
for virulence in the pathogen and genes for susceptibility in the host.”
Not a science question. “Quantifying X will provide significant increases
in income for the industry.”
Sometimes a GOAL is preferable to an
Hypothesis. We recommend:
Nabel, Gary J., 2009
The Coordinates of Truth
Science, 326 (5949): 53-54
What Happens After Review?
No, Minor or Moderate Revision
Lead Scientist responds to comments. Scientific Quality Review Officer
certifies compliance with recommendations.
Major Revision or Not Feasible
Lead Scientist revises and responds to comments.
Panel performs a second review assessing response to their comments
and assigns a new Action Class Score. If still Major or Not Feasible,
project is returned for administrative action. No further review.
Projects are reviewed no more than two times
(There are no page limits for revised plans)
What Happens After Review?
The Impact of Vertically Striped Voles (VSV) on Wheat, Rye, and Egg Production
R. U. Kidding
Frontiers of Vole Biology and Relativity Theory
Can I disagree with the panel?
This is a dialogue
If you really disagree…put it away for a few days!
Honestly consider panel opinions.
Be polite but if you disagree say why
DON’T skip changes to plan
DON’T insult or impugn panelists
DO provide justification for your alternative
Panels are NOT perfect…they are fellow scientists
How not to disagree
Q: The panel does not see any [expertise] in this plan.
A: “I disagree.” [no explanation]
Q: Can you provide some preliminary data to support this idea.
A: “Yes, we have preliminary data but can’t/won’t show it to you.”
Q: The panel suggests you try this approach.
A: But that’s just too difficult.
A: We’re not allowed to alter this project in any way. [not true!]
A: We’ve done it our way for [x] years and see no reason to change.
Q: This is not a hypothesis. Fix it or change to a goal statement.
A: I looked at Tom’s plan and Bill’s and their panel didn’t make them
do this so I don’t think I should have to do it.
Q: Did you do a power analysis?
A: No we did not, but we’ve always done it this way before.
A. Yes. It said we needed more so we ignored it.
Good to know…
• A final copy of your responses the panel (for their information) after it is
• For plans scoring Moderate or higher, OSQR reviews the responses to
assure they are thorough and appropriate; and may return them for
additional work before certification if needed.
• The Officer can decline certification if, after several attempts, it is judged
that the researchers have not or cannot adequately address reviewer
comments (i.e., your plan does not “pass” until it is certified).
• For plans scoring Major Revision or below, while OSQR may briefly check
to see if the responses are thorough and respectful…this is not a detailed
review and does not assure panel re-review success.
Last Words
Seek Review
proofread and seek more review
And lastly
Proofread and Seek Review
…Correct Grammar and Spelling are
Important -- but not enough
`Twas brillig, and the slithy toves
Did gyre and gimble in the wabe;
All mimsy were the borogoves,
And the mome raths outgrabe.
Lewis Carroll
Be thoughtful, clear, and thorough
…and beware of overconfidence…