Cycle 28 Virtual Panel Review Guidelines

  • BlueJeans Instructions (To be posted in May)
  • Slack/Communications Instructions (To be posted in May)
  • Cycle 28 Peer Review Agenda (To be posted in May)

Introduction

The Hubble Space Telescope science peer review process has changed considerably in recent years, and this year we will have our first fully remote review, so even if you have served previously on an HST Panel or TAC, please review this documentation carefully. This review is organized by the Science Policy Group (SPG) of the Science Mission Office (SMO) of the Space Telescope Science Institute (STScI).

HST proposals are of three main types: General Observer (GO), Snapshot (SNAP), and Archival Research (AR). Briefly, GO proposals request observing time in an integral number of spacecraft orbits; GO proposals are now classified as Small (1-34 orbits), Medium (35-74 orbits), or Large (75 or more orbits), summed over Cycles 28-30. SNAP programs use small amounts of otherwise idle time in the HST schedule; they are allocated a number of targets of which in practice only a fraction, ~30 - 70%, will actually be observed. AR programs are to carry out investigations using the HST Archive and are awarded a dollar amount (proposers from U.S. institutions only). We are continuing three proposal categories that were initiated in Cycle 11: Treasury proposals are a special class of GO proposals that provide enhanced data products for the community; AR Legacy proposals are Treasury-style AR proposal; and Theory proposals, supported as part of the HST AR program.

Dependent on their size, proposals in Cycle 28 will be reviewed either by external panelists or by virtual review panels.

The Cycle 28 Peer Review will comprise eight topical panels, one each for:

  • Solar System Astronomy,
  • Exoplanets and Exoplanet Formation,
  • Stellar Physics and Stellar Types,
  • Stellar Populations (and the ISM),
  • Galaxies,
  • The Intergalactic Medium and the Circumgalactic Medium,
  • Supermassive Black Holes and Active Galaxies,
  • and Large-Scale Structure of the Universe.

Panelists are chosen based on their expertise in one or more of the areas under review by the panels. With one exception, the virtual discussion panels will assess and grade Medium GO proposals and Small GO proposals requesting 16-34 orbits; the exception is Solar System, where all proposals will be assessed in the discussion review process as the number of proposals is not sufficient for a split review. Each panel will be managed by a panel Chair and a Vice-Chair, and there will be a TAC Chair overseeing the review process. To assist the review process, each panel will also be assigned a Panel Support Scientist (PSS) and a Leveler; the role of the PSS is to ensure the process runs smoothly and act as liason between the panel and the institute, and the role of the Leveler is to ensure the discussion remains focused on the scientific merits of the proposals. More information about the review process is given below.

The remaining GO proposals (requesting less than 16 orbits), regular AR and SNAP proposals will be distributed for external review. Those proposals will be assessed by five experts who will grade on an absolute scale against the primary criteria: scientific merit within the field, broader importance for astronomy & the strength of the data analysis plan (where applicable); HST’s unique capabilities must also be required to achieve the scientific goals. Each external panelist will receive a limited number of proposals. The proposals will be grouped by subject area, matching the virtual panels. The rank ordered list from each subject area will be provided to the chair of the appropriate virtual panel prior to the meeting to allow the identification of potential conflicts. In addition, the chairs will have an opportunity to flag proposals with divergent grades for discussion by the virtual panel.

The Large GO, Treasury and AR Legacy proposals are reviewed by the panel Chairs and Vice-Chairs, three at-large TAC members and the TAC Chair. The super-TAC will also rank the Pure Parallel programs, and will adjudicate any cross-panel scientific issues, as needed. All recommendations for the Cycle 28 science program are advisory to the STScI Director, who is responsible for the final allocation of HST observing time and funding.

Anonymous Proposals

The primary objective of the review is to select the best-justified science. Removing the names of the proposal team from the proposal allows reviewers to focus directly on the science, rather than the scientist. The TAC panels and chairs rank proposals in order of scientific merit, and recommend the resources that should be allocated to each. The experience of the team with HST or otherwise is not a consideration for the initial science ranking.

Please do not spend time attempting to identify the team or the principal investigator.

A reviewer's preliminary grading should center on the scientific merit within the field, the broader importance for astronomy & the strength of the data analysis plan (where applicable); HST’s unique capabilities must also be required to achieve the scientific goals. The discussion during the meeting should focus on the specific criteria. Chairs and Levelers should be quick to refocus or terminate discussion when it moves to PI or team. 

Please note that the goal in anonymizing proposals is not to reduce content to the extent that useful information is eliminated. Proposers are encouraged to present relevant past HST observations and theoretical analyses, together with complementary observing programs. However, as outlined in the Call, the information should be presented in an anonymized fashion ( eg "observations of NGC 950 were obtained in Cycle 24 by Smith et al (2020)" as opposed to "we obtained observations of NGC 950 in our Cycle 24 proposal (PI Smith)."). Any privileged information should be cited as having been obtained through private communication. Panelists should contact their panel support scientist if they feel that a proposal is not compliant with the anonymizing guidelines. As far as possible, panelists should ignore any such breaches or errors in grading proposals. However, particularly egregious cases may be flagged for potential disqualification by STScI staff.

Observing time with HST should be openly available to any scientist who presents a highly compelling scientific case. However, that observing time is also a highly valued resource that must be used responsibly. To that end, the panels will have the opportunity to review a proposing team's expertise to ensure that they are capable of executing the program they have described, and in disseminating those results. This review will take place after the panel has ranked all programs based on the selection criteria. Panelists will be given the opportunity to review prepared statements by the proposers on their team's expertise and background. If, by consensus, the panel finds that a team lacks sufficient expertise or resources to execute the project described in the proposal, they may flag that proposal for potential disqualification to the Director. Panels must provide an appropriate justification for that action.

Proposal Review Process

Proposals reviewed by the virtual panels are subject to a two-stage review process: 1) preliminary grading and triage; and 2) the review meeting.

In the preliminary grading and triage phase, each proposal is reviewed and graded by 5 or 6 panel members; those preliminary grades are used to develop a ranked list. Proposals will be assessed on an absolute scale against the 3 primary criteria, with a separate grade given for each:

  • scientific merit within the field,
  • broader importance for astronomy & the strength of the data analysis plan (where applicable);
  • HST’s unique capabilities must also be required to achieve the scientific goals.

Our grading scale is: 

1 - Excellent
2 - Very Good
3 - Good
4 - Fair
5 - Poor

The top 50% of proposals are flagged for further discussion during the review meeting, and the rest marked for triage. The preliminary rankings are not made available, but, several days in advance of the meeting, panelists are informed which proposals will be discussed and which have been triaged. Panelists have the option of raising a limited number of triaged proposals for discussion and re-grading if they feel there are compelling reasons; panelists may not raise a proposal where they have a conflict. Panelists should inform the Chair and PSS of these proposals as soon as possible so as to give other panelists time to review the proposals ahead of the meeting.

In preparation for the review meeting, each proposal flagged for discussion will be assigned a Primary and Secondary reviewer who will be responsible for leading the discussion of the proposal in the review meeting. Panelists may wish to reread those proposals in preparation. Additionally, panelists should ensure that they have read all proposals flagged for discussion ahead of the meeting, including those they did not read in the triage phase (except for those on which they are conflicted).

During the review meeting, proposals will be discussed and re-graded in turn. The Primary will start the discussion, the Secondary will comment next, and then each (unconflicted) panelist will be given the opportunity to comment in turn before the floor is opened for general discussion. After the discussion, each proposal will be re-graded, with every unconflicted panel member voting on each proposal. Once all proposals have been re-graded, the ranked list will be compiled by the PSS for review. Panels can re-rank proposals to take account of factors such as progammatic balance. Once the ranked list is finalized, panels will be given the opportunity to review the team expertise submissions.

Every HST proposal receives feedback comments that should reflect the selection criteria outlined below. The Primary and Secondary reviewers are responsible for collating those comments; primary and secondary reviewers should enter their preliminary comments into the reviewer tool in advance of the panel meeting. There will be an opportunity to modify and adjust those comments during the meeting, but only limited time is available. Comments are sent verbatim and should be couched in an appropriately respectful manner.

The review comments are broken down into different subsections. The first two are compulsory, the rest are optional (and will likely not be required for most proposals):

  • Strengths: self-explanatory. Compulsory.
  • Weaknesses: self-explanatory. Compulsory.
  • Resources: notes about the Orbits or Targets or Funding levels, or instrument/configuration/filters, only needed if something is out of the ordinary. Optional.
  • Comments: anything else that you want to pass on that doesn’t fit under Strengths, Weaknesses, or Resources. Optional.
  • Technical Notes: updated based on technical reviews, usually relevant for Joint Observatory programs or if reviewers have concerns about instruments (in which case they should request a consultation from an instrument expert via the PSS, and the notes from that consult will go here). Optional.
  • Instructions: for approved programs only. Additional instructions for the proposers to help them prepare their Phase II submissions, usually a change in orbits or targets. Targets or orbits may be changed if two non-identical approved proposals contain duplicate observations; one of the proposals would be instructed to remove the duplicated observations, while preserving the unique elements of the proposal. Optional.

All accepted proposals are assigned a Program Coordinator who works with the PI to finalize the Phase II submission for feasible observations.

Confidentiality

All HST Phase I proposals and documents related to the proposal review are strictly confidential.

  • They should not be distributed or used in any manner not directly related to the review process. 
  • They may not be discussed with anyone other than those STScI staff participating in the review activities. 
  • There should be no discussions of the proposals with your fellow committee members other than those during the actual committee meeting.
  • The proposals and all related documents are to be brought to the Panel or TAC meetings for use there and collection afterward.  Please purge any review files from your computer after the review.
  • Your involvement in the Cycle 28 review process itself should also be kept confidential until after the results of the review have been announced.
  • Please do not share your involvement in the review process on Social Media.

Selection Criteria

Evaluations of HST proposals are based on the following criteria.

Criteria for all Proposals

  • The scientific merit of the program and its potential contribution to the advancement of scientific knowledge;
  • The program’s importance to astronomy in general. This should be stated explicitly in the “Scientific Justification” section of the proposal;
  • The strength of the data analysis plan;
  • A demonstration that the unique capabilities of HST are required to achieve the science goals of the program.

Additional Criteria for all GO and SNAP Proposals

  • What is the rationale for selecting the type and number of targets? Reviewers will be instructed to recommend or reject proposals as they are and to refrain from orbit- or object trimming. Therefore, it is very important to justify strongly both the selection and the number of targets in your proposal, as well as the number of orbits requested.
  • Is there evidence that the project has already been pursued to the limits of ground-based and/or other space-based techniques?
  • What are the demands made on HST and STScI resources, including the requested number of orbits or targets, and the efficiency with which telescope time will be used?
  • Is the project technically feasible and what is the likelihood of success? Quantitative estimates of the expected results and the needed accuracy of the data must be provided.

Additional Criteria for all Joint proposals

  • Are observations with both facilities required to achieve the science goals outlined in the proposal?

Additional Criteria for Large GO, Treasury GO, and Legacy AR Proposals

  • Is there a plan to assemble a coherent database that will be adequate for addressing all of the purposes of the program?
  • Is there evidence that the observational database will be obtained in such a way that it will be useful also for purposes other than the immediate goals of the project?

Additional Criteria for SNAP Proposals

  • Willingness to waive part or all of the exclusive access period. While this is not the primary criterion for acceptance or rejection, it can provide additional benefit to any proposal and will be weighed by the reviewers as such.

Additional Criteria for Archival Research Proposals

  • What will be the improvement or addition of scientific knowledge with respect to the previous original use of the data? In particular, a strong justification must be given to reanalyze data if the new project has the same science goals as the original proposal.

  • What are the demands on STScI resources (including funding, technical assistance, feasibility of data requests, archiving and dissemination of products)?

  • Is there a well-developed analysis plan describing how the scientific objectives will be realized?

  • Will the project result in the addition of new information that can be linked to the Hubble Source Catalog (HSC)?

Additional Criteria for Treasury GO and Legacy AR Proposals

  • What scientific investigations will be enabled by the data products, and what is their importance?
  • What plans are there for timely dissemination of the data products to the community? High-level science products should be made available through the HST data archive or related channels.

Additional Criteria for Theory Proposals

  • What new types of investigations with HST or with data in the HST Data Archive will be enabled by the theoretical investigation, and what is their importance?
  • What plans are there for timely dissemination of the theoretical results, and possibly software or tools, to the community?

Review Schedule

The Space Telescope Science Institute (STScI) will conduct the Virtual HST Cycle 28 Proposal Review May 11 - 15, 2020. The Chair, Vice-Chair and all panelists will join the discussion review meeting remotely via videoconference (details TBD). SPG staff, PSSs and Levelers will be onsite at STScI if possible, or remote as the situation dictates. The eight Review Panels will meet for 2 1/2 days each, and the overall TAC will meet for 2 1/2 days, immediately after the panel meetings.

  • Preliminary Grades for virtual panelists:
    • Wednesday April 29th, 5pm EDT
  • Grades from external panelists
    • Wednesday April 29th, 5pm EDT
  • Ranked lists for externally-reviewed proposals
    • Friday May 1
  • Review Orientation:
    • TBD (Most Likely Friday before meeting)
  • Panels:
    • Monday May 11 - Wednesday May 13 (will end no later than 2pm)
  • TAC:
    • Wednesday May 13 (2-3pm) - Friday May 15 (will end before 5pm)

Conflicts of Interest

Our goal is informed, unbiased discussion of each proposal:

  • Voting committee members should have neither direct nor indirect interest vested in the outcome of the review.
  • The subset of the review committee discussing the proposal should have sufficient knowledge to assess the science.

It is critically important that conflicts of interest, or the appearance thereof, are avoided during the selection process. Conflicted reviewers should not provide preliminary grades for a proposal and will leave the room and not take part in the discussion or grading of those proposals during the panel meeting.

The following are identified as conflicts:

  • Personal involvement (PI or Co-I)

  • Recent former advisor/student of PI or Co-I

  • Involvement in a closely competing proposal regardless of whether that proposal is also before your panel.

    • Proposals are judged to be 'closely competing' if their scientific goals, or their observations of the same targets, are sufficiently similar that an impartial panel, knowing one had been approved, would be unlikely to approve the other. Such conflicts should be direct and specific; a Panelist or Chair should not be excluded from considering an entire sub-category of proposals.

  • Strong competitor as the PI or co-I on the proposal
  • Close personal ties (family, etc.) with PI or Co-I

  • Close collaborator as the PI or Co-I on the proposal

    • A close collaborator is someone who is an active collaborator on a current research program, an active collaborator on at least 3 projects completed within the last 3 years, or an active co-author on 3 or more papers in the last 3 years (co-membership of a large consortium, such as SDSS, is not a disqualifying factor).

Since Cycle 28 proposals are anonymized, STScI staff will identify conflicts using the list of close collaborators supplied by the panelists. If a reviewer feels that they have a conflict with any additional proposals, please contact the panel support scientist to update the conflicts accordingly.

Role of Levelers

The goal of the dual-anonymous review is to keep discussions focused on the science proposed and not on the proposers themselves, with the desired outcome of reducing selection biases. Levelers are STScI staff members who sit in on each panel with the goal of maintaining the appropriate focus of discussion.

Reviewers should focus the discussion on the science project presented in the proposal and refrain from guessing or insinuating the likely identities of proposers, or instigating any discussion on possible past work. This will be challenging, and there will inevitably be an occasional “slip-up” or two. This is when levelers will be important.

Levelers are charged with ensuring that the virtual panels are focused on the science merit of the proposal. If the discussion veers to comments on the (possible) proposing team, their past work, their validity, or their identity, the levelers' job is to cut off the discussion, remind the panel of the anonymizing goals and guidelines, and direct the panel to continue discussing the merits of science proposed. Unlike the panel chairs, they are not listening for issues pertaining to science; rather they are focused on the discussion itself to insure that it stays on science. The guidelines to both reviewers and proposers on preserving the anonymity of the review are found on https://hst-docs.stsci.edu/display/HSP/HST+Cycle+28+Anonymous+Proposal+Reviews.

Proposals that have egregiously violated the anonymizing rules should be flagged prior to the meeting. Panelists should ignore less flagrant errors in proposals whenever possible, and keep focused on the scientific merits. But if, in deliberation of a given proposal, the proposer's self-revealed identity becomes impossible to ignore, and if that identity has a clear impact on the discussion, then that proposal should be flagged for potential disqualification. The levelers will be especially vigilant for those cases, and will raise those to the attention of the panel when they feel this threshold has been crossed.

Levelers may also play a role in the final discussion of team expertise. The criteria for sufficient expertise is left to the panels so they can address each proposal independently ( e.g., for particularly difficult datasets, or difficult analyses, or programs of high risk with less than certain outcomes). Panels must provide an appropriate justification for disqualification if they choose to make that recommendation to the director. Inexperience with HST data is not, in itself, a disqualifier. When appropriate, the levelers should remind the panel of the expectations in the criteria in recommending disqualifications.

Step by Step Details

  1. Read all of the Documentation.
  2. Read the Web-Reviewer Tool (SPIRIT) QuickGuide.
  3. Download your Proposal Package from the Web-Reviewer Tool as soon as possible; https://spirit.stsci.edu/.
  4. Check for any additional potential Conflicts of Interest. https://spirit.stsci.edu/.
  5. Identify any additional Conflicts to your PSS. Before April 5th.
  6. Review the proposals based upon the Selection Criteria. Now thru April 29th
  7. Enter comments for proposals where you are assigned as primary or secondary reviewer. Now thru May 10th
  8. Submit your Preliminary Grades for your Review and Grading assignments, by Wednesday 4/29; https://spirit.stsci.edu/.
  9. Review the ranked list. April 30 - May 10.
  10. Read any proposals in the first quintile that were not among those you graded. April 30 - May 10
  11. Inform your PSS of your nominated “saved” proposal by Friday May 8th
  12. Travel to the review. June 8 and 9.
  13. Review Meeting steps:
    1. Review the final list of proposals for discussion, Monday May 11
    2. Orientation ??
    3. Discuss and Grade all proposals before the panel, May 11-12
    4. Rank Proposals May 13
    5. Review the balance of science in the highest ranked proposals, May 13
    6. Review the team expertise for highest ranked proposals, May 13
    7. Review the TAC (Large/Treasury/AR Legacy) Proposals that have been assigned to your panel, May 13
    8. Finalize Notification Comments May 13
    9. End of Panel Meetings May 13
    10. TAC Panel Meeting begins afternoon of May 13
    11. End of TAC Meeting late afternoon of May 15