Cycle 28 External Review Guidelines

Introduction

The Hubble Space Telescope science peer review process has changed considerably in recent years, and this year we will have our first fully remote review, so even if you have served previously on an HST Panel or TAC, please review this documentation carefully. This review is organized by the Science Policy Group (SPG) of the Science Mission Office (SMO) of the Space Telescope Science Institute (STScI).

HST proposals are of three main types: General Observer (GO), Snapshot (SNAP), and Archival Research (AR). Briefly, GO proposals request observing time in an integral number of spacecraft orbits; GO proposals are now classified as Small (1-34 orbits), Medium (35-74 orbits), or Large (75 or more orbits), summed over Cycles 28-30. SNAP programs use small amounts of otherwise idle time in the HST schedule; they are allocated a number of targets of which in practice only a fraction, ~30 - 70%, will actually be observed. AR programs are to carry out investigations using the HST Archive and are awarded a dollar amount (proposers from U.S. institutions only). We are continuing three proposal categories that were initiated in Cycle 11: Treasury proposals are a special class of GO proposals that provide enhanced data products for the community; AR Legacy proposals are Treasury-style AR proposal; and Theory proposals, supported as part of the HST AR program.

Dependent on their size, proposals in Cycle 28 will be reviewed either by external panelists or by virtual review panels.

The Cycle 28 Peer Review will comprise eight topical panels, one each for:

  • Solar System Astronomy,
  • Exoplanets and Exoplanet Formation,
  • Stellar Physics and Stellar Types,
  • Stellar Populations (and the ISM),
  • Galaxies,
  • The Intergalactic Medium and the Circumgalactic Medium,
  • Supermassive Black Holes and Active Galaxies,
  • and Large-Scale Structure of the Universe.

Panelists are chosen based on their expertise in one or more of the areas under review by the panels. With one exception, the virtual discussion panels will assess and grade Medium GO proposals and Small GO proposals requesting 16-34 orbits; the exception is Solar System, where all proposals will be assessed in the discussion review process as the number of proposals is not sufficient for a split review. Each panel will be managed by a panel Chair and a Vice-Chair, and there will be a TAC Chair overseeing the review process. To assist the review process, each panel will also be assigned a Panel Support Scientist (PSS) and a Leveler; the role of the PSS is to ensure the process runs smoothly and act as liason between the panel and the institute, and the role of the Leveler is to ensure the discussion remains focused on the scientific merits of the proposals. More information about the review process is given below.

The remaining GO proposals (requesting less than 16 orbits), regular AR and SNAP proposals will be distributed for external review. Those proposals will be assessed by five experts who will grade on an absolute scale against the primary criteria: scientific merit within the field, broader importance for astronomy & the strength of the data analysis plan (where applicable); HST’s unique capabilities must also be required to achieve the scientific goals. Each external panelist will receive a limited number of proposals. The proposals will be grouped by subject area, matching the virtual panels. The rank ordered list from each subject area will be provided to the chair of the appropriate virtual panel prior to the meeting to allow the identification of potential conflicts. In addition, the chairs will have an opportunity to flag proposals with divergent grades for discussion by the virtual panel.

The Large GO, Treasury and AR Legacy proposals are reviewed by the panel Chairs and Vice-Chairs, three at-large TAC members and the TAC Chair. The super-TAC will also rank the Pure Parallel programs, and will adjudicate any cross-panel scientific issues, as needed. All recommendations for the Cycle 28 science program are advisory to the STScI Director, who is responsible for the final allocation of HST observing time and funding.

Anonymous Proposals

The primary objective of the review is to select the best-justified science. Removing the names of the proposal team from the proposal allows reviewers to focus directly on the science, rather than the scientist. The TAC panels and chairs rank proposals in order of scientific merit, and recommend the resources that should be allocated to each. The experience of the team with HST or otherwise is not a consideration for the initial science ranking.

Please do not spend time attempting to identify the team or the principal investigator.

A reviewer's preliminary grading should center on the scientific merit within the field, the broader importance for astronomy & the strength of the data analysis plan (where applicable); HST’s unique capabilities must also be required to achieve the scientific goals. The discussion during the meeting should focus on the specific criteria. Chairs and Levelers should be quick to refocus or terminate discussion when it moves to PI or team. 

Please note that the goal in anonymizing proposals is not to reduce content to the extent that useful information is eliminated. Proposers are encouraged to present relevant past HST observations and theoretical analyses, together with complementary observing programs. However, as outlined in the Call, the information should be presented in an anonymized fashion ( eg "observations of NGC 950 were obtained in Cycle 24 by Smith et al (2020)" as opposed to "we obtained observations of NGC 950 in our Cycle 24 proposal (PI Smith)."). Any privileged information should be cited as having been obtained through private communication. Panelists should contact SPG if they feel that a proposal is not compliant with the anonymizing guidelines. As far as possible, panelists should ignore any such breaches or errors in grading proposals. However, particularly egregious cases may be flagged for potential disqualification by STScI staff.

Proposal review

The external review of proposals will be done based solely on the review and assessment of the external members.  Each reviewer will provide the assessment and grading on about 10-20 proposals assigned to their scientific expertise.  Proposals will be assessed by five experts who will grade on an absolute scale against the 3 primary criteria, with a separate grade given for each:

  • scientific merit within the field,
  • broader importance for astronomy & the strength of the data analysis plan (where applicable);
  • HST’s unique capabilities must also be required to achieve the scientific goals.

Our grading scale is: 

1 - Excellent
2 - Very Good
3 - Good
4 - Fair
5 - Poor

Each external panelist will receive a limited number of proposals. The proposals will be grouped by subject area; the recommended list will be provided to the chair of the appropriate on-site panel prior to the meeting to allow them to identify potential conflicts with the proposals reviewed by the panel. In addition, the chairs will have an opportunity to flag proposals with divergent grades for discussion by the on-site panel. The panel chairs will use this information to monitor the programmatic balance of the recommended list of on and off-site proposals.

Reviewers should provide review comments for each proposals. These are broken down into different subsections. The first two are compulsory, the rest are optional (and will likely not be required for most proposals):

  • Strengths: self-explanatory. Compulsory.
  • Weaknesses: self-explanatory. Compulsory.
  • Resources: notes about the Orbits or Targets or Funding levels, or instrument/configuration/filters, only needed if something is out of the ordinary. Optional.
  • Comments: anything else that you want to pass on that doesn’t fit under Strengths, Weaknesses, or Resources. Optional.
  • Technical Notes: updated based on technical reviews, usually relevant for Joint Observatory programs or if reviewers have concerns about instruments (in which case they should request a consultation from an instrument expert, and the notes from that consult will go here). Optional.
  • Instructions: A place to flag any duplicated targets, and your recommendation for which proposal should be allocated the duplicated target(s) if both were approved. Optional.

Confidentiality

All HST Phase I proposals and documents related to the proposal review are strictly confidential.

  • They should not be distributed or used in any manner not directly related to the review process. 
  • They may not be discussed with anyone other than those STScI staff participating in the review activities. 
  • There should be no discussions of the proposals with your fellow committee members other than those during the actual committee meeting.
  • The proposals and all related documents are to be brought to the Panel or TAC meetings for use there and collection afterward.  Please purge any review files from your computer after the review.
  • Your involvement in the Cycle 28 review process itself should also be kept confidential until after the results of the review have been announced.
  • Please do not share your involvement in the review process on Social Media.

Selection Criteria

Evaluations of HST proposals are based on the following criteria.

Criteria for all Proposals

  • The scientific merit of the program and its potential contribution to the advancement of scientific knowledge;
  • The program’s importance to astronomy in general. This should be stated explicitly in the “Scientific Justification” section of the proposal;
  • The strength of the data analysis plan;
  • A demonstration that the unique capabilities of HST are required to achieve the science goals of the program.

Additional Criteria for all GO and SNAP Proposals

  • What is the rationale for selecting the type and number of targets? Reviewers will be instructed to recommend or reject proposals as they are and to refrain from orbit- or object trimming. Therefore, it is very important to justify strongly both the selection and the number of targets in your proposal, as well as the number of orbits requested.
  • Is there evidence that the project has already been pursued to the limits of ground-based and/or other space-based techniques?
  • What are the demands made on HST and STScI resources, including the requested number of orbits or targets, and the efficiency with which telescope time will be used?
  • Is the project technically feasible and what is the likelihood of success? Quantitative estimates of the expected results and the needed accuracy of the data must be provided.

Additional Criteria for all Joint proposals

  • Are observations with both facilities required to achieve the science goals outlined in the proposal?

Additional Criteria for Large GO, Treasury GO, and Legacy AR Proposals

  • Is there a plan to assemble a coherent database that will be adequate for addressing all of the purposes of the program?
  • Is there evidence that the observational database will be obtained in such a way that it will be useful also for purposes other than the immediate goals of the project?

Additional Criteria for SNAP Proposals

  • Willingness to waive part or all of the exclusive access period. While this is not the primary criterion for acceptance or rejection, it can provide additional benefit to any proposal and will be weighed by the reviewers as such.

Additional Criteria for Archival Research Proposals

  • What will be the improvement or addition of scientific knowledge with respect to the previous original use of the data? In particular, a strong justification must be given to reanalyze data if the new project has the same science goals as the original proposal.

  • What are the demands on STScI resources (including funding, technical assistance, feasibility of data requests, archiving and dissemination of products)?

  • Is there a well-developed analysis plan describing how the scientific objectives will be realized?

  • Will the project result in the addition of new information that can be linked to the Hubble Source Catalog (HSC)?

Additional Criteria for Treasury GO and Legacy AR Proposals

  • What scientific investigations will be enabled by the data products, and what is their importance?
  • What plans are there for timely dissemination of the data products to the community? High-level science products should be made available through the HST data archive or related channels.

Additional Criteria for Theory Proposals

  • What new types of investigations with HST or with data in the HST Data Archive will be enabled by the theoretical investigation, and what is their importance?
  • What plans are there for timely dissemination of the theoretical results, and possibly software or tools, to the community?

Schedule

The Space Telescope Science Institute (STScI) will conduct the HST Cycle 28 Proposal Review May 11 - 15, 2020. The Virtual Review meetings will take place at the Space Telescope Science Institute. Eight Review Panels will meet for 2 1/2 days each, and the overall TAC will meet for 2 1/2 days, immediately after the panel meetings.

  • Proposal Distribution:
    • Monday March 30th
  • Grades and Review Submissions:
    • Wednesday April 29th

Conflicts of Interest

Our goal is informed, unbiased discussion of each proposal:

  • Reviewers should have neither direct nor indirect interest vested in the outcome of the review.
  • The Reviewers should have sufficient knowledge to assess the science.

It is critically important that conflicts of interest, or the appearance thereof, are avoided during the selection process. Conflicted reviewers should not provide grades for a proposal and should identify any new conflicts to SPG.

The following are identified as conflicts:

  • Personal involvement (PI or Co-I)

  • Recent former advisor/student of PI or Co-I

  • Involvement in a closely competing proposal regardless of whether that proposal is also before your panel.

    • Proposals are judged to be 'closely competing' if their scientific goals, or their observations of the same targets, are sufficiently similar that an impartial panel, knowing one had been approved, would be unlikely to approve the other. Such conflicts should be direct and specific.

  • Strong competitor as the PI or co-I on the proposal
  • Close personal ties (family, etc.) with PI or Co-I

  • Close collaborator as the PI or Co-I on the proposal

    • A close collaborator is someone who is an active collaborator on a current research program, an active collaborator on at least 3 projects completed within the last 3 years, or an active co-author on 3 or more papers in the last 3 years (co-membership of a large consortium, such as SDSS, is not a disqualifying factor).

Since Cycle 28 proposals are anonymized, STScI staff have identified conflicts using the list of close collaborators supplied by the panelists. If a reviewer feels that they have a conflict with any additional proposals, add the conflict in the Reviewer tool, which will update the conflicts accordingly and will notify SPG.

Step by Step Details

  1. Read all of the Documentation.
  2. Read the Web-Reviewer Tool (SPIRIT) QuickGuide.
  3. Download your Proposal Package from the SPIRIT tool as soon as possible; https://spirit.stsci.edu/.
  4. Check for any additional potential Conflicts of Interest. https://spirit.stsci.edu/.
  5. Identify any additional Conflicts via SPIRIT. 
  6. Review the proposals based upon the Selection Criteria. Now thru April 29th
  7. Enter comments for proposals. Now thru April 29th
  8. Submit your Preliminary Grades, by Wednesday 4/29; https://spirit.stsci.edu/.