HST Cycle 26 Anonymous Proposal Reviews

Introduction

STScI has a responsibility to simultaneously ensure that the community has equal opportunity for the use of HST and that the best science is being done with the finite amount of observing time available. The Institute places a high value on the equity and integrity of the proposal review process. The goal is to enable each reviewer to focus on the science, not the scientist. Several studies have shown that a reviewer's attitude toward a submission may be affected, even unconsciously, by the identity of the lead author or principal investigator (see the Anonymous-Double Blind Review Annotated Bibliography [.docx]).  We have noted that over the last 15 cycles, HST proposals led by women have had systematically and successively lower success rates than those led by men. While the exact cause is unknown, independent studies (see On Gender Bias in Hubble Proposal Ratings [.pptx]) of our reviews suggest a double-anonymous process might help resolve this inequity, and may balance out other areas of potential bias including affiliation and country of origin. Such a process may also level the playing field between new and established researchers. The focus of the TAC review is to recommend the best science. The identity of the proposing team should not be a consideration in making this judgement.

In the spring of 2018, STScI convened a working group from the astronomy community to explore the idea of a dual-anonymous system and issue a set of recommendations to the STScI Director. The working group's report, along with detailed instructions to proposers and reviewers, and a list of FAQs, can be found here. A summary of those guidelines, along with a description of how the review process will work, is given below.

The Review Process for Cycle 26

As in past cycles, proposers submit their proposals through APT. However, the PDF attachment that is uploaded containing the scientific and technical justifications must be anonymized following the guidelines below. Additionally, proposers must submit, via email to STScI (hubblereview@stsci.edu), a separate document titled "Team Expertise and Background." This short document, described below, should be emailed to STScI after the main proposal is submitted, and should contain the APT generated submission number. The review panels (and the TAC) will conduct their review without seeing any of the names associated with the proposal, and without seeing the "Team Expertise and Background" document. The panels will discuss the proposals and generate a final ranked list of proposals that are recommended for selection. In addition to the Panel Chair, each review panel (including the TAC) will have a full-time "Leveler" present in the room during all panel discussions. The job of the Leveler is to ensure that discussions remain focused on the scientific merit of the proposal.

Once the ranked list is set, the panels will be given access to the "Team Expertise and Background" documents associated with each proposal. At this point, proposals may only be flagged for downgrade, where a downgrade would result in a non-selection of the proposal. Proposals may not be upgraded once the identity of the team is known. This flag, assigned by majority vote of the panel, should only be used in the most extreme circumstances of a team being clearly unqualified to undertake the work proposed. Should a proposal be suggested for downgrade, both the Panel Chair and the Leveler will participate in the discussion about why this recommendation is necessary. A detailed description of the reason for the flag must be given. This flag will then be passed on to the STScI Director, along with the proposal's initial ranking, and a statement by the panel on the rationale for flagging the proposal. The Director will make the final decision, in consultation with appropriate personnel from STScI, including the Science Mission Office (SMO), HST Mission Office, ESA, and operations/scheduling staff. Finally, any proposals that are downgraded will have the reasons for downgrade passed on to the proposers. The same process will be applied to Large proposals by the TAC.

Guidelines for Proposers

Provided here are guidelines to assist proposers in preparing their proposals, specifically their PDF Submissions, to help conceal the identities of the proposers, and ensure a fairer proposal evaluation process. The anonymous review does not mean proposals will be accepted from anonymous sources. As with previous cycles, proposers must still enter the names and affiliations of all investigators into the APT system. APT will not include names or affiliations in the versions generated for the reviews.

While APT will largely obscure the proposing teams identities in cover materials, it will not change or alter information contained in the PDF submission. Thus, it is necessary for proposers to take additional steps to further anonymize their PDF attachment before it is uploaded to APT. Below are some guidelines to accomplish this:

• Do not include author names or affiliations anywhere in the PDF attachment. This includes but is not limited to, page headers, footers, diagrams, figures, or watermarks. This does not include references to past work, which should be included whenever relevant (see below).
• Referencing is an essential part of demonstrating knowledge of the field and progress. When citing references within the proposal, use third person neutral wording. This especially applies to self-referencing. For example, replace phrases like “as we have shown in our previous work (Doe et al. 2010)” with “as Doe et al. (2010) showed...” Do not refer to previous campaigns using HST or other observatories in an identifying fashion. For instance, rather than write "we observed another cluster, similar to the one we are proposing under HST program #XXXXX," instead write "HST program #XXXXX has observed this target in the past..."
• We encourage references to published work, including work citable by a DOI. It may be occasionally important to cite proprietary datasets or non-public software that may reveal (or strongly imply) the investigators on the proposal. We suggest proposers use language like "obtained in private communication" or "from private consultation" when referring to such potentially revealing work.
• Do not include acknowledgements, or the source of any grant funding. 

It takes some effort by authors to anonymize their PDF submissions. As the guidelines show, grammar and structure are expected to be different than in previous HST submissions. Some examples of re-worked text can be found on Example text for Cycle 26 anonymous proposing. Take sufficient time to prepare the manuscript, especially if one plans to resubmit a proposal from a previous cycle or other submissions. 

Proposers should make an effort to describe the past work in the field, and how this proposal will improve, build-upon, or complete that past work. Many successful proposals include a discussion of stated-sample goals or statistical completeness and how this proposed work fits into this broader context. Similarly, proposals may also discuss the uniqueness of the sample, and goals in comparison to similar work.

Guidelines for Reviewers

The primary objective of these reviews is to select the best science, not the best science teams. The TAC panels and chairs rank proposals in order of scientific merit, and recommend the resources that should be allocated to each. The experience of the team with HST or otherwise is not a consideration. Do not spend time attempting to identify the team or the principal investigator. All accepted proposals are assigned a Program Coordinator who works with the PI to finalize the Phase II submission for feasible observations. MAST provides "science ready" data for most uses, and there is help/documentation for further data processing. A reviewer's preliminary grading should be centered on the scientific merit of the proposal. This includes technical issues in the design of the study, as described in the technical justification and elsewhere. The discussion should focus on the scientific merit of the proposal. Chairs and Levelers should be quick to refocus or terminate discussion when it moves to PI or team. 

Team Expertise and Background Section

After submitting their proposal, proposers should email a document to STScI at hubblereview@stsci.edu containing a PDF of their "Team Expertise and Background" section. This document should be a brief description of the expertise, background, and roles of the team members, as they relate to the science proposed. This document should be no more than one page in length; for most proposals, a paragraph or two will suffice. For proposals with a large number of Co-Investigators, it is not necessary to report on the qualifications of every team member, only the most relevant ones. This document does not need to be a bio of the team members. A sample document is provided on Proposer Guidelines in Anonymous Reviews.

Compliance

Compliance with this policy is mandatory. Proposals received with flagrant violations will be subject to disqualification before the review-panel stage. Proposals with less serious violations (e.g., forgetting to change a reference from first person to third person) will be allowed to remain in contention, but will be flagged for review by SMO and the STScI Director for a final decision. Feedback will be provided to the proposers regarding any violations.

A possible concern that may arise is the following: "I've made every effort to anonymize my proposal, have followed all the guidelines, changed all my references to third-person, but I fear that my work is so specialized (or my analysis methods so unique) that panelists who know me will still be able to figure out who I am. Will my proposal be disqualified?" So long as the guidelines above are followed, the answer is NO, such a proposal will not be considered to be in violation. It is not necessary to "water down" or obscure your science, your methods, or your tools; it is simply your responsibility to write about them in the third-person, in a way that does not intentionally identify yourself. 

Next: HST Cycle 26 General Information, Resources, Documentation, and Tools