Cycle 28 Panel Support Guidelines
- Panel Support Orientation from March 18, 2020 Meeting
- BlueJeans Information
- Slack Information
- Web-Reviewer Guide
- Panel Support Checklist for Mock and Review Meetings.
- Cycle 28 Peer Review Agenda
The Hubble Space Telescope science peer review process has changed considerably in recent years so even if you have served previously as a support staff, please review the documentation carefully. The Panel members have their own guidelines that will also be useful for your review. To minimize duplication, these guidelines describes only those issues unique to the panel support duties.
Dependent on their size, proposals in Cycle 28 will be reviewed either by external panelists or by discussion (virtual) review panels. Panel Support staff will only be responsible for supporting the virtual discussion review, and will not be required to support the external review process.
The Cycle 28 review will comprise eight topical panels, one each for:
- Solar System Astronomy,
- Exoplanets and Exoplanet Formation,
- Stellar Physics and Stellar Types,
- Stellar Populations (and the ISM),
- The Intergalactic Medium and the Circumgalactic Medium,
- Supermassive Black Holes and Active Galaxies,
- and Large-Scale Structure of the Universe.
Panelists are chosen based on their expertise in one or more of the areas under review by the panels. With one exception, the discussion panels will assess and grade Medium GO proposals and Small GO proposals requesting 16-34 orbits; the exception is Solar System, where all proposals will be assessed in the discussion review process as the number of proposals is not sufficient for a split review. Each panel will be managed by a panel chair and a co-chair, and there will be one overall TAC chair overseeing the review process.
The remaining GO proposals (up to 15 orbits), regular AR and SNAP proposals will be distributed for external review, which will be handled separately.
Confidentiality and Conflicts of Interest
The proprietary nature of the proposals and related documents is taken extremely seriously by STScI. Proposals are the intellectual property of the PIs, and the sensitivity of the process is enhanced by the heavy over-subscription of HST time and the significant funding implications for US proposers. Review-related materials should not be left in the open when not in use, and should be locked away overnight or when otherwise inactive.
Panel membership is confidential except within a panel, until after the review. It is critically important that conflicts of interest are avoided during the selection process. See the Panel Member website for their conflicts of interest rules.
As a panel support member you have a conflict of interest with a proposal only if one of the following occurs:
- Personal involvement: i.e. PI or CoI
- Any Relation to PI or CoI (family ties)
- You feel conflicted
You leave the room for discussion, grading, voting and ranking if that proposal is near the cutoff. All HST Phase I proposals and documents related to the proposal review are strictly confidential.
- They should not be distributed or used in any manner not directly related to the review process.
- They may not be discussed with anyone other than those STScI staff participating in the review activities.
- There should be no discussions with your fellow support staff members other than those during the actual committee meetings.
- The proposals and all related documents that you might have printed, are to be brought to the Panel or TAC meetings for use there and collection afterward. Please purge any review files from your computer after the review.
The Space Telescope Science Institute (STScI) will conduct the Virtual HST Cycle 28 Proposal Review May 11 - 15, 2020. The Chair, Vice-Chair and all panelists will join the discussion review meeting remotely (details TBD). PSS and SPG staff will be onsite at the Space Telescope Science Institute if possible, or remote as the situation dictates. The eight Virtual Review Panels will meet for 2 1/2 days each, and the overall TAC will meet for 2 1/2 days, immediately after the panel meetings.
- Preliminary Grades:
- Wednesday April 29th, 5pm EDT
- Review Orientation:
- TBD EDT (most likely Friday before)
- Monday May 11 - Wednesday May 13 (will end no later than 2pm)
- Wednesday May 13 (2-3pm) - Friday May 15 (will end before 5pm)
Role of the Panel Support Staff (PSS)
The PSSs are intermediaries between the external reviewers and STScI. Their main tasks are to answer questions, to ensure that the panel meetings proceed smoothly, to record the results of the panel discussions, and to bring any critical issues promptly to the attention of Science Policies Group (SPG) and/or relevant STScI technical staff. The PSSs do not take part in the scientific discussions of the panel and do not vote, so that they have no conflicts of interest other than direct participation in a proposal.
The PSS performs a vital task. They represent STScI and provide continuity through their insight into the operations of HST, STScI and the review process. It is the PSSs who minimize the number of issues that 'slip through the cracks' and cause problems later on in the process. Their participation increases the quality of the process, which depends substantially on their scientific and technical expertise, commitment, and integrity.
Review Process and Reviewer Assignments
Proposals and panelists are assigned to specific panels by SPG. The review is split into two phases, the Triage Process and the Review Meeting. The Triage Process happens before the meeting. Proposals are assigned to panelists (using software to minimize conflicts of interest) such that each proposal is reviewed and graded by 6 panelists. SPG will average the 6 grades and make a ranked list. Proposals in the bottom 50% will be triaged, the rest will be discussed at the Review Meeting. Each remaining proposal is assigned one primary reviewer and one secondary reviewer (again using software to minimize conflicts of interest) who will lead the discussion of the proposal during the Review Meeting. This assignment process will happen after the Triage Process to balance the workload for each reviewer. Before the meeting, primary and secondary reviewers should re-read the proposals to prepare to lead those discussions, and all panel members should read any proposals passing triage that the did not already read (unless conflicted). All unconflicted reviewers will be required to grade during the Review Meeting. Reviewers will be assigned by specific scientific subfield expertise. The responsibilities of these reviewers are described on their webpage.
Conflicts of interest for panelists are also defined on their webpage. The software finds (and avoids) only conflicts that can be identified by the name or the institutional affiliation of a reviewer as well as any pre-defined conflicts by the reviewers. Reviewers who feel that they have a conflict of interest on a proposal to which they have been assigned as a primary or secondary reviewer or as a preliminary grader are instructed to contact their PSS. In such cases the PSS should reassign the proposal to another reviewer or grader. The new reviewer must not have a conflict (please check explicitly with the potential new reviewer), and have specific expertise in the scientific area of the proposal if at all possible.
The PSS may make multiple reviewer reassignments (or perform swaps), if necessary to keep a similar number of assignments for each reviewer (the balance does not have to be exact). Any reviewer reassignments should be communicated by the PSS via email to the reviewers involved. The Panel Chair should also be informed about reassignments; the PSS should ascertain directly from the Chair whether they prefer to be updated with each reassignment or to receive the final assignment list after all swaps are made (just before the review).
The PSS should read (at a minimum) each proposal's abstract and observation summary in order to acquire an overall familiarity with the proposal content of their panel, to assist the Chairs in organizing the material and running the meetings, and to flag duplication, resource, and feasibility issues for further review by STScI specialists as appropriate (see below).
In Phase I there will not be a systematic technical review of all proposals, nor will there be a systematic computer-aided target/exposure duplication review. Such reviews are only done in Phase II for proposals that have been approved in Phase I. Nonetheless, both the PSS and the panelists should be alert to any such issues in Phase I, either before or during the panel meetings. Any such issues that are noted should be recorded in the Web-Reviewer tool, and if appropriate, attempts must be made to resolve them. Cross-panel duplications will be dealt with only after preliminary rankings, and should be flagged by the Chairs. Any duplications (as defined in Data Rights and Duplications section of the Call for Proposals) that are not specifically noted in the 'notification comments' of the database as being approved by the panel will be disallowed in Phase II.
If a PSS or reviewer has any policy, scheduling, technical or scientific question(s) about a proposal, (e.g., proposed observations of Mercury, use of filters that don't exist on a given instrument, etc.), you should direct any questions about a proposal, to SMO: firstname.lastname@example.org. In the subject field write "TAC question for SI", where SI should be replaced by the name of the Scientific Instrument (ACS/COS/FGS/STIS/WFC3) or use Scheduling, Policies, etc to which the question pertains.
In either case, the body of the email should contain the proposal ID number, and the question itself. We will forward your question to an appropriate expert and provide a response usually within two business days.
A written record of any noted technical feasibility issues must be kept for each proposal. These should be entered into the 'Technical Comments' section of the Web-Reviewer tool. These comments must be augmented with any resolutions (or lack thereof) that emerge from the panel deliberations.
During the meetings, the PSS functions as an executive assistant to the Chair. They organize the materials, provide input on any technical issues noted before the meeting, coordinate real-time consultation with STScI specialists, monitor the grade voting and final ranking process as well as the reviewer comments status. More specifically, the principal activities during the meetings are as follows.
- Assist the Chair in the organization of the materials and discussion, and provide any other assistance requested by the Chair.
- Coordinate requests for consultations with STScI experts. A list of contacts and phone numbers for on-call experts will be available in each meeting room. A STScI specialist should be consulted on every significant feasibility or policy issue, and no proposal may be rejected on feasibility or policy grounds without such consultation. The reason for this is that we have had problems in prior cycles with peer reviewers making their own judgments that later turned out to be wrong. Any consultation must be recorded.
- The PSS should keep careful notes on any controversies, conflicts, or other exceptional issues, and the resolutions of them recommended by the panels, for subsequent consideration during the Director's Review. These notes should be of a quality such that they would be intelligible to someone else (e.g., SPG staff responding months later to appeals from rejected proposers). An example of such information is keeping track of who leaves the room for each proposal, due to a conflict of interest. Another example would be noting potentially prejudicial or extra-scientific remarks about proposers.
- Although it may be difficult, the PSS should completely refrain from making any scientific, technical, scheduling or policy comments, even if asked to do so by the Chair.
- Monitor comments (see below) progress via the Web-based system. Ensure that all comments are finished and signed off by the primary reviewers before they leave the meeting. Ensure that the Panel Chair and Co-Chair are aware of the comments status and proof-read the submitted comments as time permits.
- The voting procedure is that the panel first agrees on a specific allocation to vote for; then, on this basis, each member gives a numerical grade to the proposal in the Web-based Review System. The PSS must verify the orbit allocations are the same as entered in the Web Tool. The importance of care and accuracy in this activity cannot be overemphasized, since it is the only record of the panel's intent.
- After the voting stage follows a ranking stage. The final product of each panel will be a ranked list for all GO, SNAP, AR and Theory proposals. Ranks should be modified in the Web-Reviewer tool, as agreed upon by the panel. Grades can not be changed during the ranking stage.
Before the end of the Panel meetings on Wednesday, the PSSs will be asked to proof-read and edit the proposal comments to make them suitable for dissemination to the proposers. We will ask the Panel Chairs and Co-Chairs to proof-read the final comments as well. Use the following guidelines:
- All rejected proposals (those below the Cutoff line), should have a clear statement on why the proposal was "rejected".
- Correct any obvious scientific or technical errors. However, PSSs must not add qualitatively new information.
- Correct grammatical and typographical errors.
- Eliminate all offensive, gratuitous or personal remarks.
- Remove any explicit identifications of other Cycle 28 proposals or proposers, reviewers, or STScI staff. References to specific prior-cycle GO approved programs are allowed.
- Only those comments that contain substantive, useful technical or scientific information should be kept. Comments that contain only insubstantial or superficial remarks should be removed, whether for approved or unapproved proposals (since they create an erroneous impression of the review itself).
- Remove remarks that do not make sense because the reviewers expected the proposal to be approved, while in the end it was not. However, it is okay if a rejected proposal has positive remarks that are useful; that possibility is covered in the notification letter as a consequence of the heavy over-subscription.
- Make sure you verify in the proposals, any references to Proposals that are over the page limits and correct comments as appropriate.
- Make sure you identify any proposals to SPG that violate the Anonymous Review Guidelines.
- Consult with SPG on any questionable cases.
- Inform SPG when a complete set of the final, edited comments are available.
Specific PSS Duties
- Attend Panel Support Meetings as required.
- Familiarize yourself with the proposals before your panel.
- Handle any review conflict reassignments.
- Experiment with the Web-Based Reviewer System.
- Answer and resolve any questions brought up by the reviewers as needed.
- Assist Panel Chair as required prior to the actual meetings.
- Monitor the voting process, as document any issues about grading, conflicts of interest, etc., into the Web-Reviewer tool during the panel meetings.
- Summon assistance, as needed, from SPG (policy questions), Instrument Division (INS) (instrument questions) or OPB (implementation questions).
- Monitor adherence to all conflict-of-interest rules.
- Inform Brett Blacker that your panel is finished at the conclusion of your panel meeting.
- Review final Comments.
As a valuable member of the Cycle 28 Peer Review process, you are a critical part of the system and we value your comments and suggestions, and welcome feedback throughout the process.