Selection Criteria and Scoring System
Selection Criteria
Proposals reviewed by external panelists are subject to a single-phase review; proposals reviewed by the virtual topical panels are subject to a two-stage review process: 1) preliminary grading and triage; and 2) the review meeting. In all cases, panelists use the same scoring system.
Each topical panel covers a very broad science category, and each science category contains a number of narrower sub-fields. Ideally a proposal will be impactful to both the narrow sub-field of the proposal and to other sub-fields within the science category or in other science categories. Proposals will be assessed on an absolute scale against three primary criteria described in the Call for Proposals with a separate grade given for each. Each grade receives equal weight.
- Impact within the sub-field:
- The scientific merit of the program and its contribution to advancement of knowledge.
- Will the proposed program improve our understanding of the objects, classes of object, or specialist topics under study in the proposal? By how much? How relevant is the proposed work to the immediate sub-field of the proposal? Is the science within the sub-field timely?
- The immediate sub-field of the proposal is the niche area of the program, not the whole broad science area of the topical panel to which it was assigned. The evaluation should be based on what is written in the proposal, not on the reviewer's broader knowledge, even if the reviewer is an expert in the sub-field. Though, in most cases, the reviewer will not be an expert in the sub-field of the proposal, and the proposal should have been written accordingly.
- Out of field impact:
- The program’s impact for astronomy in general.
- Are there implications for other science areas and/or insights into larger-scale questions? Will the proposed program improve our understanding of science areas beyond the immediate sub-field of the proposal? How broad and how significant is this new understanding? Is the out-of-field impact timely?
- The proposal does not have to impact all of astronomy, but should ideally impact a number of other sub-fields or provide significant impacts in at least one other sub-field. The out-of-field impacts could be in other areas within the topical science panel of the proposal, or in other topical science areas. This evaluation should be based on what is written in the proposal, not on the reviewer's broader knowledge.
- Suitability:
- The necessity for HST observations, including special requirements, or relevance to HST science. The feasibility of the science program.
- For GO programs: Has the proposed program demonstrated that the unique capabilities of HST are required to achieve the science goals? How much of an advantage does HST data offer over other facilities? Has the time request been well justified? Have any special requirements been well justified? Have parallel observations been well justified? Have any duplications been well justified? If joint time was requested, have those additional observations been well justified? Does the observing plan make efficient use of resources?
- For AR programs: Has the proposed program demonstrated that the unique capabilities of HST are required to achieve the science goals? How much of an advantage does HST data offer over other facilities? Is HST the predominant source of data for the program? If data from other facilities will be used, has it's use been well justified.
- For Theory programs: Has the proposed program demonstrated a broad applicability to HST observational programs or data products? Will the proposed program provide results that assist in planning future HST observations, improve analysis of HST data or data products, or improve interpretation of HST data or data products?
- The necessity for HST observations, including special requirements, or relevance to HST science. The feasibility of the science program.
The final grade is the straight average of these values.
AR and GO calibration proposals are required to provide an analysis plan; reviewers should also consider the strength of the analysis plan in assessing the first two criteria.
Descriptions of additional criteria by type of proposal are given in the Proposal Selection Procedures section of the Call for Proposals.
While reviewing the proposals if you notice and/or identify any issues with the proposal template formatting, page limit violations or resource request issues, please contact SPG to discuss before downgrading that proposal.
Scoring System
Preliminary Grades and External Grading
The full set of criteria to apply in assessing different types of proposals are described in the Proposal Selection Procedures section of the Call for Proposals. Those criteria should be taken into account when grading each proposal.
The preliminary scoring and the grades from external panelists should be on an absolute scale with the framework set by the following criteria. Reviewers may submit grades in decimal form, but please limit to one decimal place.
Grade | Impact within the sub-field | Out-of-field impact | Suitability |
|---|---|---|---|
1 | Transformative advancement in the sub-field. The proposed program has the potential for transformative results in the immediate sub-field of the proposal. The program will transform understanding of the objects, class of objects, or specialist topics under study. | Transformative implications beyond the sub-field. The proposed program has transformative implications for one or more other sub-fields of astronomy. The impacts of the program are extremely broad and/or extremely significant. | Extremely well-justified. GO: The program's science goals can only be achieved with the proposed observations. Only HST is capable of collecting the required observations. Archival data may supplement the analysis, but is insufficient or does not exist in key areas, so new observations are needed. All aspects of the requested observations have been well justified and are necessary for the success of the science program. The observing plan uses resources very efficiently. AR: Analysis of archival HST data is critical to reach the stated science goals. The analysis may be supplemented by data from other observatories but the majority of the data will come from HST. Theory: The proposed theoretical or computational work will be transformative for planning future HST observing programs, for future analysis of HST data or data products, or interpreting results obtained from HST data or data products. |
2 | Major advancement in the sub-field. The proposed program has the potential for major advancement in the immediate sub-field of the proposal. The program will provide major advances in understanding of the objects, class of objects, or specialist topics under study. | Major implications beyond the sub-field. The proposed program has major implications for one or more other sub-fields of astronomy. The impacts of the program are broad and/or significant. | Well-justified. GO: The program's science goals will certainly be achieved with the proposed observations. HST offers major advantages over other facilities for collecting the required observations. The need for new data instead of or to supplement archival data is clear. Most aspects of the requested observations have been reasonably well justified and they will contribute to the success of the science program. The observing plan uses resources efficiently. AR: Analysis of archival HST data offers major advantages over data from other facilities to reach the stated science goals. The analysis may be supplemented by data from other observatories but a significant fraction of the data will come from HST. Theory: The proposed theoretical or computational work will offer major advances for planning future HST observing programs, for future analysis of HST data or data products, or interpreting results obtained from HST data or data products. |
3 | Moderate advancement in the sub-field. The proposed program has the potential for moderate advancement in the immediate sub-field of the proposal. The program will provide moderate advances in understanding of the objects, class of objects, or specialist topics under study. | Moderate implications beyond the sub-field. The proposed program has moderate implications for one or more other sub-fields of astronomy. The impacts of the program are moderate in scope and significance. | Some justification lacking. GO: The program's science goals will be achieved with the proposed observations. HST offers some advantages over other facilities for collecting the required observations. The need for new data over archival data is discussed but not compelling. Some aspects of the requested observations have been justified but other aspects are lacking adequate justification and it is not clear that all are needed. The observing plan could use resources more efficiently. AR: Analysis of archival HST data offers some advantages over data from other facilities to reach the stated science goals. The analysis may be supplemented by data from other observatories but some of the data will come from HST. Theory: The proposed theoretical or computational work will offer some advances for planning future HST observing programs, for future analysis of HST data or data products, or interpreting results obtained from HST data or data products. |
4 | Minor advancement in the sub-field. The proposed program has the potential for minor advancement in the immediate sub-field of the proposal. The program will provide minor advances in understanding of the objects, class of objects, or specialist topics under study. | Minor implications beyond the sub-field. The proposed program has minor implications for other sub-fields of astronomy. The impacts of the program are limited in scope and significance. | Many aspects poorly justified. GO: The program's science goals will probably be achieved with the proposed observations. HST offers minor advantages over other facilities for collecting the required observations. The need for new data over archival data is not discussed in depth or is not clear. Many aspects of the proposed observations have been poorly justified. There are questions about many aspects of the observing set up. The observing plan uses resources inefficiently. AR: Analysis of archival HST data offers minor advantages over data from other facilities to reach the stated science goals. The analysis will mostly use data from other observatories but small fraction of the data will come from HST. Theory: The proposed theoretical or computational work will offer minor advances for planning future HST observing programs, for future analysis of HST data or data products, or interpreting results obtained from HST data or data products. |
5 | Limited advancement in the sub-field. The proposed program has limited or no potential for advancement in the immediate sub-field of the proposal. The program will not advance understanding of the objects, class of objects, or specialist topics under study. | Limited implications beyond the sub-field. The proposed program has little or no implications for other sub-fields of astronomy. The impacts of the program are extremely limited in scope and significance. | Very poorly justified. GO: The program's science goals will not be achieved with the proposed observations or HST offers little to no advantage over other facilities for collecting the required observations. The need for new data over archival data is not discussed. The proposed observations are poorly justified or not explained at all. The observing set up is unlikely to be successful. AR: Analysis of archival HST data offers little to no advantage over data from other facilities to reach the stated science goals. The analysis will predominantly use data from other observatories with only very little or no data coming from HST. Theory: The proposed theoretical or computational work will offer little or no advance in planning future HST observing programs, for future analysis of HST data or data products, or interpreting results obtained from HST data or data products. |
Examples
The following examples aim to give guidance in applying these rubrics to grading proposals; reviewers should use their best judgement.
Case 1: UV observations of gas in young stars
In field | Highly significant improvement in our understanding of gas flow in young stars. | 1-2 |
Out of field | Potential for significant changes in our understanding of gas flows in a wide range of other environments. | 1-2 |
Suitability | UV observations are essential to achieve the science goals and can only be acquired through HST observations. | 1 |
Case 2: Analysis of archival near-IR imaging of a nearby galaxy for stellar population investigations
In field | Major advance in understanding stellar populations in that galaxy. | 2 |
Out of field | Some implications for stellar populations and stellar evolution in other galaxies. | 3 |
Suitability | The increased spatial resolution offered by HST provides some advantages over other facilities in addressing the science goals. The analysis offers significant improvements and/or additional value with respect to the original use of the data. | 2-3 |
Case 3: Optical/near-IR spectroscopy of an emission-line galaxy
In field | Moderate increase in understanding of prevalence of star formation in that galaxy. | 3 |
Out of field | Minor implications for the properties of other galactic systems, but no wider impact. | 4 |
Suitability | Only optical data are required for the science case; limited gains in performance at near-IR wavelengths as compared with larger ground-based facilities | 4-5 |
Case 4: Developing theoretical tools to characterize gas and dust in Galactic star-forming regions
In field | Potential significant increase in understanding of chemical composition in dusty environments. | 1-2 |
Out of field | Results have significant implications for interpreting dust composition in other galaxies. | 2 |
Suitability | The theoretical analysis will enable and support additional HST observational programs. | 2 |
The grades for each virtual panelist are normalized by SPIRIT to a mean value of 3.0 and a standard deviation of 1.0. The reviewer's grade for that proposal is the average of the three normalized grades.
The preliminary grade for each proposal is determined by averaging the overall grade from each reviewer. The preliminary grades are used to create a rank order list for each panel and the lowest-ranked proposals (typically ~40%) are triaged from further discussion.
Next: Proposal Feedback Comments