This is post 2 of 5 in the Broadening Participation and Broader Impacts Series

The terms broadening participation and broader impacts has been used extensively in science, technology, engineering, and mathematics (STEM) disciplines, especially within STEM funding solicitations and proposals. Although used interchangeably at times, these two terms have their own unique history and definitions. By understanding these differences and similarities, researchers, educators, and administrators can implement and evaluate them more successfully. In this five-part series we’re calling “Broadening Participation and Broader Impacts” we’ll explore the history of these terms, their implementation, and frameworks to evaluate their success.

Broader impacts – the potential to benefit society and contribute to the achievement of specific, desired societal outcomes. –PAPPG, NSF 19-1

Although most sources of funding, including both federal agencies and private foundations, expect a proposal to discuss the impact, significance or relevance of the proposed work, the National Science Foundation (NSF) is currently the only federal agency in the U.S. that has an explicit broader impacts requirement for their proposals. The following section summarizes the series of events that led NSF to create a broader impacts criterion.

Creation of the Broader Impacts Criterion

The National Science Board (NSB) was established by an Act of Congress in 1950 to serve as the independent governing board of the NSF. In this capacity, the NSB oversees the NSF’s proposal review process. Up until the 1980’s reviewers used two main criteria for evaluating proposals: the intrinsic scientific merit of the proposal, and the qualifications and competence of the principal investigator. To make the distinction between “basic” and “applied” research less rigid, NSB in 1981, adopted a merit-review standard for NSF proposals, which consisted of four criteria: 1) research performance competence of the principal investigator and supporting institution, 2) intrinsic merit of the proposed research, 3) utility or relevance of the research, and 4) effect of the proposed research on the infrastructure of science and engineering. 

In 1997, based on recommendations from the Committee on Equal Opportunities in Science and Engineering (CEOSE) which provides advice to the NSF on policies and programs that encourage full participation by women, minorities, and persons with disabilities in STEM, NSB simplified the merit review criteria for proposals from four to two – intellectual merit and broader impacts (NSB/MR-97-05, NSF News Release 97-028). NSF defined broader impacts using five general subcategories: 1) integrating research and education, 2) broadening the participation of underrepresented groups, 3) enhancing the infrastructure for research and/or education, 4) disseminating project results broadly to enhance understanding of science and technology, and 5) describing potential benefits to society at large.

As mentioned in our previous post, the inclusion of an explicit broader impacts criterion in the merit review process was to require proposers to address areas of societal concern within the context of the proposed activity, specifically the area of broadening participation of underrepresented groups within STEM (NSF Important Notice No. 125), a core value adopted by NSF in 1980. Yet, NSF also listed four other general activities that would fulfill the broader impacts criterion and these do not specifically address broadening participation of women, underrepresented minorities, and people with disabilities (NSF 08-062). This effectively dilutes the importance and attention paid to explicit broadening participation activities by NSF proposers and reviewers. In practice, this also creates general confusion around the broader impacts criterion, leading to varied and often conflicting interpretation.

Broader Impacts Criterion in Practice

NSF has a broad portfolio of programs that encompasses ten research areas, each with their own divisions and funding opportunities. Areas of support range from fundamental research, development and enhancement of resources, to education and workforce programs. As a result of these various programs, broader impacts may be intrinsic to the research itself, such as tornado research benefiting people living in high-tornado areas. While in others, the focus may be on education in STEM and both intellectual merit and broader impacts are inherent in the educational work. Individual funding solicitations may also more narrowly define broader impacts activities.

To assess the broader impacts criterion across these varied programs and contexts, the National Alliance for Broader Impacts (NABI) formed in 2014. NABI is a network of individuals and organizations working together to build institutional capacity, advance broader impacts, and demonstrate societal benefits of research. NABI sponsors annual summits with stakeholders to elicit feedback on the broader impacts criterion and recently produced the Broader Impacts Guiding Principles and Questions for National Science Foundation Proposals. In January 2018, NABI compiled data from their own annual summits, findings from NSF’s Office of Integrative Activities two-year study of broader impacts implementation, and NSF’s annual Committee of Visitors report on how the broader impacts criterion is being applied across programs and directorates. Their findings, published in the report,  “Current State of Broader Impacts: Advancing Science and Benefiting Society” lists seven common issues across all stakeholder groups:

1. Broader impacts criterion is unclear.
2. Random judgments on broader impacts are common in the merit review process.
3. Relative weighting of intellectual merit and broader impacts is not consistent; broader impacts is used by reviewers as a tie-breaker rather than a more substantial and equally weighted criterion.
4. It is unclear whether broader impacts need to be specifically related to the research aspects of the proposal.
5. Academic culture does not reward broader impacts activities and dissemination.
6. Resources to support broader impacts are lacking at the individual, institutional, and national levels.
7. Universities, governmental representatives, and non-academic partners need better ways to understand and communicate about broader impacts internally and externally to demonstrate research value.

Ironically, the majority of these issues were brought to the attention of the NSB-NSF Staff Merit Review Task Force prior to implementing the new two-criteria system in 1997 (NSB/MR-97-05). Specifically, a third of respondents during the public comment period brought up the “weighting or threshold” issue. Many expressed concern that adopting the new criteria will lead to a decline in NSF’s standards of excellence with “okay research” (i.e., intellectual merit) being upheld by “excellent relevance” (i.e., broader impacts). Others stated that, for research proposals, intellectual merit is much more important than broader impacts, and should be weighted accordingly. Still others criticized the broader impacts criterion as irrelevant, ambiguous, or poorly worded. To resolve this issue, the Task Force chose to include introductory wording for reviewers, stating that the two criteria need not be weighted equally but should depend upon either additional guidance received from NSF and/or the reviewer’s judgment of the relative importance of the criteria to the proposed work. The Task Force believed this was the best option because it wouldn’t polarize the research and education communities and could be applied very flexibly. Yet, this level of flexibility has resulted in random judgements and inconsistencies being applied to the broader impacts criterion as reported by NABI’s 2018 report referenced above.  

NABI is not the only group to raise concerns about NSF’s broader impacts criterion. As stated earlier, under the current rubric proposers do not need to address broadening participation explicitly in order to be compliant. This can lead to the continued exclusion of these groups over long periods of time. Various advisory and oversight bodies, including CEOSE in their Biennial Reports to Congress and participants in NSF-sponsored workshops on broadening participation and broader impacts projects (Clewell & Fortenberry, 2009) have insisted NSF weave broadening participation issues of diversity, equity, and accessibility into each of the five broader impacts categories (Fig. 1).

Broader Impacts Criterion Going Forward

The various advisory and oversight bodies mentioned above all provide recommendations to NSF concerning the broader impacts criterion, and although their focus is slightly different, their recommendations are not mutually exclusive. For example, COESE and other advisory boards want NSF to leverage the broader impacts criterion to support the agency’s core value of broadening participation. Others, such as NABI are generally focused on creating a cultural shift around the value of broader impact activities, both at the institutional and funding agency level. Collectively, integrating broadening participation issues into each of the five broader impacts subcategories, then incorporating NABI’s recommendations could greatly enhance the broader impacts criterion. Which, if any, of these recommendations NSF will ultimately choose to pursue is unclear. 

In the meantime, our third post in the Broadening Participation and Broader Impacts series will summarize successful approaches of funded broadening participation grant programs that can be utilized by broader impacts activities.

Dr. Cheryl L. Bowker

Associate Director – STEM Center

Cheryl has worked at the STEM Center since 2013 and has evaluated and managed several STEM education projects. Cheryl enjoys impassioned discussions about research, education, and insects. Check out the Staff page for contact details.