Chapter 13: Lessons Learned and Suggestions for Future Analyses

Abstract

Although significant effort went into developing the cost sheet and stakeholder questionnaire, important lessons were learned that could benefit future studies. For example, the time and effort needed for State Geological Surveys (SGS) and the U.S. Geological Survey (USGS) to complete the cost sheets were significantly underestimated. Although federal funding through the National Cooperative Geologic Mapping Program was relatively easy to track, other sources of funding were more challenging to tally. Partitioning annual funding was challenging as many of the responses included funding spanning multiple years. Cost reporting was also not consistent over the 26-year period, with the earlier years posing the greatest challenge. Reported costs for geological mapping should therefore be considered a minimum but probably are not significantly lower than actual. Two major omissions on the cost sheet included requests for: (1) the number of geological maps produced at various scales relative to funds allocated, which precluded estimates on the average cost of generating a geological map at a specific scale; and (2) the number of maps sold or downloaded. Some of these data were obtained later, which extended the time needed for the study. The stakeholder questionnaire was comprehensive to capture the many user groups and applications of geological maps for the entire nation. However, many considered it to be too long with too many questions. Also, some respondents worked in many parts of the United States or in more than one sector, so it was not easy to apply answers to an individual state, region, or sector. Other complications included the possible lack of knowledge by some respondents of the complex processes required to generate geological maps coupled with the long-standing tradition of such maps existing as a public good. Thus, estimating the willingness to pay for a geological map was difficult for some respondents, which is compatible with the wide range in estimates. Narratives were requested on eight questions to help clarify responses, but 14,000 individual descriptions led to a lengthy process of evaluating these results.

Lessons learned from this study may enable future, more robust economic analyses of geological mapping. Major elements for any such study include: (1) cost information from SGS and the USGS; (2) sampling the many sectors of society that use and benefit from geological maps; (3) crafting a questionnaire on perceived benefits broad enough to apply to diverse sectors yet short enough to facilitate completion; and (4) obtaining data on geological map demand and use. Although cost sheets should be kept simple and focused on costs to allow for timely completion, they could include data on the number of maps produced as well as information on how many were downloaded and sold. SGS and the USGS should be encouraged to maintain datasets on the costs while tracking map downloads, views, and sales to facilitate future analyses, tout their programmatic impact, and support state or federal funding requests. For evaluating the benefits of geological mapping, the development of questionnaires should incorporate statistical and economical proficiency, in addition to geological expertise, to ensure more robust results. The questionnaire should also undergo thorough beta testing prior to distribution to avoid any confusion in meaning or intent of specific questions. Significant resources should be dedicated for targeted outreach to ensure higher return rates. Although important to include at some level, narrative responses should be minimized. A national study for a country as large and complex as the U.S. will, however, incur challenges in adequately covering the diversity between regions and sectors in a single questionnaire. Future studies may therefore wish to consider options to narrow emphasis, including: (1) employing control groups for sampling various sectors; (2) focusing on particular economic sectors or geographic regions; (3) developing separate questionnaires for different user groups, economic sectors, or regions; and (4) focusing on specific types of geological maps and derivatives (e.g., 3D maps).

13.1: Introduction

This report is the first national study on the costs and benefits of geological mapping in the U.S. As described in Chapter 2, detailed cost sheets were distributed to State Geological Surveys (SGS) and the U.S. Geological Survey (USGS) to assess the costs of geological mapping (Appendix 1) while a comprehensive questionnaire was sent out to more than 81,000 stakeholders to capture the perceived value and benefits of geological maps (Appendix 2). Although significant effort went into developing and distributing the cost sheet and questionnaire, some modification to both the approach and content of these documents would probably have yielded more robust results. In this chapter, we describe the lessons learned in our analysis and provide suggestions for future studies. We address the cost sheets that provided expense data for producing geological maps, assess the stakeholder questionnaire and perceived benefits of the maps, review some of the major concepts and takeaways of the entire study, and conclude with specific suggestions for future analyses.

13.2: Data from Cost Sheets

The cost sheet requested information from the SGS and USGS on funds expended annually for mapping from 1994–2019 from state, federal, and other sources, the percentage of each state mapped at various scales, and the availability of different types of derivative maps within individual states (Chapter 2 and Appendix 1). Federal funds allocated to the states through the STATEMAP component of the National Cooperative Geologic Mapping Program (NCGMP) were easy to obtain, but levels of funding from other sources were difficult for many states to document. While the NCGMP 1:1 match requirement for SGS also was relatively easy to obtain, many SGS “over-matched” the federal funds but maintained poor records of doing so. Fortunately, USGS records showed some of this SGS matching information, but not all. Cost reporting capabilities were not consistent among SGS nor consistent through time for individual SGS for the 26-year period covered by the cost sheet. Not every SGS was able to provide all of the required data because of a lack of resources, insufficient staffing, and/or lack of record keeping. Similarly, the USGS had challenges in assessing funds allocated to geological mapping outside of the FEDMAP and STATEMAP programs of the NCGMP. Thus, the reported costs by SGS and the USGS should be considered minimums but probably are not significantly lower than the actual costs. Also, parsing out mapping expenditures into particular years presented challenges, since most projects (even STATEMAP and FEDMAP projects) spanned more than one year, and fiscal years for most states differed from the federal government. Providing information on the percentage of a state mapped at various scales and availability of multiple types of derivative maps also were particularly challenging for many SGS, and thus many were unable to report such data. Even for those states that did have such data, questions arose as to what to report. For example, should all maps at 1:24,000 scale be included in the estimate or should only those maps still considered useful be included? Because of the various complications regarding the information requested of SGS and the USGS, we underestimated the time required by them to provide all types of data requested on the cost sheets, as this generally required careful analysis of past budgets and projects, much of which predated the digital age and commonly preceded the tenure of the current administrative staff.

Two major omissions in the cost sheet that would have provided valuable information included requests for the number of geological maps produced at various scales relative to funds allocated to mapping in a given year and the number of maps that were sold or downloaded. The lack of data on the quantity of maps produced by SGS in a specific year precluded estimates on the average cost of generating, for example, a 1:24,000-scale map, at least from the original questionnaire. However, the number of maps produced from STATEMAP by SGS was eventually provided by the USGS, and thus average costs per map were ultimately estimated for representative states in Chapter 8. As described in Chapter 7, information on the number of maps sold and downloaded was requested after the cost sheets were submitted. For most SGS and the USGS, digital copies of maps can be viewed and downloaded for free from their websites, while hard copies incur a minimal cost. For both digital copies downloaded and hard copies sold, such numbers are not tracked by many SGS, and it was discovered that web crawlers (i.e., robotic action or bots) further complicated estimates for downloaded copies. The USGS and about half of the SGS were able to report reliable estimates for the impact of web crawling bots. Estimates for both downloaded and sold copies of geological maps are clearly minimums. Requesting these data earlier in the process would have resulted in more robust estimates.

13.3: Benefits Data and Stakeholder Questionnaire

The questionnaire requested information on the general benefits and perceived value of geological maps from users as well as information on their profession (e.g., private vs. public sectors, type of industry, type of government organization, etc.). Both quantitative and narrative answers were sought for a series of 25 questions (Chapter 2 and Appendix 2). The questionnaire was sent to over 81,000 individuals, with nearly 4,800 responses returned.

Significant feedback was received regarding the content of the questionnaire. Although the questionnaire was generally considered comprehensive, many commented that there were too many questions and that some questions were too long. Notably, some respondents provided unreliable or no answers to some of the questions. The intent of this economic analysis was a nationwide study but also to evaluate differences between various economic sectors and geographic areas (regions of the U.S. and/or individual states). Once questionnaire responses were summarized, it was obvious that many respondents worked in numerous professional, commercial, and industrial sectors, which precluded attributing their responses to specific sectors. Also, some respondents worked in many parts of the U.S., which made it difficult to apply their answers to specific states or geographic regions. Therefore, as the study progressed, time did not permit a detailed evaluation of geographic regions or states beyond the discussion of general regional differences in Chapter 8 and identifying sectoral contributions to per capita GDP in Chapter 11. Future studies that further break down the data acquired in this study might reveal additional insight into regional, state, and sectoral differences or similarities.

In addition, the wording of some questions could have been more discrete in defining whether information on costs or benefits was being sought. For example, there may have been some confusion as to whether long-term value referred to in question 10 and an estimate of what one would expend on a map in question 17 implied benefits or estimates of costs. However, the median responses for these questions aligned much more closely with the willingness to pay (Chapter 6) as opposed to the estimated costs per map (Chapters 4 and 8), so confusion on the intent of these questions may have been minimal. Nonetheless, more discrete wording would have facilitated a more direct interpretation of the results and perceived intent of those filling out the survey, allowing the data analysis to be clearer and more reproducible.

Another possible issue was the lack of knowledge by some respondents of the general costs of geological mapping while providing estimates of perceived value or willingness to pay for these maps. The nearly 4,800 respondents came from a wide range of backgrounds, sectors, and geographical regions, and thus their knowledge of the production process for geological maps probably varied extensively, which imparted potential bias in their responses.

In addition to potential impacts from the lack of understanding of producing geological maps, additional bias may have been imparted by different perspectives between the public and private sectors. Those in the private sector may have viewed geological maps from more of a market or for-profit perspective, whereas those from the public sector may have viewed them strictly as a public good, thus imparting differences in perceived value of such geological maps. In addition, perceived value and a long-standing “culture” of SGS and the USGS providing geological maps and related information at no or very low cost perhaps affected some private and public sector respondents from divulging their ability or willingness to pay. This was evidenced by many saying that they would pay nothing or a very small amount for a geological map. However, the opposite was also true, as several others (obviously for very large projects) were willing to pay millions. It is, for this reason, that we chose to report median rather than mean results to all of our questions (Table 6.5.1).

Another lesson learned regarding the stakeholder questionnaire was the lengthy process to evaluate the overwhelming response to eight text-based narrative questions (e.g., “Please describe an example of […]” or “Provide additional comments on […]”) and their associated ~700 pages of information, ~14,000 individual responses, and an average of 26 words per response. This required manually reading and categorizing 15% of the responses for each question, initiating lists of keywords, and then applying word-use frequency to generate additional predictive keywords. An automated procedure resulted in the categorization of up to 90% of the responses, with remaining outliers categorized manually. However, reflecting on the time and effort spent with these eight text-based questions, rephrasing the questions and providing “discrete selection categories” would have been considerably more efficient.

13.4: Broader Concepts and Takeaways

Assessment of the costs and benefits of geological maps is complex and requires a broad approach to capture the diverse uses of such maps, but it should also provide sufficient detail to yield quantitative data on the costs incurred by SGS and the USGS and the resulting spectrum of benefits to the many user groups. The cost of producing geological maps is the sum of the equipment, travel, labor, and analyses needed to complete: (1) the fieldwork; (2) map compilation; (3) scientific analyses; (4) cartography; and (5) publication of the map. The benefits of a geological map are the integrated value of the scientific data, analyses, and interpretations, along with the consideration that the said analyses and interpretations are not possible without the map.

Geological maps produced by SGS and the USGS operate as an intermediate public good, which has an eternal diminishing unit cost with additive value with each use by public or private users. The effective value of a map is directly related to the sufficiency of an individual map, which hinges on its detail, potential applications associated with its location, and the scientific competence of the mapmakers. Geological maps produced by SGS and the USGS typically meet high levels of scientific criteria and are of sufficient detail to be recognized as the gold standard by map users. However, the actual value of an individual geological map may vary significantly depending on its location. For example, a detailed geological map of a wilderness area may have less overall value to society compared to a map within or adjacent to a major urban area or that containing appreciable mineral or energy resources that can be developed in an environmentally sound manner.

Private entities have a financial threshold for producing geological maps, constrained by expected returns on investment while also considering the sufficiency of the public maps to meet their economic decision-making needs. Interestingly, statistical analyses in this study showed consistent behavior among respondents across the U.S., with some regional variations, such as higher demand for detailed maps in the northeastern U.S. Application of geological maps clearly provides value in economic decision-making in a large number of economic sectors in the U.S. Public good maps provide access to less capitalized players who have a limited capacity to invest. Likewise, limits on the capacity to invest for users can lead to suboptimal decisions when access to sufficient geological maps for decision-making is unavailable and economic decisions are based on risk assumptions rather than factual data.

An important underlying thread to the above discussion and this report in general is that this study is an economic analysis of an intermediate public good that impacts many segments of society and realizes maximum value with expert use. Although the results of this analysis are meaningful and consistently demonstrate a high value of geological maps produced by SGS and the USGS to society, the analysis may not be as straightforward as compared to products fully produced within the private sector that are generally driven by market conditions. Although the wide variation in respondents to the questionnaire has the advantage of sampling a broad spectrum of society in terms of perceived benefits, the varying backgrounds and lack of control groups amongst the respondents also resulted in some challenges for interpreting the results, as mentioned above. This begs the question as to how the lessons learned in this study could be applied to enable future analyses that would yield more statistically robust results?

13.5: Suggestions for Future Studies

In considering the lessons learned from this project in designing future studies, we are faced with some of the same questions recognized by the steering committee in the initial discussions of how best to approach this study. These include:

• How can the project design be broad enough to sample the many segments of society, including a wide array of both private and public sectors that utilize and benefit from geological maps?

• How can a questionnaire seeking information on perceived benefits be crafted for these many diverse sectors and yet be simple enough to be easily understood and completed in a timely fashion?

• How can the costs for geological mapping be easily obtained from SGS and the USGS?

• For a study as broad as this national assessment, what are the best means by which to assess geological map demand and use?

These questions remain for any subsequent studies, but with the knowledge gained from this study, how would we design a future analysis to avoid some of the challenges faced in this study while providing for and facilitating far-reaching results.

Changes to the cost sheet are the most easily addressed. In future studies, the cost sheets could be simplified to exclusively focus on funds expended for geological mapping from federal, state, and other sources. Data on the number of maps produced at certain scales should also be requested to facilitate estimates of the average cost per geological map. SGS and the USGS should be encouraged to compile such data in the coming years such that relevant data are readily available for future studies. However, information on the percentage of states mapped at various scales and availability of derivative maps can be the subject of other studies.

It is important to note that the economic analyses conducted in this report are based on traditional geological mapping techniques — ​that is, the representation of the subsurface is conveyed to the user in two dimensions (e.g., as a paper or digital map). With the emergence of digital technology, the science of geological mapping, like all Earth science research, has progressed into three dimensions or 3D data. The added dimension opens up many new opportunities for private and public sector applications that will benefit society in the future. It would therefore be useful if SGS and the USGS could parse out regions where 3D geological maps and models have been completed, and then evaluate their specific associated costs and benefits, both of which should be proportionately larger than that portrayed in the present economic analysis. Much higher data acquisition costs (e.g., drilling and geophysics) are required. However, even higher overall benefits are anticipated as society will be adjusting to uncertain climate change scenarios and transitioning to a “greener economy”, both of which will rely on a more robust understanding and depiction of Earth’s subsurface. This transition is already occurring as evidenced by recent increased interest by federal agencies and industry for energy storage, identification of buried critical minerals and geothermal resources, and delineation and modeling of groundwater resources. Thus, it is important to strongly advocate for increased funding for data acquisitions that will elucidate the subsurface (e.g., gravity and magnetic data), which will complement surficial data and allow for more widespread applications of 3D geological mapping.

Some of the most difficult information to obtain was online map download, view, and sales data from SGS and the USGS. It would be beneficial to all SGS and the USGS to consistently track these data, while at the same time account for bot activity that can significantly skew web statistics. Currently, very few SGS account for bot activity. Commonly, all of these statistics are most readily available at the end of calendar years. For any future assessment on the value of geological maps and related information, these data provide a metric of demand that SGS and the USGS can tout as showing significant programmatic impact.

How best to assess the value and benefits of geological maps is much more complex. At the root of this challenge is ensuring that appropriate statistical and economic expertise complement the geological proficiency in project design and development such that the questionnaires/surveys will yield more statistically robust results. We recognize that the steering committee for this project was weighted too heavily on geological expertise. In addition to a more scientifically diverse steering committee, more salient results may require: (1) thorough beta testing of the questionnaires; (2) separate questionnaires for different user groups; (3) implementation of control groups for sampling of different user sectors, including some groups that have a clear understanding of the process of producing geological maps; and (4) more targeted outreach to ensure a higher return rate on the questionnaires, which will be much easier to accomplish if not in the midst of a pandemic, as was the case for the present study. In addition, designing questions for stakeholders that better emphasize state-to-state and sectoral contributions is recommended to facilitate more detailed geographic and sectoral analyses.

Although care should be taken to keep future questionnaires as concise as possible, there are a number of research questions that could be posed to enhance understanding of the needs of various user groups and/or more broadly be analyzed by research teams. These include the following:

• How might the integration of scientific data, analysis, and interpretation in geological maps be optimized for various applications and users?

• To what extent does the value of a geological map diminish if any of its core components (data, analysis, interpretation) are compromised or missing?

• What are the specific economic and societal benefits of ensuring that geological maps are produced and maintained as a public responsibility?

• What is the value of updating a geological map? As technology, interpretation, and accuracy improve, the quality of a map, if updated, also increases, but how frequently and at what scale should maps be updated based on costs, value, and necessity?

• What factors determine the threshold for private entities to invest in the production of geological maps, and how might this threshold be increased? Is this a function of evolving applications of geological maps over time and as the economy changes?

• Are there regional-specific needs for geological maps that have not been addressed by the current models, and how can these needs be incorporated into future production?

• How do variations in demand for geological maps, such as the heightened demand for detailed maps in the northeastern U.S., or the need for detailed 3D subsurface maps and models, affect the overall economic value of these maps?

• How might the accessibility and availability of geological information influence other sectors of the economy, beyond the ones currently studied, especially by increasing the number of entities that can utilize geological maps within their capacity to invest.

• What is the potential value of studying the impact and role of geological maps at finer granular spatial levels as they apply to varied intensity of sectoral activities.

This long list of relevant research questions for future studies demonstrates the complexity of any such economic analysis, as well as the challenges facing any group attempting to assess the costs and benefits of geological mapping in a single study. Although the results of this study reflect strongly on the high value of geological mapping to many public and private sectors throughout the U.S., it may be most prudent for subsequent studies to initially dissect the analysis into discrete user sectors and geographic regions prior to pursuing an all-inclusive national study.