The Challenge of Different Metrics

by Chad Hart
 

In the pursuit of research, scientists ask a series of questions and devise methods to address and answer those questions. The vast majority of these answers strive for statistical significance, showing evidence that the data support one statement over another. Economics is no different, in that we strive to show statistical significance for our work. However, there are times when we should recognize that statistical significance and economic relevance are not the same. There are times when statistically significant factors are not economically relevant (while the factor is definitely related to what we are studying, its impact is small enough that prices and/or returns do not change with it or do not cover the cost of employing the factor) and other times when economically relevant factors are not statistically significant. It is this second case I want to bring out in this article, as recent research has led me into one such case, which shows we may miss economically relevant choices if we only concentrate on statistical significance.

I have had the good fortune to work with agronomists and soil scientists here at Iowa State University and USDA-Agricultural Research Service on the subject of sulfur fertilization (Crespo et al. 2025). The original work for the study happened in 2006–2017 with field trials testing the impacts of various sulfur fertilizer products on corn growth and production in central Iowa. Statistical analysis of the data shows very limited statistical significance. For example, for a sulfur fertilizer known as AMS (ammonium sulfate), out of the 12 years of studies, AMS application only showed a statistically significant yield increase in one year. Table 1 displays the average yields from the field trials of the control plots (those without sulfur fertilizer) and the AMS plots, along the yield differences. Only in the 2014 growing season was the yield difference large enough to be statistically significant. However, as the data show, in half of the years, the yield difference between the AMS and control plots was greater than 5 bushels per acre. In fact, the average yield difference across all years is 6.4 bushels per acre.

Given these sizable yield differences, it made sense to approach the analysis in a different way, exploring the economic impacts of the fertilization. While the yield differences were not statistically different from zero in this study (with the exception of 2014), the new question is whether the yield differences were large enough to create positive economic returns. Do the yield differences capture enough return to consistently cover the cost of application and provide profit?

To explore that question, we captured the prices for corn and various fertilizers as of September 2024 to compute the financial costs and benefits of sulfur fertilization outlined by the field trials. As the sulfur fertilizer was applied during the corn planting pass, the additional cost above the control plots is only the cost of the AMS itself. Table 2 details the economic results by year and averaged across years. On average, there was a positive return of sulfur fertilization of $11.16 per acre. While the return to sulfur fertilization is variable, half of the study years showed positive returns. So based on these results, farmers who apply AMS would only capture a positive return from sulfur half of the time, but those returns not only cover the additional costs, but also provide a profit margin.

To ensure this was not just an artifact from the September 2024 prices, we also computed the costs and benefits based on historical (2006–2014) corn and AMS prices from USDA-ERS (2019). The results were fairly consistent. Under the various pricing situations, AMS applications were profitable 50%– 75% of the time. The average returns from the AMS applications ranged from $6.65 per acre (given prices from 2014) to $31.22 per acre (given prices from 2012). While we can not say sulfur fertilizer has been statistically shown to increase corn yields, we can say sulfur fertilization is correlated with positive returns at least half of the time and the average net return from the fertilizer applications has been several dollars per acre.

In science, we rely on the standard of statistical significance to define meaningful relationships among factors. But the challenge of finding statistical significance is also linked to the size of the experiment, with smaller experiments resulting in larger standard errors, requiring larger data differences to show significance. In our case here with sulfur fertilizer, yield differences often needed to be larger than 10 bushels per acre to claim statistical significance. However, in farming or in business, smaller yield differences can still be meaningful and profitable. Given the corn and fertilizer prices in September 2024, a 3.7 bushel-yield difference would create a profit given the sulfur fertilizer application and a 10-bushel yield difference would lead to a $25.83 per acre profit for the farmer. This is a case where the statistics did not show significance, but the economics argue that we should keep checking.

References

Crespo, C., J.L. Kovar, C.E. Hart, R.T. Roth, P.L. O’Brien, and S.J. Ruis. 2025. “Long-term Field Study on Corn response to sulfur fertilization in Iowa, USA.” Field Crops Research 330, forthcoming. https://doi.org/10.1016/j.fcr.2025.109990.
United States Department of Agriculture Economic Research Service (USDA-ERS). 2019. “Fertilizer Use and Price.” Extracted from: https://www.ers.usda.gov/data-products/fertilizer-use-and-price/.

Suggested citation

Hart, C. 2025. “The Challenge of Different Metrics.” Agricultural Policy Review, Spring 2025. https://agpolicyreview.card.iastate.edu/spring-2025/challenge-different-metrics

For a richer discussion of economic versus statistical significance, see McCloskey, D. N. and S. T. Ziliak. 1996. "The Standard Error of Regressions." Journal of Economic Literature 34(1): 97-114. https://www.jstor.org/stable/2729411.