Social Science Validation and Validity Concepts

Humans in the Loop: Validation and Validity Concepts in the Social Sciences in the Context of Applied and Operational Settings.

Author | Editor: Cabayan, H. (Joint Staff), Ehlschlaeger, C. (ERDC), McGee, A. (Naval Postgraduate School) & Ackerman, G. (University of Maryland, START).

Threats in the 21st century are increasingly complex, requiring multiple perspectives and disciplines to understand and anticipate challenges. National security issues will require consideration of the insights that might be gained through social science analysis in order to provide broader understanding of both challenges and potential solutions. Many of the challenges and potential missions faced by the Department of Defense (DoD) will be multi-faceted in their most fundamental nature. To better understand these issues, their origins, and potential solutions, we need to think clearly about insights provided about the psychological and social dynamics impacting complex security problems. We can anticipate military operations and missions in a networked, dynamic global environment where modern media, the pace of technological change, and speed of events overlay often long standing historic social legacies and conflicts. We can turn to the social sciences to better understand the intersection of new technologies and legacies and, therefore, assist in crafting strategies to deal with current and emerging issues.

This white paper discusses the validity concepts and validation of the social sciences in the context of applied and operational settings. It focuses on a key issue: How do we gauge the degree to which our frameworks, models, and measures of human social behaviors correspond to the real issues with which DoD operators are concerned? It addresses these issues from several perspectives:

1. Scientific Validation in Social Science: What concepts are appropriate for assessing the “goodness” of the social science within the scope of DoD missions?
2. Determining Mission Applicability: Social sciences play roles at various phases of military planning. Will these necessitate varying degrees of validation?
3. The need to develop military social scientists to bridge the gap between social sciences as an academic discipline and their potential applications in strategic, operational, and tactical decision making.

Whether planning how to help train and develop a military unit, motivate an individual from another culture to participate in an activity, deter a nation-state from a course of action, stabilize a village, or persuade individuals to reject an extremist group, analytical efforts require thinking about the fundamental dynamics of people, groups, and societies. As a result, analysts, planners, and operators need to become more informed and active consumers of social science knowledge. This white paper is aimed at both motivating readers to confront that challenge and at providing some basic insights to reduce the scope of the challenge. Greater engagement with both the body of social science knowledge, and with the process of social science itself, has the potential to increase DoD effectiveness in an increasingly complex, less predictable, more connected, and more problematic world.

Use of the social science techniques in the DoD has a long pedigree. Yet the challenge remains over how to best leverage social science tools to support military operations.

Challenges in assessing reliability and validity of social science tools, techniques, and models, especially those developed for use in Information Operations, include the following.

1. Human and social science fields typically lack theoretical maturity as compared with the physical sciences. This challenges the accuracy or representation and the expertise of users.
2. Human social behavior often reflects a rich and complex problem space. This challenges the realism of the representation.
3. Human social behavior involves many unobservable phenomena. This challenges the expertise of subject matter experts and the credibility of the model to users.
4. Both psychological awareness and human behavior involve socially constructed factors and variables. This challenges the accuracy and realism of the representation.
5. Consistency in assumptions related to a model’s purpose when reusing scientific statements/theories that are encapsulated in software is essential.

Essentially, validation is about what works and what does not. At the operational level, people want workable tools to current problems. In this context, a “validated” theory has to be relevant to the operations community.

In addition, the white paper highlights the need for a cadre of military social scientists that understand both the academic and operational environments and act as a bridge between these two now very different environments. These individuals should be able to elicit user requirements and then employ their domain knowledge to satisfy those requirements. Therefore, there is a need develop a strategy as to how to best provide a focused, comprehensive, and integrated social science training program for officers and enlisted that spans their entire career.

The past decade of combat has once again brought the “human domain” of warfare front and center. The classic geographic factors of physical, cultural, economic, and political have demanded attention from the strategic to tactical level of range of military operations. Operational requirements have historically inspired innovation in organizations and diversification in cross-sector collaboration. This legacy can be used as a foundation for the growth of “military social science.”

The essential concern of the social sciences is human behavior. The classic disciplines (e.g., economics, political science, sociology, anthropology, etc.) loosely align with the levels— individual, group, societal—and the subjects of those behaviors. What social scientists do is attempt to describe and explain the influences and interactions among complex sets of factors that span human behaviors. This white paper continues the SMA White Paper series, The Role of the Social Sciences in DoD Mission Analysis and Planning, with a discussion of validity concepts and validation in the context of applied and operational settings. In so doing, it focuses on a key issue, namely: How do we gauge the degree to which our frameworks, models, and measures of human social behavior correspond to the real issues with which DoD operators are concerned?

Contributing Authors

Mr. Gary Ackerman (START, Univ. of MD); Mr. Ricardo Arias (SOUTHCOM); Dr. Allison Astorino-Courtois (NSI); Lt Col Alexander Barelka (USAF AFMC); Mr. David Browne (PACOM); Dr. Charles Ehlschlaeger (ERDC); Dr. Dana Eyre (SoSA); Ms. Laurie Fenstermacher (AFRL); Mr. Ben Jordan (TRADOC/G-2); Dr. Anne McGee (NPS); Ms. Jane Peplaw (SOUTHCOM); Mr. Eric Ruenes (DIA); Dr. Laura Steckman (PACOM); Dr. Ivan Welch (TRADOC/G-2) Editors: Dr. Hriar Cabayan (JS/J39); Dr. Charles Ehlschlaeger (ERDC); Dr. Anne McGee (NPS); Mr. Gary Ackerman (START, Univ. of MD)

 

Download Publication