Empirical Test Guidelines for Content Validity: Wash, Rinse, and Repeat until Clean

2020 | Communications of the Association for Information Systems | Citations: 15

Authors: Schmitz, Kurt; Storey, Veda C

Abstract: Empirical research in information systems relies heavily on developing and valid ... Expand

Abstract: Empirical research in information systems relies heavily on developing and validating survey instruments. However, researchers’ efforts to evaluate content validity of survey scales are often inconsistent, incomplete, or unreported. Thjs paper defines and describes the most significant facets of content validity and illustrates the mechanisms through which multi-item psychometric scales capture a latent construct’s content. We discuss competing methods and propose new methods to assemble a comprehensive set of metrics and methods to evaluate content validity. The resulting recommendations for researchers evaluating content validity emphasize an iterative pre-study process (wash, rinse, and repeat until clean) to objectively establish “fit for purpose” when developing and adapting survey scales. A sample pre-study demonstrates suitable methods for creating confidence that scales reliably capture the theoretical essence of latent constructs. We demonstrate the efficacy of these methods using a randomized field experiment. Collapse

Topics: self efficacy psychological ownership evaluation criteria human resource management

Methods: survey experiment survey design theory development field experiment

Toward an Understanding of Online Lending Intentions: Evidence from a Survey in China

2015 | Communications of the Association for Information Systems | Citations: 7

Authors: Chen, Dongyu; Lou, Hao; Van Slyke, Craig

Abstract: The online peer-to-peer lending marketplace has experienced rapid growth since i ... Expand

Abstract: The online peer-to-peer lending marketplace has experienced rapid growth since its inception in 2005. It has played a significant role in helping small and micro-enterprises resolve financing problems. However, this marketplace is still in its infant stage. To better understand the lending activities associated with peer-to-peer lending, we need theoretically grounded empirical research. In this study, we investigate the perceptual drivers of online lending from the perspective of lenders. We empirically test our research model with survey data collected from 217 lenders in a major online peer-to-peer lending website in China. Our results reveal that trust was the most critical determinant of willingness to lend. Perceived information quality was important in mitigating perceived risk and enhancing trust, and perceived social capital impacted trust in borrowers. Furthermore, perceived risk did not significantly influence lending willingness, but had a negative impact on trust. These findings indicate that transaction behaviors in the peer-to-peer market may not be the same as that in the purchase-oriented e-commerce settings. We conclude by discussing the study’s implications for research and practice along with the its limitations. Collapse

Topics: crowdsourcing peer-to-peer lending social capital electronic commerce data quality

Methods: survey partial least squares path modeling factor analysis statistical hypothesis test theory development

Theories: lemon market theory trust theory

Is Negative Attention Better Than No Attention? The Comparative Effects of Ostracism and Harassment at Work

2015 | Organization Science | Citations: 0

Authors: O'Reilly, Jane; Robinson, Sandra L.; Berdahl, Jennifer L.; Banki, Sara

Abstract: Ostracism has been recognized as conceptually and empirically distinct from hara ... Expand

Abstract: Ostracism has been recognized as conceptually and empirically distinct from harassment. Drawing from theory and research that suggests that employees have a strong need to belong in their organizations, we examine the comparative frequency and impact of ostracism and harassment in organizations across three field studies. Study 1 finds that a wide range of employees perceive ostracism, compared with harassment, to be more socially acceptable, less psychologically harmful, and less likely to be prohibited in their organization. Study 2 surveyed employees from a variety of organizations to test our theory that ostracism is actually a more harmful workplace experience than harassment. Supporting our predictions, compared with harassment, ostracism was more strongly and negatively related to a sense of belonging and to various measures of employee well-being and work-related attitudes. We also found that the effects of ostracism on well-being and work-related attitudes were at least partially mediated by a sense of belonging. Study 3 replicated the results of Study 2 with data collected from employees of a large organization and also investigated the comparative impact of ostracism and harassment on employee turnover. Ostracism, but not harassment, significantly predicted actual turnover three years after ostracism and harassment were assessed, and this was mediated by a sense of belonging (albeit at p < 0.10). Implications for theory, research, and practice are discussed. Collapse

Topics: IT personell turnover job satisfaction logistics management database system electronic market

Methods: statistical hypothesis test survey structural equation modeling theory development survey design

Theories: big five model

An Extended Perspective on Individual Security Behaviors: Protection Motivation Theory and a Unified Security Practices (USP) Instrument

2014 | ACM SIGMIS Database | Citations: 0

Authors: Crossler, Robert; Bélanger, France

Abstract: Security threats regularly affect users of home computers. As such, it is import ... Expand

Abstract: Security threats regularly affect users of home computers. As such, it is important to understand the practices of users for protecting their computers and networks, and to identify determinants of these practices. Several recent studies utilize Protection Motivation Theory (PMT) to explore these practices. However, these studies focus on one specific security protection behavior or on intentions to use a generic measure of security protection tools or techniques (practices). In contrast, this study empirically tests the effectiveness of PMT to explain a newly developed measure for collectively capturing several individual security practices. The results show that PMT explains an important portion of the variance in the unified security practices measure, and demonstrates the importance of explaining individual security practices as a whole as opposed to one particular behavior individually. Implications of the study for research and practice are discussed. Collapse

Topics: IT security wireless computer network self efficacy electronic mail information system use

Methods: survey qualitative interview partial least squares regression survey design data transformation

A Rasch Approach to Empirical Field Research in Information Systems

2012 | Americas Conference on Information Systems | Citations: 0

Authors: Fraser, Steven

Abstract: The traditional method for construct development in information systems (IS) is ... Expand

Abstract: The traditional method for construct development in information systems (IS) is based on classical test theory (CTT). While the Rasch approach is more theoretically sound than CTT, it has not been widely adopted in the IS literature. This is probably because CTT and Rash often produce similar results but Rash it is more difficult to perform. However, test equating, an important aspect of Rasch analysis, provides IS researchers with an effective way to develop complex IS constructs such as those that are formative in nature. The current wisdom is that formative constructs should be determined as part of a nomological network (see Petter, Straub and Rai, 2007), but this approach has been shown to be problematic (Edwards, 2010). The approach taken in this study is to treat formative constructs as second order reflective with composites based on the component dimensions (Treiblmaier, Bentler, and Mair, 2010). These notions are illustrated by teasing out the components of the formative construct Interactive Project Control (IPC) in software development projects using a Rasch equating approach. Collapse

Topics: IT development project project productivity JD Edwards systems development

Methods: survey classical test theory exploratory factor analysis data transformation descriptive statistic

INFORMATION SYSTEM EXPERTS: DEFINITION, IDENTIFICATION, AND APPLICATION

2011 | European Conference On Information Systems | Citations: 0

Authors: Dey, Sharmistha; Sedera, Darshana

Abstract: Experts’ views and commentary have been highly respected in every discipline. Ho ... Expand

Abstract: Experts’ views and commentary have been highly respected in every discipline. However, unlike traditional disciplines like medicine, mathematics and engineering, Information System (IS) expertise is difficult to define. Despite seeking expert advice and views is common in the areas of IS project management, system implementations and evaluations. This research-in-progress paper attempts to understand the characteristics of IS-expert through a comprehensive literature review of analogous disciplines and then derives a formative research model with three main constructs. A validated construct of expertise of IS will have a wide range of implications for research and practice. Collapse

Topics: systems evaluation knowledge type evaluation criteria IS impact IS discipline

Methods: Student's t-test literature study classification method classical test theory

CONSTRUCT VALIDITY IN PARTIAL LEAST SQUARES PATH MODELING

2010 | International Conference on Information Systems | Citations: 0

Authors: Rönkkö, Mikko; Ylitalo, Jukka

Abstract: Partial least squares path modeling (PLS) has seen increased use in the informat ... Expand

Abstract: Partial least squares path modeling (PLS) has seen increased use in the information systems research community. One of the stated key advantages of PLS is that it weights the indicator variables based on the strength of the relationship between the indicators and the underlying constructs, which presumably decreases the effect of measurement error in the analysis results. In this paper we argue that this assumption is not valid. While PLS indeed does weight the indicators to maximize the explained variance, it does this by including error variance in the model thus reducing construct validity. We use a simulation study of a simple PLS model to show that when compared to traditional sum scale approach, PLS estimates are actually often less valid. Although our study has its limitations, it hints that the use of PLS as a theory testing tool should be reevaluated and that more research testing the effectiveness of the PLS approach is in order. Collapse

Methods: partial least squares regression structural equation modeling simulation Monte Carlo simulation factor analysis

Theories: measurement theory

Formative vs. Reflective Measurement: Comment on Marakas, Johnson, and Clay (2007)

2008 | Journal of the Association for Information Systems | Citations: 0

Authors: Hardin, Andrew M.; Chang, Jerry Cha-Jan; Fuller, Mark A.

Abstract: In a recent issue of the Journal of the Association for Information Systems, Mar ... Expand

Abstract: In a recent issue of the Journal of the Association for Information Systems, Marakas, Johnson, and Clay (2007) presented an interesting and important discussion on formative versus reflective measurement, specifically related to the measurement of the computer self-efficacy (CSE) construct. However, we believe their recommendation to measure CSE constructs using formative indicators merits additional dialogue before being adopted by researchers. In the current study we discuss why the substantive theory underlying the CSE construct suggests that it is best measured using reflective indicators. We then provide empirical evidence demonstrating how the misspecification of existing CSE measures as formative can result in unstable estimates across varying endogenous variables and research contexts. Specifically, we demonstrate how formative indicator weights are dependent on the endogenous variable used to estimate them. Given that the strength of formative indicator weights is one metric used for determining indicator retention, and adding or dropping formative indicators can result in changes in the conceptual meaning of a construct, the use of formative measurement can result in the retention of different indicators and ultimately the measurement of different concepts across studies. As a result, the comparison of findings across studies over time becomes conceptually problematic and compromises our ability to replicate and extend research in a particular domain. We discuss not only the consequences of using formative versus reflective measures in CSE research but also the broader implications this choice has on research in other domains. Collapse

Topics: spreadsheet self efficacy online learning system word processing internet technology

Methods: PLS tool scale validation classical test theory Student's t-test latent variable model

Formative vs. Reflective Measurement: A Reply to Hardin, Chang, and Fuller

2008 | Journal of the Association for Information Systems | Citations: 8

Authors: Marakas, George M.; Johnson, Richard D.; Clay, Paul F.

Abstract: The article presents a response on the views of Joseph Joubert, an essayist and ... Expand

Abstract: The article presents a response on the views of Joseph Joubert, an essayist and moralist, which says that it is better to debate a question without settling it. It focuses on the formative versus reflective measurement issues in which a detailed response to the specific concerns raised from both theoretical and statistical basis. It then presents the views of author Hardin and offers insights intended to provoke additional debate and statistical basis and notes that formative items are not created, they were selected as the development reflected measures. Collapse

Topics: self efficacy organizational productivity knowledge representation

Methods: simulation latent variable model ontological modelling classical test theory

Theories: theory of relativity

Development and validation of a model for assessing the success of employee portals

2008 | European Conference On Information Systems | Citations: 0

Authors: Urbach, Nils; Smolnik, Stefan; Riempp, Gerold

Abstract: Many companies utilize employee portals to improve information exchange and comm ... Expand

Abstract: Many companies utilize employee portals to improve information exchange and communication between their employees as well as to better support business processes. Owing to limited IT budgets and the need to justify portal investments, assessing employee portal benefits is important. In this article, we propose a model for measuring employee portal success on the basis of the DeLone and McLean IS Success Model as well as an extensive review of employee portal success measurement literature. The resulting model is put forward as the foundation for future empirical work in this area. A card-sorting and item-ranking approach as well as a survey trial is conducted in order to validate the survey instrument. Methods of classical test theory are proposed for the further validation of the survey instrument and the assessment of the measurement model. Collapse

Topics: IS success user success user satisfaction service quality enterprise information portal

Methods: survey design conceptual modelling survey literature study card sorting

Theories: information systems success model