Chapter 1 Summary of NCA
NCA is based on necessity causal logic. If a certain level of the condition is not present, a certain level of the outcome will not be present. Other factors cannot compensate for the missing condition. The necessary condition allows the outcome to exist, but does not produce it. This is different from sufficiency causal logic where the condition produces the outcome.
Conventional theories and methods are based on additive logic. This logic assumes that several factors contribute to the outcome and can compensate for each other. For example, conventional quantitative methods like multiple regression analysis and structural equation modeling describe the complexity of different contributing factors producing the outcome. In these models a low value of one factor can be compensated by changing another factor. When making causal interpretations with these models, factors are usually interpreted as ‘generic causes’ that can change the probability of the outcome.
Conventional quantitative methods focus on generic causes and do not cover necessity logic and are thus not able to identify necessary conditions in data sets. This was the main reason for developing NCA. NCA ensures theory-method fit when the theory includes necessity relations between factors and outcome and these relationships need to be evaluated empirically.
Conducting NCA consists of four stages:
Formulate the necessary condition hypothesis.
Collect the data.
Analyse the data.
Report the results.
Each step is explained below in more detail in sections 1.3, 1.4, 1.5, and 1.6.
In research projects and publications, NCA can be used as a stand-alone method or be used together with conventional methods such as multiple regression analysis, structural equation model, or QCA. The decision to apply NCA as a stand-alone method or a complementary method depends on the goal of the research.
1.1 NCA as stand-alone method
A researcher may have two reasons to use NCA as a stand-alone method in a particular study. First, the researcher may want to employ just a necessity view on the phenomenon of interest. He may have formulated a parsimonious ‘pure necessity theory’ (see Chapter 2) consisting of one or more necessity relations (e.g., Karwowski et al., 2016; Knol et al., 2018). To ensure theory-method fit, NCA is used for testing pure necessity theories. A regression-based method is not appropriate for this purpose. [Similarly, when a researcher wants to test a theory consisting only of generic causal relations (most current theories), a conventional probability-based method such as regression analysis should be selected and NCA is not appropriate]. A second reason for using NCA as the stand-alone method is that the researcher may want to add a necessity view to an existing generic causal theory from the literature. Without (re)testing the generic causal relationships, she may want to test whether some concepts of the theory are (also) necessary conditions.
The advantage of using NCA as a stand-alone method is that the study and its theoretical reasoning can focus on the single necessary concepts. There is no need to include other concepts (e.g., contributing factors, control variables) into the reasoning and analysis. This allows a clear story line and efficient data collection and analysis.
1.2 NCA as a complementary method
A researcher may also have reasons to use NCA in combination with other methods in a single study. The researcher may want to add a necessity view to a generic causal theory and to analyse necessity and generic relationships in combination. This can be done in two ways. First, the generic causal theory is leading and some concepts from this theory are tested for necessity. Second a combined necessity-generial causal theory is developed with both necessity and generic causal relationships between concepts.
In the first option, an existing or new generic causal theory has concepts that are considered being contributing factors that can change the probability of the outcome (including control variables). From these concepts, potential necessary conditions are selected. The necessity and generic causal relationships of these concepts with the outcome are tested with the appropriate methods and the methods are conducted successively. The integration occurs when the results are discussed. Several NCA multimethod studies that start with a generic causal theory have been reported in the literature. For example, when NCA is used in combination with (multiple) regression (e.g., Jain et al., 2022; Klimas et al., 2022; Stek & Schiele, 2021) or structural equation modeling (e.g., Della Corte et al., 2021; W. Lee & Jeong, 2021; Renner et al., 2022; Richter, Schubring, et al., 2020) “important” factors are identified with a regression-based method, and the necessity of these and other factors is analysed with NCA for identifying whether the factors are necessary or not (see section 4.4). When NCA is used in combination with Qualitative Comparative Analysis- QCA (C. S. Kopplin & Rösch, 2021; e.g., Torres & Godinho, 2022) the results of NCA are compared to the sufficient configurations that are identified by QCA (see section 4.5).
In the second option, the researcher starts the study with a combined necessity and generic causal theory. This is a complex ‘embedded necessity theory’ (see Chapter 2) that consists of both necessity and generic causal relations from the start. Some concepts may only have a generic causal but not a necessity relationship with the outcome; other concepts may be a necessary cause but not a generic cause, and yet others may be both a necessary cause and a generic cause. Embedded necessity theories that combine necessity and generic causal theorizing for describing relations are still rare. In QCA, most studies theorize from the perspective of sufficiency and test the potential necessity of the single factors, but do not include necessity theorizing from the start. Also, most studies that combine NCA with multiple regression or structural equation modeling do not theorize about necessity and generic causation in combination (for an exception see Dul, 2019). Testing embedded necessity theories requires a multimethod approach with both regression-based method and a necessity-based method.
When combining NCA with regression analysis or QCA, the order of conducting NCA and the other method is not relevant. When NCA is used in combination with structural equation modeling (SEM) the order matters. First SEM is applied followed by NCA. The reason is that the outcome of the SEM measurement model is used to define the constructs to be tested for necessity with NCA.
1.3 Formulate the necessary condition hypothesis
NCA starts with a theoretical notion that a necessity relation may exist between \(X\) (the potential condition) and \(Y\) (the outcome). This is usually done by formulating and justifying a hypothesis that is part of a theory (see Chapter 2).
NCA is mainly used in theory-testing research, which starts with formulating the theory and proceeds with empirical testing the theory with data. Hence, theory formulation comes before data collection and analysis. This is also the focus in this book. However, NCA can also be used in theory-building and exploratory research. Then formulating the theory is based on the results of the data analysis, and thus comes after it (e.g., Stek & Schiele, 2021).
1.4 Collect the data
Collecting data in NCA is not different from collecting data in general. The goal of data collection is to have scores (values, levels) for the condition \(X\) and the outcome \(Y\) for each case. The selected research design (e.g., ‘experiment’, ‘survey’, ‘case study’) must meet common quality standards. Also the selection of cases for measurement and data analysis must fit the goal of the research (e.g., random sampling, purposive sampling for specific reasons). The data must be ‘good’. This means that the data must be valid (the measurement scores reflect what they are intended to reflect) and reliable (when measurement is repeated, the results are the same).
NCA has no new requirements on collecting the data. There are a few exceptions. First, the setup of a ‘necessity experiment’ is different than the setup of a ‘sufficiency experiment’ or a common ‘average treatment effect experiment’ (see section 3.2). Second, in certain situations it is possible to sample just a single case to perform NCA (see section 3.3). Third, the way that NCA is conducted may differ depending on the types of data that are used: quantitative data (section 3.4), qualitative data (section 3.5), longitudinal data (section 3.6), and set membership scores (section 3.7). Furthermore, the identification of potential outliers is partly different from the common way of identifying outliers (see section 3.8).
1.5 Analyse the data
Data analysis is at the core of NCA. NCA’s data analysis assumes that it makes theoretically sense to analyse the data with necessity logic (see section 1.3) and that the data are meaningful (see section 1.4). For a quantitative necessary condition analysis the ‘scatter plot’ approach can be used. The scatter plot maps cases in the XY plane and NCA conducts a bivariate analysis on the scatter plot. Figure 1.1 shows an example of a scatter plot with \(X\) = Contractual detail and \(Y\) = Innovation of 48 buyer-supplier relationships for evaluating the hypothesis that Contractual detail is necessary for Innovation (Van der Valk et al., 2016).
Figure 1.1: Scatter plot approach of NCA estimating the empty space in the upper left corner when \(X\) is supposed to be necessary for Y. Data from Van der Valk et al. (2016).
According to this hypothesis it is not possible to have cases with low level of Contractual detail (\(X\)) and high level of Innovation (\(Y\)). This means that the upper left corner of the scatter plot remains empty. The space without cases is called the empty space or ceiling zone. NCA draws a border line called ceiling line between the space without cases and the space with cases. Two default ceiling lines are the Ceiling Envelopment - Free Disposal Hull (CE-FDH), which is a step function that can be used when \(X\) or \(Y\) are discrete with a limited number of levels or when the border is irregular, and the Ceiling Regression - Free Disposal Hull (CR-FDH), which is a straight trend line through the upper left corner points of the CE-FDH line. When a few cases are present in the otherwise empty space the ceiling line is not entirely accurate. The ceiling-accuracy (c-accuracy) is the percentage of cases on or below the ceiling line. By definition the CE-FDH line is 100% accurate and the CR-FDH line is usually not 100% accurate. A low c-accuracy indicates that the ceiling line may not properly represent the border between empty and full space, and another ceiling line may be selected. The scope (S) is the area of the total space where cases can appear given the minimum and maximum possible values of \(X\) and \(Y\). The effect size (d) is the area of the ceiling zone (C) divided by the scope: d = C/S. The effect size can have values between 0 and 1. NCA estimates the ceiling line and its effect size from the sampled data. The statistical test of NCA consists of estimating the p value of the effect size. With the NCA software the NCA parameters can be computed.
The remaining part of this section demonstrates NCA’s data analysis by using the NCA software.
1.5.1 Prepare the analysis
For performing a (quantitative) NCA the NCA software can be used. This is a free package in R and the researcher must have access to R and RStudio. Researchers who are not familiar with R and RStudio can consult the NCA Quick Start Guide on the NCA website (https://www.erim.eur.nl). This guide explains how R and RStudio can be downloaded from internet and gives further details about the software. This demonstration uses the RStudio interface for conducting the analysis. After R and RStudio are installed and RStudio is opened, the script window is used to type and run the instructions. The first instructions for this demonstration are as follows:
#Demonstration NCA
#Install and load the NCA package
#install.packages("NCA") # to install the NCA package (only ones)
library (NCA) # load the NCA package (for each new session)The script starts with a remark after the hashtag (#), followed by instructions to install (download) the NCA package with the install.packages function. Installing the NCA package must be done once. Afterwards, for each new NCA session, the NCA package must be loaded (activated) using the library function.
1.5.2 Load the data
Next, in this demonstration three datasets are loaded. A dataset must be organised such that rows are cases and columns are variables (condition(s) and outcome(s)). Often, data files have the .csv extension but also other data file formats can be loaded, for example .xls (Excel), .sav (SPSS) or .dta (Stata). The researcher may conduct NCA on new or existing datasets, including archived datasets that are publicly available.
#Load the data
#Data on own computer:
data1 <- read.csv("myData.csv", row.names = 1) # load and rename my dataset
#Data on internet (example Worldbank):
install.packages("WDI") #install package for loading Worldbank data
library(WDI)
data2 <- WDI (indicator = c("SH.IMM.IDPT","SP.DYN.LE00.IN"))
#Data in the NCA software:
data(nca.example2)
data3 <- nca.example2 #load the example data from the NCA packageThe first dataset that is loaded is a .csv dataset. The dataset is assumed to be stored in the working directory on the user’s computer. After loading the dataset in R it is renamed as ‘data1’.
The second dataset is obtained from the internet. It is a dataset from the Worldbank website, which can be extracted with the WDI package. The first variable (indicator) that is extracted refers to the vaccination level of a country and the second to the country’s life expectancy. The vaccination level is the percentage of children aged 12-23 months who received vaccination against diphtheria, pertussis (or whooping cough), and tetanus (DPT). Life expectancy is the number of years a newborn infant would live if prevailing patterns of mortality at the time of its birth were to stay the same throughout its life.
The third dataset is part of the NCA package from version 3.1.2. and has data on 48 buyer-supplier relationships as the rows. This dataset has three conditions (Contractual detail, Goodwill trust, and Competence trust) and one outcome (Innovation). The dataset is published as Table 2 in an article entitled “When are contracts and trust are necessary for innovation in buyer-supplier relationships? A necessary condition analysis” by Van der Valk et al. (2016).
The instruction head(data3) shows the top rows of the last dataset. The first row has the variable names and the first column the names of the cases.
## Innovation Contractual detail Goodwill trust Competence trust
## 1 3.57 3.24 2.71 4.0
## 2 3.57 2.71 2.43 3.0
## 3 1.29 2.29 4.00 4.0
## 4 2.14 4.14 3.71 4.0
## 5 1.00 2.43 3.29 3.5
## 6 3.43 1.86 3.86 4.51.5.3 Estimate effect size and the p value with nca_analysis
After the data are loaded the NCA analysis starts with estimating the size of the space that is expected to be empty given the hypothesis. By default, the software assumes that the expected empty space is in the upper left corner, representing the hypothesis that the presence or a high value of \(X\) is necessary for the presence of a high value of \(Y\). The software can also analyse other corners depending on whether the presence or absence of \(X\) is necessary for the presence or absence of \(Y\) (see section 1.3) by using the corner argument in nca_analysis. By default the software uses the CE-FDH and CR-FDH ceiling techniques to draw the ceiling line and to calculate the effect size. Other ceiling techniques could be selected as well using the ceilings argument in the nca_analysis function (see section 6.3.3).
The estimation of the effect size for data3 is done with the function nca_analysis. The first analysis with the name ‘model 1’ evaluates the hypothesis that Contractual detail is necessary for Innovation:
The nca_analysis instruction specifies first the dataset and then the names of the condition and the outcome. After running this instruction the analysis is done, but the output is not yet shown.
A short summary of the results can be obtained by calling model1. This prints the effect sizes of the condition for the two default ceiling lines in the console window of RStudio.
##
## --------------------------------------------------------------------------------
## ce_fdh cr_fdh
## Contractual detail 0.24 0.19
## --------------------------------------------------------------------------------The second analysis (‘model2’) shows that the condition and outcome can also be specified by their column numbers in the dataset. The results are the same.
##
## --------------------------------------------------------------------------------
## ce_fdh cr_fdh
## Contractual detail 0.237 0.188
## --------------------------------------------------------------------------------The third analysis (‘model3’) shows that multiple bivariate analyses can be done in a single run.
##
## --------------------------------------------------------------------------------
## ce_fdh cr_fdh
## Contractual detail 0.237 0.188
## Goodwill trust 0.307 0.255
## Competence trust 0.321 0.214
## --------------------------------------------------------------------------------In this example three conditions are included each representing a different necessary condition hypothesis. An analysis with multiple conditions in a single run is always done with one outcome.
The fourth analysis (‘model4’) shows that the ceiling line can be selected. In this example the CR-FDH line is selected.
#Conduct NCA with selected ceiling line
model4 <- nca_analysis(data3,2:4,1, ceilings = 'cr_fdh')
model4##
## --------------------------------------------------------------------------------
## cr_fdh
## Contractual detail 0.19
## Goodwill trust 0.26
## Competence trust 0.21
## --------------------------------------------------------------------------------The fifth analysis (‘model5’) also includes NCA’s statistical test to calculate the p value. An empty space could be a random result of variables that are actually unrelated. The p value protects the researcher from concluding that the empty space is empty because of necessity, whereas actually it is a random result of unrelated variables. The statistical test can be activated by specifying the number of permutations for the estimation of the p value in the test.rep argument of nca_analysis as follows.
## Preparing the analysis, this might take a few seconds...
## Do test for : ce_fdh - Contractual detailDone test for: ce_fdh - Contractual detail
## Do test for : cr_fdh - Contractual detailDone test for: cr_fdh - Contractual detail
## Do test for : ce_fdh - Goodwill trustDone test for: ce_fdh - Goodwill trust
## Do test for : cr_fdh - Goodwill trustDone test for: cr_fdh - Goodwill trust
## Do test for : ce_fdh - Competence trustDone test for: ce_fdh - Competence trust
## Do test for : cr_fdh - Competence trustDone test for: cr_fdh - Competence trust##
## --------------------------------------------------------------------------------
## ce_fdh p cr_fdh p
## Contractual detail 0.24 0.008 0.19 0.009
## Goodwill trust 0.31 0.002 0.26 0.004
## Competence trust 0.32 0.002 0.21 0.008
## --------------------------------------------------------------------------------The selection of the number of permutations is a balancing act between p value accuracy and computation time. The summary output now includes the p value for each effect size.
1.5.4 Create output with nca_output
More details of the results can be obtained with the nca_output function. The output is shown for analysis model6 that only includes Contractual detail and the CR-FDH ceiling line.
## Preparing the analysis, this might take a few seconds...
## Do test for : cr_fdh - Contractual detail##
## --------------------------------------------------------------------------------
## --------------------------------------------------------------------------------
##
## Number of observations 48
## Scope 15.40
## Xmin 1.86
## Xmax 5.71
## Ymin 1.00
## Ymax 5.00
##
## cr_fdh
## Ceiling zone 2.893
## Effect size 0.188
## # above 2
## c-accuracy 95.8%
## Fit 79.2%
## p-value 0.009
## p-accuracy 0.002
##
## Slope 0.544
## Intercept 2.214
## Abs. ineff. 9.614
## Rel. ineff. 62.428
## Condition ineff. 15.298
## Outcome ineff. 55.642The output first displays descriptive information about the sample such as number of cases (‘observations’), the scope, and the observed extreme values of \(X\) and \(Y\). Next, the NCA parameters for the selected ceiling line are displayed including effect size, number of cases above the ceiling line, and corresponding ceiling line accuracy. The fit measure is an indication how well the ceiling line represents the border line between the space with and without cases. In this example the fit of the CR-FDH ceiling line is 79.2% (maximum fit is 100%), suggesting that the ceiling line is not very regular. Next, the p value and the accuracy of the estimated p value, which depends on the selected number of permutations in nca_analysis, are displayed. If the ceiling line is a straight line, the output also gives the slope and intercept of the ceiling line. The final four parameters are related to ‘inefficiency’, which is discussed in section 5.1.
Scatter plots can be displayed by adding the argument plots = TRUE or plotly = TRUE in the nca_output function. The regular scatter plot of ‘model 1’ can be obtained as follows:
The regular scatter plot shown in Figure 1.1 is displayed in the Plots window of RStudio.
An interactive scatter plot (‘plotly’) is displayed in the Viewer window of RStudio as follows:
The non-interactive version of this plot is shown in Figure 1.2 and the interactive version can be approached here.
Figure 1.2: Example of a plotly scatter plot.
During inspection of the scatter plots, the researcher may observe potential outlier cases that may have a large effect on the estimated parameters. The NCA software from version 4.0.0 includes the possibility to detect and evaluate potential outliers (see section 3.8).
The scatter plots can be graphically adapted in different ways as described in section 7.3.
The results of all nca_output can be saved as a pdf file with the pdf = TRUE argument in the nca_output function.
1.5.5 Perform the bottleneck analysis with nca_output
After the effect size and its p value are computed and the scatter plots are inspected the researcher can make a judgment whether or not a necessary condition in kind (\(X\) is necessary for \(Y\)) has been identified. A common reasoning for not having identified a necessary condition is that theoretical support is lacking (no theoretically reasonable hypothesis can be formulated), or the effect size is practically irrelevant (e.g., d < 0.1), or that the empty space is likely a random result of two unrelated variables (e.g., p > 0.05). After the three conditions are satisfied (theoretical support, large effect size, small p value), the researcher may judge that there is empirical support in the present study for a necessity relationship. Then the researcher can start to conduct a necessary condition analysis in degree with the conditions that meet the three criteria. This analysis can be done with the bottleneck table. In this demonstration the three conditions and the CR-FDH ceiling line are selected for the bottleneck table analysis. The values of \(X\) and \(Y\) in the bottleneck table are ‘percentages of the range’. This means that 100 corresponds to the maximum value, 0 to the minimum value, 50% to the middle value, etc.
#Show the bottleneck table
model7 <- nca_analysis(data3, 2:4, 1, ceilings = 'cr_fdh')
nca_output(model7, bottlenecks = TRUE, summaries = FALSE)##
## --------------------------------------------------------------------------------
## --------------------------------------------------------------------------------
## Y 1 2 3
## 0 NN NN NN
## 10 NN NN NN
## 20 NN NN NN
## 30 NN NN NN
## 40 NN NN NN
## 50 NN NN NN
## 60 8.3 2.2 NN
## 70 27.4 35.2 26.0
## 80 46.5 68.2 54.0
## 90 65.6 NA 81.9
## 100 84.7 NA NATo show only the bottleneck table, the nca_output function includes the argument summaries = FALSEto suppress the text output.
The first column of the bottleneck table is a set of values of the outcome \(Y\), and the next columns show the corresponding required levels of the conditions. The table can be read row by row to find the levels of the conditions that are required for a given level of the outcome. NN means that the condition is not necessary for the corresponding level of the outcome. In the default bottleneck \(X\) and \(Y\) values are expressed as percentages of the range. As discussed in section 4.3, the values can also be expressed as actual values, percentiles and percentage of maximum, which allows useful interpretations of the bottleneck table. This section also explains why there can be NA’s in the bottleneck table. In this example there are a few NA’s, which could be replaced by the highest observed value of 5 using the argument cutoff = 1 in the nca_analysis function.
1.6 Report the results
Two types of NCA reports exist: methodological reports and application reports. In methodological reports, the method or part of the method is introduced in a specific field, often illustrated with examples from that field (Dul, Karwowski, et al., 2020; Dul, Hauff, et al., 2021; Hauff et al., 2021; Richter & Hauff, 2022; Tóth et al., 2019; Tynan et al., 2020). Introducing the NCA method in a new specific field has particularly an added value if it can be explained:
Why necessity logic is particularly useful for the field in general or for specific topics and challenges (e.g., a list of potential topics/challenges that can benefit from NCA).
That necessity thinking already (often implicitly) exists in the literature of the field (e.g., a list of necessity statements from the field).
How NCA is different from conventional methods (e.g., by explaining the steps of NCA –theory, data, data analysis, interpretation–, by referring to an example from the field).
How NCA and can give new insights to the field (e.g., an example of applying NCA to a topic from the field).
In NCA application reports the focus is on better understanding a specific phenomenon, and necessity logic and NCA are used to serve that goal. The NCA-specific parts of these application publications are (for details see Dul, 2020):
Introduction/theory: introduction of necessity logic/theory.
Methods: Description of NCA’s data analysis approach.
Results: Presentation of scatter plots and NCA parameters (e.g., effect size, p value).
Discussion: Description of the importance of including identified necessity in theory and practice (e.g., to avoid failure of the outcome or waste of efforts).
In application publications NCA can be used as a stand-alone method (section 1.1) or in combination with other methods (section 1.2).
Although NCA has already been broadly applied, there are still many fields where the NCA method has not been introduced and applied. Most likely, NCA could be applied in any of the 249 research categories defined by the Journal of Citation Reports. But currently, only about a quarter is covered, with the largest number of publications in the two categories ‘management’ and ‘business’.
1.7 Basic guidelines for good NCA practice
The guidelines are also published in Dul (2023) and Dul et al. (2023).
Based on this summary and other publications about the NCA methodology (Dul, 2016b, 2020, 2023; Dul, Van der Laan, et al., 2020), basic guidelines for good NCA practice can be formulated (Table 1.1). These general guidelines can support researchers and reviewers of research in conducting, reporting and evaluating NCA studies, and can help to ensure good quality of NCA applications.
| Topic | Chapter |
|---|---|
| Theoretical justification | |
| Explain why X can be necessary for Y. | Ch. 2 |
| Formulate the relationship between X and Y in terms of a necessity hypothesis (e.g., in terms of “X is necessary for Y”). | |
| In explorative research, theoretically justify a necessary condition that is found ex post. | |
| Meaningful data | |
| Use a good sample. | Ch. 3 |
| Use valid and reliable scores of X and Y (e.g., by using common approaches for evaluation of validity and reliability). | |
| Scatter plot | |
| Show the scatter plot (or contingency table) of all conditions that are evaluated for necessity. | Ch. 4 |
| Visually inspect the scatter plot (e.g., pattern of the border, potential outliers). | |
| Ceiling line | |
| Select the ceiling(s) based on the number of levels of X and Y and the expected or visually observed (non-)linearity of the border. | Ch. 4 |
| Only show the selected ceiling lines in the scatter plot. Do not show the two (default) ceiling lines if these are not selected for the analysis. | |
| Effect size | |
| Report the estimated effect size. | Ch. 4 |
| Evaluate the practical relevance of the effect size (e.g., threshold level > 0.1). | |
| Statistical test | |
| Report the estimated p-value. | Ch. 6 |
| Evaluate the statistical relevance of the effect size (e.g., threshold level < 0.05). | |
| Bottleneck analysis (necessity in degree) | |
| Present the bottleneck table for the non-rejected necessary conditions. | Ch. 4 |
| Decide how to present the bottleneck table (e.g., using percentage of range, actual values, or percentiles). | |
| Descriptions of NCA | |
| Refer to NCA as a method (including logic/theory, data analysis and statistical testing), not just as a statistical tool or data analysis technique. | |
| Acknowledge that: | |
| • NCA’s necessity analysis differs from fsQCA’s necessity analysis. | |
| • NCA differs from a “moderation analysis” in regression analysis. | |
| • NCA is not a robustness test for other methods. | |
| Properly describe elements of NCA, e.g.: | |
| • Use only necessity wordings to describe the necessity relationship between X and Y (avoid imprecise or general words like (cor)related, associated, and incorrect sufficiency-based words like produce, explain). | |
| • Refer to NCA’s statistical test as a permutation test (avoid incorrect descriptions like bootstrapping, (Monte Carlo) simulation, robustness check, or t-test). | |
| • Use the name ‘multiple NCA’ or ‘multiple bivariate NCA’ rather than ‘multivariate NCA’ when several conditions are analysed in one run. | |
1.8 Checklist for reviewing an NCA publication
An NCA publication can be categorized as:
Empirical publication: Application of NCA’s necessity logic and data analysis approach with empirical (or simulated) data. The goal is to provide a theoretical, empirical, methodological, practical contribution (or a combination) by using data.
Theoretical publication: Use of NCA’s necessity logic for theory development without using data and NCA’s data analysis. Specific characteristics of NCA’s necessity logic are used in the publication, for example NCA’s ‘conditional logic with a causal assumption’, NCA’s ‘necessity in degree’, or NCA’s ‘typicality’ necessity perspective.
Methodological publication: The Description, explanation, illustration or further development of the NCA method.
Referencing publication: reference to (elements of) NCA, such as referencing (but not using) NCA’s necessity logic or to (elements of) NCA’s data analysis approach, or suggesting the use of NCA for future research (for example, in a review article or discussion section of a non-NCA publication).
This section provides a draft checklist to evaluate the quality of an empirical NCA publication (the final checklist will be published in Dul (2025); please send your comments on the draft checklist to jdul@rsm.nl). The checklist can be used by authors to ensure the proper application of NCA, proper reporting of NCA results. It can also be used by peers, reviewers, and editors to evaluate the quality of the publication and provide constructive feedback.
The checklist builds on the published basic guidelines for conducting and reporting NCA (Dul et al., 2023), and recent developments in NCA such as about NCA’s causality (Dul, 2024a), necessity theorizing (Bokrantz & Dul, 2023), NCA’s robustness checks (Section 4.6), sampling for NCA (Dul, 2024b), use of archival data in NCA (Dul et al., 2024), and developments related on the application of NCA in combination with QCA (Dul (2022); Section 4.5) or PLS-SEM (Hauff et al., 2024; Richter et al., 2023). Therefore, the checklist provides an up-to-date set of recommendations for a high-level application of NCA, based on the current knowledge about NCA.
Following the structure of a ‘standard’ academic publication, the checklist has a general section and five specific sections (Introduction, Theory, Methods, Results, Discussion) each with 5-15 NCA items that need to be addressed in an NCA publication. The items have different priority levels. Some items are essential (‘must-haves’) that must be addressed in any NCA publication. Other items are important (‘should haves’) to increase the quality of the publication, and yet other items are the ‘nice-to-haves’. When addressing all items, the publication meets the highest quality standard, for example for being published in a high-ranked journal. Some items can be easily addressed, others need more reflection. For each item, requirements, recommendations, and sources for further reading are provided.
In the checklist below, the user can select all item or a selection of items for specific publication sections (Introduction, Theory, Methods, Results, Discussion) or specific priority levels (Must-have, Should-have, Nice-to-have).
You can interact with the checklist below:
| Section | Priority | Item | Question | Recommendation | Further reading |
|---|---|---|---|---|---|
| General | Nice-to-have | 1 | Does the title (and abstract) reflect the necessity perspective/NCA? | If an important part of the publication and/or its conclusion are based on necessity logic and NCA, include in the title (and abstract) words that reflect necessity logic or NCA. | |
| General | Must-have | 2 | When referring to necessity, are only words that correctly describe a necessity relationship used? | Refer to necessity only by using words that describe a necessity relationship, and avoid imprecise descriptions like \(X\) (cor)relates with \(Y\), \(X\) is associated with \(Y\), \(X\) affects \(Y\), \(X\) causes \(Y\), etc., or incorrect sufficiency-based descriptions like \(X\) produces \(Y\), \(X\) drives \(Y\), \(X\) increase/decreases \(Y\), etc. | This book: Table 2.1. |
| General | Must-have | 3 | Is it explained that NCA is a method that consists of a specific causal logic and a related data analysis technique? | Refer to NCA as a method that combines necessity causality (rather than common probabilistic or configurational causal logic) and a related data analysis technique (ensuring theory-method fit). It is not just a data analysis technique or statistical tool. | Article: Dul (2024a) |
| General | Must-have | 4 | Is a proper comparison made between necessity (causal) logic of NCA and the (causal) logic used in conventional (regression-based) approaches? | When comparing necessity (causal) logic and NCA with causal interpretations of conventional (regression-based/statistical) approaches, use the term “probabilistic sufficiency” and not “sufficiency” for these approaches. | Article: Dul (2024a) |
| General | Must-have | 5 | Is a proper comparison made between the necessity logic of NCA and the logic used in QCA? | When NCA is compared with QCA, use the term “sufficiency” when referring to the (configurational) sufficiency approach of QCA, and the term ‘necessity analysis of QCA’ (and not ‘NCA’) when referring to the standard necessity approach of QCA. Explain the differences between the necessity analysis of QCA and that of NCA. | Articles:Dul (2016a); Vis & Dul (2018); Dul (2022); This book: Section 4.5 |
| General | Must-have | 6 | Is NCA used as a different perspective on causality and data analysis, and not as a robustness or confirmation test for other methods (and vice versa)? | Use NCA as an approach for a different perspective on causality and data analysis, and not as a robustness or confirmation test for other methods (and vice versa). | Article: Dul (2024a) |
| Introduction | Must-have | 7 | Are the primary and (possibly) secondary goals and contributions of applying NCA explicitly stated? | State the primary and (possibly) secondary goals and contributions of the NCA applicatons explicitly, such as theoretical (explanatory) contribution: developing theory with data (by testing or building hypotheses); empirical (predictive) contribution: by analyzing data from a necessity perspective without additional theorizing; methodological contribution: application of (a certain aspect) NCA and its data analysis for the first time in a particular field; practical contribution: how NCA results can be used in practice. | |
| Introduction | Should-have | 8 | For a theoretical contribution: is the (expected) added value explained? | Explain that the publication contributes to theory by testing (formulating necessity hypotheses before data analysis) or by exploration (formulating necessity hypotheses after data analysis). Note the difference between HARKING (unacceptable), SHARKING (unacceptable) and THARKING (acceptable) (See Hollenbeck & Wright, 2017). Explain the added value of the theoretical contribution (see this book Section 2.4) and why the perspective of necessity is important in the substantive context of the publication. For example, explain (not only mention) the theoretical added value of necessity causality compared to probabilistic sufficiency causality (see Dul, 2024a). Also explain how the publication contributes to the ‘conversation’ (see this book: Section 2.4) in the substantive field. For example, the conversation about reconsidering existing probabilistic sufficiency causal relationships in formal theories, in published studies, or in the current publication (when also a regression based-method is used); or the conversation about untested necessity claims made in formal theories (give quotes), in published studies (give quotes), or in the current publication. Explain if the publication contributes to a ‘pure necessity theory’ or an ‘embedded necessity theory’ (see Bokrantz & Dul, 2023). | Articles: Bokrantz & Dul (2023); Dul (2024a); Hollenbeck & Wright (2017); This book: Section 2.4 |
| Introduction | Should-have | 9 | For an empirical contribution: is the (expected) added value explained? | Explain the added value of the empirical contribution (without theorizing) and why this is important in the substantive context of the publication. For example, explain the added value of a description, prediction, exploration, replication, reproduction, or generalization and why this is important for theory or practice in the field. | Articles: Shmueli (2010); Köhler & Cortina (2023) |
| Introduction | Should-have | 10 | For a methodological contribution: is the (expected) added value explained? | Explain the added value of the methodological contribution and why this is important in the substantive context of the publication. For example, explain that the study applies NCA for the first time in this specific field and specify topics in the field that could benefit from using the necessity causal perspective and NCA. Also it is possible that the study applies a specific feature of NCA for the first time, for example necessity mediation (e.g., 2.2.4) or sub-sampling (Allard-Poesi & Massu, 2023). Both contributions would be called a ‘moderate’ methodological contribution according to Bergh et al. (2022). | Articles: Allard-Poesi & Massu (2023); Bergh et al. (2022); This book: Section 2.2.4 |
| Introduction | Should-have | 11 | For a practical contribution: is the (expected) added value explained? | Explain the added value of the practical contribution and why this is important in the substantive context of the publication. For example, explain the potential specific practical impact of an accepted hypothesis (practitioners must act on bottlenecks to be effective and avoid waste, no compensation possible) or of a rejected hypothesis (compensation is possible). | |
| Theory | Must-have | 12 | Is the hypothesized necessity relationship explicitly formulated as: ‘\(X\) is necessary for \(Y\)’? | Formulate the necessity relationship explicitly as a necessary condition hypothesis: ‘\(X\) is necessary for \(Y\)’. For hypothesis testing research this is done before data analysis; for hypothesis building/exploration research this is done after data analysis when a necessity relationship is found. When multiple necessary conditions are investigated, formulate separate hypotheses rather than combined hypotheses because one could be rejected and the other one not and because each hypothesis needs a separate theoretical justification. | This book: Section 2.2 |
| Theory | Should-have | 13 | Is the ‘direction’ of the necessity relationship specified? | If the ‘direction’ of the necessity relationship differs from presence/high level of both \(X\) is necessary for presence/high level of \(Y\)), formulate the ‘direction’ of necessity in the necessary condition hypothesis by adding presence/high level or absence/low level of \(X\) and \(Y\), where appropriate. | This book: Section 2.2.3 |
| Theory | Nice-to-have | 14 | Is a relevant ‘nc’ symbol added above the arrow in a figure that represents the XY necessity relationsip? | In a figure that represents the (expected) XY necessity relationship by an arrow, add the relevant symbol (+nc+, or -nc+ or -nc- or +nc-) near the arrow. | This book: Section 2.2.3 |
| Theory | Nice-to-have | 15 | Is the focal unit of the necessity theory specified? | Clearly specify the focal unit of the necessity theory (is the necessity theory about person, team, organization, project, dyad, country, etc.). | This book: Section 2.1 |
| Theory | Should-have | 16 | Is the theoretical domain of the necessity theory specified? | Specify the theoretical domain of the necessity theory where you claim/expect that the necessity theory will hold, given its boundary conditions. Note that the theoretical domain is usually larger than a population from which the sample is drawn. | This book: Section 2.1 |
| Theory | Should-have | 17 | Is the existing literature about the (probabilistic) relationship between \(X\) and \(Y\) re-considered from a necessity causal perspective | Consider the existing literature describing probabilistic relationships between your variables \(X\) and \(Y\) of the necessity hypothesis from the perspective of necessity causality (rather than referring to, describing or summarizing probabilistic findings/ideas). Where possible include quotes in the literature that hint to necessity logic. Avoid using sufficiency/probabilistic causal logic for making a justification for necessity causal logic. | |
| Theory | Must-have | 18 | Is it convincingly explained why \(X\) is necessary for \(Y\) using necessity logic | Explain why \(X\) is necessary for \(Y\) using necessity logic (why cases with the outcome will almost always have the condition (if \(Y\) then \(X\)); why cases without the condition rarely have the outcome (if no \(X\) then no \(Y\)); why the absence of the condition cannot be compensated or substituted by another factor. Consider doing and reporting the results of a thought experiment (searching for exceptions) in the development of a necessity hypothesis: possible re-specification of the condition, the outcome or the theoretical domain of the hypothesis. | This book: Section 2.2.2 |
| Theory | Nice-to-have | 19 | Is the temporal order of \(X\) and \(Y\) specified? | Include a plausible explanation of the temporal order (first \(X\) then Y) in the description of why the necessity causal relationship exists. If the data would support ‘X is necessary for Y’, why is the alternative explanation, ‘Y is sufficient for X’, less plausible? | This book: Section 2.3.1 |
| Theory | Nice-to-have | 20 | Is it specified if rare exceptions are allowed? | Explain in the necessity theory description if and why rare exceptions are allowed (typicality perspective of necessity: ‘almost always necessary’) or not (deterministic perspective on necessity). | Article: Dul (2024a) |
| Methods - data | Nice-to-have | 21 | Is the research design specified (experiment, large N observational study, small N observational study | Specify the research design for getting data. This can be the experiment (where \(X\) is manipulated and \(Y\) is observed), the large N observational study (where \(X\) and \(Y\) are observed) and the small N cases study (where \(X\) and \(Y\) are observed). | This book: Sections 3.2, 3.3.2, 3.3.1 |
| Methods - data | Should-have | 22 | Are cases (or a sample of cases) only selected from (a part of) the theoretical domain? | Explain that the selected cases or the selected/obtained sample consists of cases that are from (a part of) the specified theoretical domain (where the necessity hypothesis is claimed/expected to be true) to avoid sampling error (cases outside the theoretical domain). | |
| Methods - data | Nice-to-have | 23 | For large N quantitative research: is the type of sampling specified? | In large N quantitative research: Specify the population in the theoretical domain and the type of sampling from this population. If probability sampling is not applied, specify the type of sampling (e.g., convenience sampling) and its limitations for statistical generalization to the population. | This book: Section 3.3.2 |
| Methods - data | Nice-to-have | 24 | For large N quantitative research: are the results of a pre-study power analysis reported? | In large N quantitative research: Report if a pre-study power analysis was done to estimate the required sample size for finding necessity if it exists. If not, explain why not. Note that doing a post-hoc power analysis makes no sense for the present study but could inform future studies only. | Article: Dul (2024b); This book: Section 6.3.2 |
| Methods - data | Must-have | 25 | For simulation studies: are bounded \(X\) and \(Y\) variables used for obtaining artificial data? | If data are generated by simulation, ensure that the variables \(X\) and \(Y\) are bounded (have minimum and maximum values). For example, sampling from uniform or truncated normal distributed data are eligible, but not sampling from normal distributed data. | This book: Section 6.3.1 |
| Methods - data | Nice-to-have | 26 | For small N qualitative research: is the type of case selection specified? | For small N qualitative research: Specify the purposively selection of cases (based on presence of \(Y\) or absence of \(X\)). If a different type of purposive cases selection, random case selection, or convenience case selection is used, specify the type of case selection and its limitations. | Article: Dul (2024b); This book: Section 3.3.1 |
| Methods - data | Should-have | 27 | Are valid measures used for scoring of \(X\) and Y? | Ensure that scores for \(X\) and \(Y\) represent levels of the condition and the outcome as defined in the necessity hypothesis (valid measures). | This book: Sections 3.4, 3.5, 3.6, 3.7 |
| Methods - data | Should-have | 28 | Are levels of \(X\) and \(Y\) practically interpretable? | Ensure that scores for \(X\) and \(Y\) that are practically interpretable such that ‘a certain level of \(X\) is necessary for a certain level of Y’ has a practical meaning. For example, when Likert scales are used with meaningful anchor points, unstandardized scores for levels of \(X\) and \(Y\) are better interpretable than standardized scores. | This book: Sections 4.4; Richter et al. (2023) |
| Methods - data | Must-have | 29 | Are non-linear data transformations avoided or explained? | Do not conduct non-linear data transformation (e.g., log-transformation, logistic transformation) unless the transformed \(X\) and \(Y\) represent the concepts \(X\) and \(Y\) in the hypothesis. NCA only allows linear data transformation such as min-max transformation, normalization, or percentage transformation. | This book: Section 3.1 |
| Methods - data analysis | Should-have | 30 | If applicable: is the use of NCA justified when NCA is used as addition or alternative for QCA’s necessity analysis? | Justify (by explaining the added values) the use of NCA for testing necessity in the context of QCA (e.g., necessity in degree versus only necessity in kind, applying a necessity analysis on raw data, and why is this valuable). | Articles:Dul (2016a); Vis & Dul (2018); Dul (2022); This book: Section 4.5 |
| Methods - data analysis | Nice-to-have | 31 | Is a short description of NCA’s data analysis approach provided? | Provide to readers who are unfamiliar with NCA’s data analysis approach a short description of the NCA’s main elements ceiling line, empty space, scope, effect size) and refer to references for further reading. | This book: Section 1 ; Appendix in Dul (2024a) |
| Methods - data analysis | Must-have | 32 | Is the selected primary ceiling line specified and justified? | Specify and justify the primary selection of the ceiling line (e.g., CE-FDH when \(X\) or \(Y\) is discrete or when the ceiling line is expected to be non-linear; CR-FDH when \(X\) and \(Y\) are continuous or when the ceiling line is expected to be straight). Explain if, after visual inspection of the XY scatterplot, the choice of ceiling line was reconsidered. | This book: Sections 4.1.2, 4.6.1 |
| Methods - data analysis | Must-have | 33 | Is the threshold value for the effect size specified and justified? | Specify and justify the selection of the threshold value for necessity effect size for identifying necessity in kind. This threshold depends on the specific context of the study. If no argument is available for a specific value, a common benchmark value may be used (d = 0.10). | Article: Dul (2016b); This book: Sections 4.2, 4.6.2 |
| Methods - data analysis | Must-have | 34 | Is NCA’s statistical test correctly described, and is the threshold p value specified? | Refer to NCA’s statistical test as a permutation test (avoid incorrect descriptions like bootstrapping, (Monte Carlo) simulation, robustness check, or t-test). Specify the number of permutations that were selected from the permutation distribution (commonly 10,000), and the threshold p value that is selected (commonly 0.05). | Article: Dul, Van der Laan, et al. (2020); This book: Section 6.2 |
| Methods - data analysis | Nice-to-have | 35 | When several conditions are analyzed, is the name ‘multiple NCA’ rather than ‘multivariate NCA’ used? | When several conditions are analyzed, use the name ‘multiple NCA’ rather than ‘multivariate NCA’. | This book: Section 5.2 |
| Methods - data analysis | Should-have | 36 | Is the scope correctly described? | Make sure that the scope is correctly described as the area bounded by the minimum and maximum values of \(X\) and \(Y\) (the total square). | This book: Section 5.1.1 |
| Methods - data analysis | Should-have | 37 | Is the feasible area correctly described? | Make sure that the feasible area is described as the area under the ceiling where points/cases are possible. Avoid names such as “full area”. | This book: Section 5.1 |
| Methods - data analysis | Nice-to-have | 38 | Is the selected scope specified and justified (empirical, theoretical)? | Specify and justify the selection of the scope. By default, the empirical scope can be used, but reasons may exist to select a theoretical scope (e.g. representing the minimum and maximum values of the scales that are used for scoring \(X\) and Y. | This book: Section 5.1.1 |
| Methods - data analysis | Nice-to-have | 39 | Is the software that is used for conducting NCA specified? | Specify which software and which version of the software was used to conduct NCA. | This book: Section 7.1 |
| Methods - data analysis | Nice-to-have | 40 | Is it reported if a visual inspection of XY plots was done? | Report that a visual inspection of XY plots was done for all hypothesized/explored necessity relations regarding (1) the expected empty corner given the hypothesis, (2) what is the best ceiling line (e.g. irregular or straight border) and (3) if potential outliers exist (ceiling and scope outliers). If a visual inspection was not done explain why not. | This book: Section 4.1 |
| Methods - data analysis | Should-have | 41 | Is it reported if an NCA robustness check was done? | Report that a robustness check was done to evaluate the sensitivity of the results for choice of ceiling line, removal of outliers, and other choices. If a robustness check was not done explain why not. | This book: Section 4.6 |
| Results | Must-have | 42 | Are XY tables or XY plots of all tested/explored relationships shown? | Show in the main text or an appendix the XY tables or XY plots of all tested/explored relationships such that the reader can inspect the patterns. | This book: Section 4.1 |
| Results | Should-have | 43 | Is only the primary ceiling line that is used for the primary analysis shown in the XY plot? | Show only the primary selected ceiling line in the XY plot that is used for the main analysis for drawing a conclusion about the necessity relationship. | |
| Results | Nice-to-have | 44 | Are the conclusions from visual inspection of the XY plots summarized? | Summarize the conclusions from the visual inspection of the XY plot reported, for example, regarding the expected empty corner given the hypothesis, the selected ceiling line (e.g., irregular or straight border) and potential outliers (ceiling outliers and scope outliers). | This book: Section 4.1 |
| Results | Must-have | 45 | For large N quantitative research: is the effect size reported? | Report the estimated effect size with two decimal places (e.g., d = 0.09 or d = 0.11) , ensuring that the two digits have meaningful information about the precision of the effect size (e.g., for making a comparison with a threshold, e.g., d = 0.10 possible), and to avoid suggesting an unwarranted level of accuracy. | |
| Results | Must-have | 46 | For large N quantitative research: is the p value reported? | Report the estimated p value with three decimal places (e.g., p = 0.049 or p =0.051), ensuring that the three digits have meaningful information about the precision of the p value (e.g. for making a comparison with a threshold, e.g., p = 0.05 possible), and to avoid suggesting an unwarranted level of accuracy. Avoid using only ‘stars’ for the p value. | |
| Results | Must-have | 47 | Is the conclusion about necessity in kind based on three criteria: theoretical justification, effect size, and p value? | Conclude about necessity in kind using three criteria: 1. theoretical justification (the hypothesis); 2. Effect size large enough (above the selected threshold value), p value small enough (below the selected threshold value). | This book: Section 1.5.5 |
| Results | Should-have | 48 | Are the results of the robustness checks summarized? | Summarize the results of the robustness checks: is the conclusion about necessity sensitive for choices like ceiling line selection, selection of effect size threshold, selection of p value threshold, choice of scope (theoretical, empirical), and outliers (ceiling outliers and scope outliers), and conclude if the results are robust or fragile. | This book: Section 4.6 |
| Results | Should-have | 49 | Is the bottleneck table reported when multiple necessary conditions are not rejected? | Report the bottleneck table when necessary conditions are not rejected. The bottleneck table is particularly useful when multiple necessary conditions are not rejected. | This book: Section 4.3 |
| Results | Nice-to-have | 50 | If applicable: is the meaning of “NN” and “NA” in the bottleneck table explained? | Explain the meaning of “NN” and “NA” in the bottleneck table. | This book: Section 4.3 |
| Results | Should-have | 51 | Is the bottleneck analysis done with non-default values in the bottleneck table? | Report the bottleneck analysis (also) with type of values for \(X\) and \(Y\) other than the default of ‘percentage of range’: (e.g., actual values, percentiles). In particular the actual values of \(Y\) and percentile values of \(X\) might be valuable. | This book: Section 4.3 |
| Discussion | Should-have | 52 | Is an interpretation given of the results of the bottleneck analysis in terms of necessity in degree? | Report an interpretation of necessity in degree using the results of the bottleneck analysis. For example, explain what level of \(X\) is necessary for what level of \(Y\), for relevant levels of \(X\) and \(Y\). | This book: Section 4.3 |
| Discussion | Must-have | 53 | If applicable: is it explained how the results of NCA and of a regression-based analysis can complement each other? | When NCA is used in combination with a regression-based analysis, report how the specific results of NCA and of the regression-based analysis complement each other. | This book: Section 4.4 |
| Discussion | Must-have | 54 | If applicable: is it explained how the results of NCA and of QCA can complement each other? | When NCA is used in combination with QCA, report how the specific results of NCA and of QCA complement each other. For example, include NCA’s necessary conditions in QCA’s sufficiency solution. | This book: Section 4.5 |
| Discussion | Should-have | 55 | Are the results of using NCA summarized in terms of which necessary conditions were tested/explored, and which were (not) rejected? | Summarize the results of using NCA in terms of which necessary conditions were tested/explored, and which were (not) rejected. | |
| Discussion | Must-have | 56 | For a theoretical contribution (see introduction): is it discussed if the added value has been realized and what key insights were gained? | For a theoretical contribution (see introduction), discuss if the added value has been realized and what key insights were gained. | |
| Discussion | Should-have | 57 | For an empirical contribution (see introduction): is it discussed if the added value has been realized and what key insights were gained? | For the empirical contribution (see introduction), discuss if the added value has been realized and what key insights were gained. | |
| Discussion | Should-have | 58 | For a methodological contribution (see introduction): is it discussed if the added value has been realized and what key insights were gained? | For the methodological contribution (see introduction), discuss if the added value has been realized and what key insights were gained. | |
| Discussion | Should-have | 59 | For a practical contribution (see introduction): is it discussed if the added value has been realized and what key insights were gained? | For the practical contribution (see introduction), discuss if the added value has been realized and what key insights were gained. For example, explain the potential specific practical impact of an accepted hypothesis (e.g., practitioners must act on bottlenecks to be effective and avoid wasted efforts, as they cannot be compensated for) or of a rejected hypothesis (compensation possible). Explain the how the results can inform about how to act on a group of cases as well on individual cases, for example based on the bottleneck analysis. Give examples of cases in the scatter plot. | |
| Discussion | Should-have | 60 | Are limitations of NCA mentioned? | Mention general limitations of NCA (e.g., only focus on necessity; potential sensitivity to outliers), and specific limitations of applying NCA in the current study. | |
| Discussion | Should-have | 61 | Are potential future research directions with NCA discussed? | Discuss potential future research directions with NCA. Note (also when the results are robust) that always more research is needed to allow generalization of the results to the population and other parts of the theoretical domain. (e.g., which replications must be done?; which new topics could be addressed, etc.). | Article: Köhler & Cortina (2023) |