Methodological Mistakes’ Example in Psychological and Criminological Research. Example number 1: The role played by the “attention shifting in children pro-social behavior” (Wilson B. J., 2003) and how re-doing the research with a better Methodology!


This article shows an example of “how” a lot of psychological and criminological research has been done with poor methodology. Behind an apparent “scientific” appearance, some of these studies hid plenty of vitia (methodological mistakes), which prejudice results. In this way, the research findings are biased in the direction wanted by the observers.

This article shows the methodological vitia done by Wilson B. J. (2003) in a research on the “attention shifting” in “pro social” and “antisocial” children.

After having explained the mistakes, the article illustrates how the study should (eventually) re-done with a better methodological awareness.

Rationale – Background (Wilson’s mistakes).  

This Paper shows an example of “how” a lot of psychological and criminological research has been done with poor methodology. Like sample, the study on the attention shifting made by Wilson B. J. (2003) is used. The Paper, after illustrating the methodological mistakes, explains how the study should be “re-done” with a better methodological awareness.

According to Miller, Galanter and Pribram (1960) the cognitive processes are a central aspect to understand the human behaviour. The social information-process research could be applied successfully to the understanding of the aggressive and antisocial behaviour in the human beings (Losel F. 2005).

From the basic information processing model TOTE (Test – Operate – Test – Exit), proposed by Miller G. A., Galanter E. and Pribram K. H. (1960) more accurate models have been developed. Dodge, for instance, applied a social information-processing model for understanding children’s aggressive responses (Losel F. 2005; Lewis M. and Miller S. M. 1990; Dodge K. A. and Coie J. D. 1987; Crick, N. R. and Dodge K. A. 1996; Dodge et al., 2003).

According to Lewis M. and Miller S. M. (1990), Dodge indicated five stages that are involved in producing appropriate or inappropriate response.

These stages are: encoding; interpretation; response research; response decision; enactment. Inappropriate aggressive responses can be produced by some deficits in one or more of these stages. For instance, subjects: can misunderstand situations; or have learned a range of few possible responses to those situations.

Although a relation between cognitive processes and behavioral responses seem to be proved, the research, on “how” individual elements (of the Information Processing Model) affect behavioral responses, presents several limitations.

Some of these studies, for instance, are examples of inaccurate research.

The study of Wilson B. J. (2003) on the rule played by the “attention shifting in children’s pro-social behavior” has presented different methodological mistakes.

First of all, experimental groups were only two: 27 aggressive/rejected participants; and 27 non-aggressive/popular participants. A control group was absent.

Second of all, the subdivision in two groups (aggressive/rejected and non-aggressive/popular) has been an arbitrary distribution. A better study should have considered four different groups: aggressive/rejected; aggressive/popular; non-aggressive/rejected; non-aggressive/popular. Dodge K. A. et al. (2003) have demonstrated that: peer rejection predicts growth in aggression. Thus, from the study of Wilson B. J. (2003), we do not know with “what” attention shifting correlates. We do not know the nature of subjects’ aggressive behaviors. Is this related with “endogenous” aggression (e.g. traits)? Is this related with exogenous aggression (e.g. like natural answer to others’ aggressive behaviors)? Is this related with social rejection? … Etc… . We do not know.

Moreover, we will never know social rejections’ factors that determined aggressive answers. It is true that correlation does not mean causation (Hagan F. E., 2005), but the study of Wilson B. J. (2003) is not able to show the nature of this correlation. The “apparent correlation”, which was found, is the outcome of a chain of methodological mistakes and prejudices. The aggressive behaviors of the aggressive/rejected group, instead of being linked with endogenous factors, could be a mere consequence of social factors (outside the subjects). Those factors could be, exempli gratia, the rejections made by others.  It is possible that who rejected was more aggressive of the rejected one. For instance, the former could have acted with pro-active aggression. The latter could have answered simply with a natural reactive aggression.  Usually, receivers of aggression are the social weaker people. Thus, at the end, the subdivision made by Wilson B. J. (2003) is: arbitrary; biased by social factors such as the relations of “power” existing among the members of the group. Furthermore, traits attributed to subjects could be consequence of social mechanisms such as: just a world; fundamental error of attribution; scapegoat. The “guilty one” should be: the social weaker person. It is easier! It is a “social tradition”!! Thus, the correlation (found by the researcher) was consequence of many bias’ mechanisms operating in the Social and Psychological Sciences Research. An illustration of them, it is given by Epis L. (2011/2015).

The study of Wilson B. J. (2003) itself gives evidences of what the present writer wrote above. One of these is the strong selection bias. No equivalent groups have been chosen for comparison (Hagan F. E., 2005; Bachman R. and Schutt R. K., 2003). Groups have been selected in biased ways, which have affected the result of the Null Hypothesis Test. The aggressive/rejected group had a majority of male; whereas the non-aggressive/popular group had a majority of female. As “girls, regardless of status, have less difficulty then boys (in) shifting attention from one affective state to an others” (Wilson B. J., 2003), the higher presence of girls in the non-aggressive/popular group has enhanced surely the performance of this group. In the same way, the higher presence of boys in the aggressive/rejected group has decreased certainly the performance of this group. This is proved and attested by the same research findings of Wilson B. J. (2003)!!!!! Thus, it is simply a matter of logic! But, RARERY is LOGIC used by Psychologists and in Psychological Research (Epis L., 2011/2015). In other words, the different number of male and female inside the two groups have, according to the same research findings, biased and prejudiced the performances of same groups, creating a statistical significance that would not be existed without these mistakes. But please, do not worry if you cannot understand these logic implications. Even a lecturer of the University of Cambridge (e.g. Painter Kate) was unable to understand those aspects[1]!!

Moreover, as I introduced supra, the study of Wilson B. J. (2003) do not make any distinction between pro-active and reactive aggression. Hence, at least, we do not know if the attention shifting correlate differently with these two types of aggressive behaviors.

The distinction between proactive and reactive aggression is fundamental. According to Vitaro F. and Brendgen M. (2005) the reactive aggression “has its roots in the frustration-anger theory”; whereas the pro-active aggression “is more in line with the social learning model of aggression”. These two types of aggression seem to be present differently in the children. According to Camodeca M. and Goossens F. A. (2005), the reactive aggresion is common both in bullies and victims; whereas the proactive aggression was “only characteristic of bullies”. The proactive aggression, however, is not only a characteristic of the bullies (Camodeca M. and Goossens F. A., 2005), since Dodge and Coie (1987) have found that proactive aggressive boys “were also viewed as leaders”. Moreover, Crick and Dodge (1996) suggested that proactive and reactive children processes social information differently[2].

Hence, also, these research findings prove the presence of biases and prejudiced discussed supra.

For all these reasons, the relation between aggression and attention shifting cannot be proved by the exanimated study.

Here below, I explain how the research could be done with more methodological awareness.

A New Research on the Attention Shifting: the Research Questions

The investigation should verify whether attention shifting operates differently between proactive and reactive aggressive children. Furthermore, more categories should be considered: bullies; leaders; popular and unpopular children[3]; and so on.

Exempli gratia, we will attempt to understand if there is a difference in attention shifting among: proactive bullies; proactive leaders; reactive popular children; reactive unpopular children.

Methodological Approach and Research Hypothesis


Methodological Approach

This investigation is a correlation study. Hence, inferential statistic will be used. The level of statistical significance will be the customarily a = 0.05 (p<0.05)[4] (Hagan F. E. 2005). The test of significance will be non-directional (two-tailed), as at the present tense, there is not enough literature (produced with good methodology) that can suggest a direction instead of another one among the considered groups.

Attention shifting will be measured with ten thematic groups of six pictures. The protagonists of each thematic groups present different combination between aggressive/hostile and non-aggressive/friendly “body language” and “expressions”. Attention shifting (between negative and positive emotions) will be measured. The importance given by subjects to opposite cues and their abilities to recognise them will be considered. The pictures will present: both/either aggressive/hostile contexts; and/or non-aggressive/friendly contexts. The pictures should be done in way that: the general population of the children (between six and seven years) recognises half of them like aggressive/hostile and half of them like non-aggressive/friendly. Validity and reliability of the thematic pictures must be checked with a precedent study[5]. A higher number of identification like aggressive/hostile behaviour will indicate more attention to the aggressive/hostile cues. A higher identification of non-aggressive/friendly behaviours will indicate more attention to the non-aggressive/friendly cues. The thematic group of six pictures will be displayed for a short time (10 seconds) on a monitor. Children will have to classify each context like: either aggressive/hostile; or non-aggressive/friendly. The short time given for observing allows researchers understanding where attention shifting focuses more.

Research Hypothesis

The research hypothesises are:

  1. there is statistical significant differences in the classification made by reactive and proactive aggressive children?;
  2. there is statistical significant differences in the classification made by reactive unpopular children and the reactive popular children?;
  3. there is statistical significant differences in the classification made by proactive bullies and the proactive leaders?

The null hypothesises, briefly, are: there is not statistical significant differences BETWEEN and AMONG groups.

Research Design and Method


Four groups (each one) composed by 40 participants (20 male; 20 female) will be used.

The children will be selected in primary schools. The age will range from six to seven. A wider age difference will be avoided as it could introduce confounding variables. The difference age can itself be a factor able to affect the reactions (Vitaro F. and Brendgen M., 2005).

The reactive and proactive children will be selected using the “teacher-rating instrument” developed by Dodge and Coie (1987). According to Poulin F. and Boivin M. (2000), this scale seems has good validity.

The subdivision of proactive aggressive children in two groups: bullies and leaders (in absence of a validated scale) will be done considering the opinion of the teachers and the opinion of their classmates. Nevertheless, this can be a bias factor. Also the subdivision of the reactive children in two groups: popular and unpopular (in absence of a validated scale) will be done considering the opinion of the teachers and the opinion of their classmates.

All these groups should have (for the reasons illustrated supra) an equal presence of male and female. Otherwise, male and female should be compared only with other male and female. This is due the fact that girls and boys perform differently in attention shift (Wilson B. J., 2003). Hence, the prevalence of male or female in a group will bias the group performance.


The four groups will have to classify 60 pictures. They are gathered in 10 thematic groups with 6 items each. Each thematic group is constituted by: the same protagonists; the same contexts; with different body language and expressions. The body language cues will be gradually changed from aggressive/hostile to non-aggressive/friendly. Exempli gratia, one should have 100% of aggressive/hostile body expressions; one should have a mix of aggressive (65%) and friendly (35%) expressions; two should have a mix of friendly (50%) and aggressive (50%) expressions; one should have a majority of friendly cues (65%) and a minority of aggressive (35%); one should have only friendly expressions.

The children will sit in front of a personal computer (PC).

Each thematic group will be displayed on monitors for 10 seconds. The children will have additional 10 seconds to give their choices (without the pictures). The thematic groups will be presented in different order (made at random).

The child will have to classify the situation represented by the six pictures like: either aggressive/hostile; or not-aggressive/ friendly.

A short break is done between each thematic group. During this break, the test-administrator asks to children if all is right and they are ready to proceed. It should last 30-40 seconds approximately.

The computer program will record the choices automatically.


The data will be analysed using the program SPSS.

An analysis of variance (ANOVA), correlation and regression, will be performed. The null hypothesises will be accepted or rejected according to these data. The level of statistical significance will be the customarily a = 0.05 (p<0.05).

The assumptions of: normality; homogeneity of variance; and continuity and equal intervals of measures; … are also tested. It is suggested by Kerlinger F. N. (1973).


A study, made according to these criteria, could be a good attempt to understand if:

  1. exist a possible correlation between the attention shifting and aggressive behaviour;
  2. this correlation is different between proactive aggressive children and reactive aggressive children;
  3. this correlation is different between aggressive children with opposite social rules.

Ethics Issues

According to the code of practice (1993) of British Psychological Society, the informed consent of the parents (or those peoples who act in loco parentis) will be asked. The permission should be given after having received full information. It should be free and informed. Researchers will organise meetings with parents to explain the research study.  The study will avoid harming participants. Children (who will feel uncomfortable) will be withdrawn. Assistance will be provided if needed. Researchers will be committed to stay away from harming the participants as it happened in some experiments and studies such as, for instance, Cambridge-Summerville (1930). In that case, the boys of the treatment group have been harmed by their participation (Kimmel A. 1988; Ruane J. M. 2005).

The anonymity of the participants will be guaranteed using a number instead of their names.

Limits of the studies

It was believed that the main limitation about all these studies was about the overlap between attention and perception. This was believed as attention and perception are two different cognitive processes (Sternberg R. J. 2000).

Nevertheless, it is not. Indeed, according to Benso F. (2013) there is not perception without attention. This thesis is corroborated by plenty of recent research findings in cognitive neurosciences.

Thus, even if this question was not considered enough by Wilson B. J. (2003), nowadays it seems not to be an important issue.

After this study, further studies should be done to verify the relation between the different types of aggression and the attention[6].

Common limits for every study done with the actual customary methodological quantitative research are those described by Meehl. Exempli gratia, Meehl (1990a) argues that the: “Null hypothesis testing of correlational predictions … is subject to the influence of ten obfuscating factors whose effects are usually (1) sizeable, (2) opposed, (3) variable; (4) unknown”[7].

Meehl (1990a) suggests a possible way to reduce at least the problem of inadequate statistical power: applying at “your sample size at a power of .9 or better”. This can reduce the source of type I error (a) (risk to reject a true null hypothesis), but, on the other side, this increases the type II of error (b) of not rejecting a false null hypothesis

Different problem is the crud factor. According to Meehl (1990a; 1990b) the crud factor is that: “in social science, everything is somewhat correlated with everything (“crud factor”), so whether Ho is refuted depends solely on statistical power” (Meehl P. 1990b). The crud factor does not deal with “some source of statistical error” (Meehl P. 1990a), but when we speak about the crud factor “we are taking about real differences, real correlations, real trends and patterns for which there is, of course, some true but complicated multivariate casual theory”. Meehl (1990a) proceeds: “I am not suggesting that these correlations are fundamentally unexplainable. They would be completely explained if we had the knowledge of Omniscient Jones, which we don’t. The point is that we are in the weak situation of corroborating our particular substantive theory by showing that X and Y are “related in a non-chance manner,” up a range of admissible values that would be counted as corroborative”.

This problem cannot be resolved by statistics, but only inside an epistemological reflection (Meehl P. 1997). From this reflection, Meehl (1997) suggests “a corroboration index C*”. On the crud factor see also Epis L. (2011/2015).

The actual research, according to Meehl (1990a; 1990b; 1997), presents a weak use of significance testing. “What make this use weak, again, has nothing to do with the ratio a:b, but involves the epistemic relation between the inference H*: d > 0 and the alleged consequent confirmation of T” (Meehl P. 1997).

The risk is: when the Ho is refuted “gives powerful support to a weak theory” (Meehl P. 1997).

In other words, whereas we can decrease the problem of inadequate statistical power; at the present time it is not possible resolve completely the problem of the crud factor, unless we do not increase an epistemic and logic reflection as, also, Epis L. (2011/2015) strongly suggested.

For these reasons, further research should be done to verify possible positive outcomes.


[1] An evidence of how very few people (nowadays) are able to understand the logical and epistemological mistakes inside the Psychological Research and Paradigm. It shows, also, how Academia, instead of keeping a critical thinking, tends to “wear” and to “defend” the easier common “group’s thinking”. The facts happened in 2006.

[2] Pro-active aggressive children “evaluated verbally and physically aggressive acts in significantly more positive ways than did non-pro-active aggressive children”; and that pro-active aggressive “children are less likely to endorse relationship-enhancing goals during social interaction”.

[3] The rejection and social isolation is a form of aggression, where the aggression from a physical domain is applied indirectly in a social domain (Vaillancourt T. 2005).

[4] The level of significance a=0.05 is considered the more appropriate by the majority of the literature:  Lipsey M.W. (1990); Neuman W.L. and Wielgand B. (2000); Ronald J.H., Douglas G. H. and Regoli R.H. (1983). The latter suggests: an a=0.05 for samples which range from 30 to 100; and an a=0.01 for both samples that are higher then 100 and unavoidable small samples (lower then 30).

[5] The validity of the cues used in the pictures will be based on the studies of Eibl-Eibelfeldt (1993). The reliability of these pictures will be done with two validation studies: a test retest; and a split half. The customary research reliability coefficient of correlation (alpha) of 0.80 (Hagan F. E. 2005) will be substitute with an alpha of 0.90 (suggested by Meehl 1990a).

[6] Nevertheless, it is possible to develop additional studies for considering: the theory of attention “bottleneck” of Broadbent (1958); the theory of “the different allocation of the limited attentive resources” proposed by Kahneman (1973). In the same way, extra studies could be done to analyze the relation between perception and aggression. Exempli gratia, the different theories of perception (the theory of the constructive perception of Bruner and Gregory and Rock; the theory of direct perception supported by Gibson (Sternberg R. J. 2000); etc…) could be tested. This is as in “sciences” is better always verifying … and re-verifying  … everything. Nothing should be given for definitive!

[7] The ten obfuscating factors are: loose derivation chain; problematic auxiliary theories; problematic ceteris paribus clause; experimenter error; inadequate statistical power; crud factor; pilot studies; selective editorial bias; detached validation claim for psychometric instruments.