Don't Believe What You Read (Only Once): Comprehension Is Supported by Regressions During Reading

Authors: Elizabeth Schotter, Randy Tran and Keith Rayner

Summary:

Schotter, Tran and Rayner examined the effectiveness of a speed reading technique that takes away the need for eye regressions.  Eye regressions are times where your eyes actually move backwards while reading to look at past words.  An app called Splitz spurred this research.  Splitz allows users to read words independently as they flash across a screen.  This speed reading technique allows users to read quicker and allows the user to save time and have increased comprehension, according to the app developers.

The authors developed an experiment that tested the claims that Splitz helps users to comprehend statements at a higher rate than normal reading techniques.  This was accomplished by using a technology developed to track eye movements.  As the participants read through a serious of sentences, the technology would track the participants eyes.  As they left a word, the word would be replaced with "x's" so that the reader could not regress back to that particular word in the sentence and must continue to move forward.  For example,

I ran to the mall.
x ran to the mall.
x xxx to the mall.
x xxx xx the mall.
x xxx xx xxx mall

Forty undergraduate's from the University of California were selected to participate in the experiment.  In total, the participants read through a mix of 40 sentences, that were categorized into three categories of difficulty, ambiguous, unambiguous, and filler.  Ambiguous sentences were rendered structurally impossible when the reader reached the disambiguated verb in the main clause. These sentences followed a structure similar to the following sentence,

"While the man drank the water that was clear and cold overflowed from the toilet"

Unambiguous sentences replaced the initial verb from the ambiguous sentences with an intransitive verb to make it hard for readers to recognize the following noun as a direct object. Unambiguous sentences took the general form of,

"While the man slept the water that was clear and cold overflowed from the toilet"

Filler sentences were well constructed sentences that were easy to read and understand such as,

"It was very hard to find the truck inside of the messy toy box"

After reading the sentences, the participants answered a question that tested their comprehension of the previous sentence.

The study found that comprehension accuracy was less for both ambiguous and unambiguous sentences than on filler sentences.  Participants who lacked the ability to re-analyze sentences that were confusing to the participant could not comprehend them as well than while reading sentences normally.  As participants had reduced accuracy for both ambiguous and unambiguous sentences, this suggests that there is regressions are important to reading comprehension globally, and not just across varying degrees of sentence structure difficulty.

Critique:
Schotter, Tran and Rayner lay out a compelling argument against Splitz.  The authors lay out the format and design of their experiment thoroughly, making the process easily reproducible with the right equipment.  The main objection I have with this research is with how words were removed from the participants.  While the participants were unable to re-read the word after reading it (due to the word being replaced with x's) the x's matched up with number of letters in the previous word. For example,

elephant = xxxxxxxx

There may have been an even more significant impact of the participants inability to regress if the word had been completely removed.  The length of the string of x's may have given participant's brains some help in identifying the previous word.  I agree with the results and the analysis of what the results meant, I do believe that the evidence could be more strongly weighed in favor of regressions assisting in brain comprehension than the study suggests.

Source:

Rayner, K. , Schotter, E. R. , Tran, R. (2014).  Don't believe what you read(only once): Comprehension is supported by regressions during reading.  Psychological Science.  P. 1-9.

Altered resting functional connectivity of expressive language regions after speed reading training

Summary
This 2014 study in the Journal of Clinical and Experimental Neuropsychology used functional connectivity magnetic resonance imaging (fcMRI) to examine if speed reading training affects the functional architecture of neural networks involved in reading in 9 participants selected for the speed reading training. 

A central premise of speed reading proponents is that subvocalization significantly slows down reading and that there are more efficient ways to read without the need of subvocalization; it is not necessary to cognitively voice words as one reads them in order to understand them. Subvocalization is a consequence of learning how to read phonetically, or sounding out strings of words in the brain. A typical example of reading something without subvocalization is reading a stop sign without cognitively sounding out the syllables. When someone relies on subvocalization as a primary method of reading, their reading speed is limited to their maximum talking speed. Reduction or elimination of subvocalization in favor of direct semantic processing from visual cues (instead of semantically processing subvocalized phonological cues) represents reduced cognitive load and faster reading. 

By measuring detectable changes in functional connectivity in brain regions associated with language via neuroimaging, the researchers found that the participants disassociated the visual input of orthographic word representations from internalized voicing and subvocalization of text while the text is being read after the speed reading training. Furthermore, reading speed measured in words per minute increased at a statistically significant level.

 The researchers recruited 9 participants with comparable reading proficiency and completed initial and follow-up MRI scans before and after performing the 6-week internet-based speed reading training program. EyeQ Advantage based in Salt Lake City provided the program, which consisted of 12 modules designed to facilitate progressively faster reading speed and increased comprehension. Each training exercise lasted ten minutes and participants performed many modules multiple times, with training sessions 3 to 5 times a week. Some of the exercises consisted of reading passages at slow, medium, and fast presentation speeds as well as following a sequence of geometric images around the screen. 

Reading speed (words per minute) pre versus post training (p = .0021 for differences between reading speed pre versus post training).

Critique 
Unfortunately the research makes no reference to the importance of sustaining a satisfactory reading comprehension level when increasing reading speed nor does the research make a reference to measuring comprehension pre versus post training. An increase in reading speed from 200 words per minute to 800 words per minute is not impressive if reading comprehension suffers. The research does provide evidence that speed reading training reduces or eliminates subvocalization using neuroimaging techniques. Subvocalization is one of the obstacles to reading faster identified by proponents of speed reading programs. Generalizability of findings to the general population could be an issue because of the small sample size of 9. 

Source
Ferguson, M., Nielsen, J. and Anderson, J. (2014). Altered resting functional connectivity of expressive language regions after speed reading training. Journal of Clinical and Experimental Neuropsychology, 36(5), pp.482-493.

The Effects of a Speed Reading Course and Speed Transfer to Other Types of Texts

The Effects of a Speed Reading Course and Speed Transfer to Other Types of Texts

By: Tran Thi Ngoc Yen

Summary:

Professors at colleges and universities are teaching speed-reading courses to students to help improve their reading speed. Yen conducted this research to determine the effects speed-reading courses have on reading rate improvements of students in, and outside, of the classroom. According to Yen, there are three fundamental indicators of speed-reading: automaticity, accuracy, and reading speed for silent reading or prosody for oral reading. Researchers suggest students need to maintain a reading comprehension rate of at least 75 percent to have speed-reading be efficient. Normally in a speed-reading course, students maintain a graph of their speed in words per minute (wpm) as well as their reading comprehension score to track progress.

Yen used first year students at a Vietnamese university as participants. The 116 participants were placed “into four groups: two experimental groups, hereafter called group A (31 students) and group B (30 students); and two control groups, hereafter called group C (26 students) and group D (29 students).” Participants in groups A, B, and C were English majors, while the participants in group D were not. Groups A and B took the speed-reading course with additional English classes. The control groups ,C and D, did not follow the speed-reading course, but group C followed the English program at the university and group D attended an English course at a language center.

Participants in groups A and B were required to reach a desired vocabulary level of 1,000 to attend the speed-reading course. In addition, all participants had to read pre and post-test texts from a 1,000 reading level and answer ten reading comprehension questions. Participants read the texts and answered the questions on a computer program. According to the study, the “texts differed from those in the course by being longer, being read on a computer screen rather than in hard copy, and involving different topics from those in the course.” The researchers told the participants to read the texts normally and not as quickly as possible. Researchers distributed 20 texts at 550 words to ensure that few participants were reading the same texts. To score the participants and make the results more reliable, researchers used four scoring methods: the 20^th minus 1^st scoring method, the average scoring method, the extreme scoring method, and the three extremes scoring method. In addition, “participants’ comprehension accuracy was measured by counting the number of correct answers they made on each of the 20 texts in the speed reading course.” 

Participants used progress charts to track their development. There were four main types of charts that participants plotted: gradual increase, erratic increase, plateau increase, and mixed increase. As seen in table 4, groups A and B had an 82 percent gradual increase in their speed change.

With respect to the speed increase transfer from the speed-reading course to other types of texts, the control groups increased an average of 15 wpm and the treatment groups averaged an increase of 48 wpm. In addition, the treatment groups outperformed the control groups on comprehension; most of the treatment groups increased their comprehension accuracy while most of the participants’ in the control groups did not. This research concluded that speed-reading courses helped participants maintain or increase their comprehension while also increasing their reading speed. In addition, “there may be a link between comprehension and reading speed improvement in that participants who greatly increased their speed tended to improve their comprehension accuracy while it was less likely that participants who marginally increased their speeds would improve their comprehension accuracy.” Results also concluded that speed-reading courses were beneficial to the participants because it helped them increase their speed on other types of texts by at least 30 wpm.

Critique:

Although the author did a very thorough job throughout the study, I would have liked to see if there were any differences in other majors besides English in speed-reading, especially since English was not their first language. In addition, the way the author scored the speed-reading was very subjective depending on which one of the four different methods were used. As the results show, taking a speed-reading course could be beneficial. Being able to read and produce more work in the same amount of time would allow analysts to increase their work output. 

Source:

Yen, T. T. N. (2012). The Effects of a Speed Reading Course and Speed Transfer to Other Types of Texts. RELC Journal, 43(1), 23–37. doi:10.1177/0033688212439996

Extensive reading: Speed and comprehension

Summary:

Bell (2001) examined the correlation between reading speed and reading comprehension in both intensive and extensive environments of children.  Subjects in the extensive environments were given longer texts.  Those in the intensive environments received about 30 short passages, usually no longer than 300 words.  Bell expected those in the extensive program to adapt speed-reading in order to meet the time demands.  Those is in the groups were then given comprehension tests on their respective passages.  A correlation analysis was then performed on the reading comprehension results and reading speed.

Subjects in the extensive environment did adapt higher reading speeds to meet time demands.  More importantly, the extensive environment scored higher on reading comprehension tests than those in the intensive environment.  Bell concludes that extensive readings improve reading speed and comprehension than “intensive language exploitation activities.” Furthermore, the more extensive readings a student does, the more his/her reading speed and comprehension increase.

Critique:

There are few items worth noting concerning the findings of this study.  First, it was focused on elementary learners.  Therefore, reading longer texts may not have the same improvements on adults – although that is certainly desirable, especially in the intelligence community.  Second, it is near impossible to make comprehension tests for different texts at the same level of difficulty.  While the tests may have been valid for each text, they may not be valid when considered as a whole in the context of this study.

Now, the findings of this study have interesting implications for intelligence summaries (INTSUMs) and short-form analytic reports (SFARs).  INTSUMs may actually be harmful to decision makers if these findings are applicable to adults and their reading comprehension.  Admittedly, SFARs would not be as harmful as they contain more content, but these findings suggest that long-form analytic reports (LFARs, usually 2 or more pages) are the most preferable tools to spread information and understanding of a current issue.

Future studies on reading speed and comprehension are required in order to support such assertions.  The current study, while interesting, is not enough.

Source:

Bell, T. (2001, April). Extensive reading: Speed and comprehension. Retrieved October 31, 2014, from http://www.readingmatrix.com/articles/bell/ 

The Effect of a Timed Reading Activity on EFL Learners: Speed, Comprehension, and Perceptions

Summary

Anna Chang, in this study, tested the effectiveness of a reading class on reading speed and comprehension. The results of the experiments showed that students doing the timed reading activities increased their reading speed on average by 29 words per minute (wpm) (25%) and comprehension by .63 (4%).

Participants of the study were divided into two groups, an experimental group (n=46) and a control group (n=38). Both groups were enrolled in a required English course for the purpose of preparing a student for the TOEIC (Test of English International Communication). The course ran for 13 weeks with one 2 hour class session per week. The experimental group spent 15 minutes at the end of each class on timed reading exercises and the control group spent the 15 minutes reviewing the previous week’s lesson.

Chang, at the beginning of the course, subjected both groups to a pretest where they were required to read two texts while it was timed. After reading the texts, the participants took a test consisting of 5 multiple choice questions. Chang repeated the process after the 13 week course.

Results of the experiment showed that reading speed increased, on average, in the experimental group by 25% from 118 wpm to 147 wpm compared to only 5% in the control group. Additionally, the number of participants reading above 150 wpm in the experimental group increased from 10% to 39%. 

Figure 1. Reading speed for the experimental and controls groups at Time 1 and 2 (in wpm)

Experimental results for comprehension did not show as much of a difference.

Reading comprehension increased by only a .05 difference between the experimental and control groups. Therefore, this study showed that increasing speed did not decrease comprehension which was found to be true in other studies.

Critique

I do not believe comprehension was fully assessed by this study. The pre and post-tests administered were only 5 to 8 questions. Additionally, the questions were multiple choice; therefore, increasing the chance of a correct answer without actually comprehending it. A pre and post with a greater amount of questions would be a better measure for assessing whether a student fully comprehends what they are reading.

Source:

Chang, Anna. (2010). The effect of a time reading activity of EFL learners: Speed, comprehension, and perceptions. Reading in a Foreign Language, 22(2). 

Summary of Findings: Delphi Technique (4 out of 5 stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 5 articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in October 2014  regarding Delphi Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

The Delphi technique is a method that relies on expert and group knowledge to make more accurate forecasts using incomplete information.  The individual forecasts are compiled after a series of rounds.  Then, the individuals’ responses are anonymized and dispersed to the remainder of the group for consideration, and new individuals forecasts are given.  

The RAND corporation created the Delphi technique in order to support accurate decision making in the face of incomplete information.  There is a substantial amount of research on the validity of the Delphi technique dating back to its creation in the 1950s, but the methodologies scholars have used to test Delphi’s effectiveness have varied in almost every study.  

Strengths:

1. Conducted in writing or electronically and does not require face-to-face meetings

2. Helps generate consensus or identify divergence of opinions among group members

3. Participants are relatively free of social pressure, influence, and dominance from other group members

4. Anonymous responses allow respondents to keep opinion until they are comfortable changing an estimate

5. Is inexpensive

Weaknesses:

1.  Time for answers may not be given to the problem and consensus may not be obtained

2. Participants may ignore feedback

3. Experts may not be defined among the group

4. Requires adequate time and participant commitment

5. More time consuming than other group methods

6.Broad guidelines-- there are at least 27 different ways to conduct the method

Step by Step:  

1. Use a group of 5-20 heterogeneous experts or people with appropriate knowledge of the subject.
2. The entire process must use a systematic process, particularly with anonymous feedback and a controlled method of dispersing responses and feedback.
3. A minimum of three iterations should be conducted with polling continuing until there is a stability in responses.

Exercise:

We used Delphi Decision Aid online software to conduct three 5 minute rounds of Delphi to forecast how many second year Applied Intelligence graduate students will have at least one full time job offer in an intelligence-related field by graduation and how many pages second year Applied Intelligence will have completed on average by October 29 toward a thesis. The first round also contained a ranking question to rank panel expertise on various topics to inform further Delphi questions for subsequent rounds. Subsequent rounds asked the two original questions in addition to predicting the outcome of the National Football League AFC division this session, how many selfies Kim Kardashian will have in her book scheduled for publishing in April 2015, and what the S&P 500 index will be in early November. After each round, the panel had a few minutes to review the feedback of the round through statistical aggregation of responses and written comments explaining why a panelist made the estimate that they did. 

What did we learn from the Delphi Exercise

1. Delphi works well with broad questions where the expertise of one person is not sufficient to encompass the entire scope of the question.
2. Literature suggests that panelists tend to perform poorly on questions asking them to rank various items from best to worst and that self-reported expertise is not a best practice for panel selection.
3. Delphi is designed to collect expert estimates  in cases where a variety of relevant factors (economic, technical, etc.) ensure that individual panelists have limited knowledge and could reasonably benefit from communicating with other experts possessing different information.   
4. Estimates from panelists do not have to be quantitative such as in prediction markets.

Additional Resources Of Interest:

Delphi Decision Aid

The Delphi technique as a forecasting tool: issues and analysis

Summary
This 1999 meta-analysis in the International Journal of Forecasting gathered all the peer-reviewed journal articles and book chapters (English-language) that experimentally evaluated the Delphi technique as a structured forecasting method with specific control conditions. The search yielded 27 studies in all and the researchers produced tables summarizing the methods and findings. Additionally, the researchers contacted the authors of the evaluative studies to comment upon the coding and interpretation of each author's own paper. The meta-analysis found that Delphi groups outperform statistical models and unstructured interacting groups. However, no conclusive evidence found that Delphi outperforms other structured group procedures such as Nominal Group Technique (NGT). Two studies found that NGT groups make more accurate judgments than Delphi groups, three studies found no notable differences in accuracy between them, and one study showed Delphi superiority. One study found that the Problem Centered Leadership (PCL) approach, which involves instructing group leaders in appropriate group-directing skills, outperforms Delphi. An unintended finding was that generalizations about Delphi from the meta-analysis is difficult because of confounding variables in various studies that did not sufficiently control for group, task, and technique characteristics such as panelist expertise and the nature of feedback used. Therefore, the researchers conclude that a Delphi conducted according to "ideal" specifications might perform better than laboratory interpretations. The researchers conclude that future research requires a shift of focus from the final estimative output to analyzing the process of judgment change within groups. 

Delphi was explicitly designed for use with experts in cases where a variety of relevant factors (economic, technical, etc.) ensure that individual panelists have limited knowledge and could reasonably benefit from communicating with others possessing different information.  

The majority of Delphi evaluative studies tend to use artificial tasks that may be easily validated, contain sampling problems, and simple feedback in the place of meaningful and coherent tasks, experts, and complex feedback. 

Two primary suggestions the researchers prescribe to make future evaluative studies more effective include following a precise definition of Delphi to prevent the technique from being misrepresented in the laboratory and requiring a much greater understanding of factors that influence Delphi effectiveness to prevent the technique's potential utility from being underestimated through inaccurate representations used in badly designed scenarios. 

The solution the researchers propose is focusing research on the way in which the estimates from an expert panel in round 1 are transformed through the Delphi process into a final round estimate. The transformation must be measured through changes over rounds in judgments, changes in the individual accuracy of panelists, judgmental intercorrelations, and characteristics such as attrition rate and group size. Through this process, researchers will identify which factors are the most important in explaining how and why an individual changes a judgment and which are related to change in the direction of increased accuracy. Only after that can research explaining the differences between structured procedures commence. 

The four necessary defining attributes of Delphi are anonymity, iteration, controlled feedback, and the statistical aggregation of group response however there are numerous ways in which they may be applied. One of the goals of Delphi is achieving greater consensus, usually measured by the variance in responses of panelists over the course of the Delphi process. While this is a typical trend, a study measuring "post-group consensus" found that Delphi produces little increased agreement and that panelists simply alter their estimates to conform to the group without actually changing their opinions. Another study found that experts with extreme views are more likely to drop out of a Delphi procedure, suggesting that consensus may be due to attrition. Further research is required to determine the extent that consensus is due to "true consensus" versus conformity pressures. 

The researchers state the the first round of a classical Delphi procedure is unstructured to allow individual experts to identify and elaborate on the issues most important to solving a problem. The monitor team then produces a structured questionnaire from which the judgments of the Delphi panelists may be elicited in a quantitative manner in subsequent rounds. After each round, the monitor team aggregates the responses and sends them back to the panelists for further consideration. From the third round onwards, panelists have the opportunity to alter prior estimates on the basis of the provided feedback and if a judgment falls outside the upper or lower quartiles they may be asked to give reasons why they believe their selections are correct against the majority opinion. The procedure continues until the judgments of the panelists attain a designated threshold.  

The majority of the studies in the meta-analysis contained structured first rounds in which event statements devised by researchers are presented to panelists for assessment with no opportunity for them to indicate the issues they believe to be the greatest importance on the topic via an unstructured round. Other studies used almanac questions involving estimating the diameter of planets in the solar system or the tonnage of a certain material shipped from New York in a certain year, which are inappropriate for Delphi.

Studies in the article showed that panels composed of experts tend to benefit from a Delphi procedure to a greater extent than groups of novices and that laboratory studies sampling from homogenous groups underestimate the value of Delphi. Technique-comparison studies ask if Delphi works yet use techniques that differ from one study to the next and that deviate from the intended purpose of Delphi. 

Pertaining to the role of feedback in improving forecasting accuracy, one study found that different types of feedback provide different effects. A study comparing iteration, statistical feedback (means and medians) and 'reasons' feedback (with no averages) found that the greatest degree of improvement in accuracy over the course of Delphi occurred in the 'reasons' condition. Although participants were less likely to change their forecasts under the 'reasons' condition, when they did change their forecasts they became more accurate, which was not the case for the iteration and statistical treatment groups. Another study found that feedback combining reasons, the median, and range of estimates was more accurate than providing only a median and range. 

Previous research used many technique formats to represent Delphi, varying on every aspect such as the type of feedback used, selection of panelists, and types of questions. Therefore, using Delphi as originally intended may lead to greater enhancement of accuracy than is reflected in the articles in the meta-analysis. Whenever an experiment changes an analytic step in Delphi procedures shown to influence the performance of the technique, the experiment is essentially studying a different technique. 
  
Critique 
This is a comprehensive meta-analysis. Out of the 27 experimental designs factored in the meta-analysis, no experimental design followed the procedures of a classical Delphi group as referenced in the article. The researchers describe the methodological features of Delphi in each experimental study over 4 tables across multiple pages and sought feedback from the original authors to ensure that the meta-analysis coded and interpreted the procedures and findings of each study appropriately. Despite problems in representing Delphi under experimental conditions, existing literature finds the formats used to represent Delphi produce more accurate forecasts than unstructured group interaction. Nevertheless, no definitive empirical comparisons to other structured techniques can be made until an evaluative study sufficiently controls for group, task, and technique characteristics such as panelist expertise, the nature of feedback used, the structure of a designated amount of rounds, and the type of resolvable questions asked of the group using Delphi. 

Source 
The Delphi technique as a forecasting tool: issues and analysis. Gene Rowe , George Wright (1999)